Lets begin by defining more precisely the objects which we are dealing with there. Let ϕ=(Xμ):Σ⟶M an embedding of our world-sheet Σ in spacetime M with metric η, h=ϕ∗g∈secT(0,2)(Σ) the metric induced on the world-sheet Σ by the background metric η, and let γ=secT(0,2)(Σ) be a metric on the world-sheet. Observe that h and γ are unrelated: the former is induced by the extrinsic geometry, the latter defines the intrinsic world-sheet geometry. The world-sheet is oriented by the 2-form ω∈⋀2(Σ) defining our element of volume. The Hodge dual ⋆γ is then the duality operator define on the oriented pseudo-Riemannian surface (Σ,γ,ω) (and not the Hodge dual induced on the bundle of differential forms of the background metric!). We may then define the action functional
S[Xμ,γ]=12∫Σgμν[X]dXμ∧⋆γdXν
where (gμν) is family of functions living on Σ depending on the non-linear sigma model you are dealing with. (For a bosonic string moving in a curved background M with curved metric Gμν, we let gμν=Gμν. To study the interaction with gravitons, let Gμν=ημν+ξμν for a small metric pertubation ξ . See Polchinski 3.7 for details).
Since you only want to know how to study the calculus of variations with this formalism, lets choose the linear sigma model for which gμν=ημν. Observe that from the calculus of exterior differential forms, the component version of this action is actually
S[Xμ,γ]=12∫Σημνγ(dXμ,dXν)ω=12∫Σημνγab∂aXμ∂bXν|γ|1/2dσ∧dτ
This is just the Polyakov action. The fact is that α∧⋆γβ=γ(α,β)ω for all α,β∈⋀Σ of the same degree. So, in the enunciation of this question, the author forgot to include the volume element, and the corresponding metric determinant, in the second line of the second paragraph).
To do the variation using geometric formalism, consider the variation Xμ↦Xμ+ˉδXμ. (The bar on the delta will distinguish the variation symbol to the coderivative, to be introduced below). The action becomes, after using the Leibnitz rule for the exterior derivative, Stokes theorem and ignoring boundary (∈∂Σ) terms,
ˉδXS[X,γ]=−∫ΣημνˉδXμ∧d⋆γdXν=0
Recalling that, for surfaces of Lorentz signatures the coderivative is δ=−⋆γd⋆γ we get the equation of motion δdXμ=0. If ◻=dδ+δd denotes the Laplace-Beltrami operator on the world-sheet, our equations of motion are just the wave-equations
◻Xμ=0
since δX=0 for all 0-forms X∈⋀0Σ.
To carry the world-sheet metric variation now, I will introduce the notation ⟨T|V⟩ for the contraction of the tensors T,V of the same degree, and α⌟β for the contraction of the forms α,β of the same degree. Observe that by varying the world-sheet metric γ, one is changing both the volume element ω and the Hodge star ⋆γ. The variation of terms involving the Hodge dual is not so trivial, and in fact, can be proven (assuming deg(α)=deg(β)=1) to be just
ˉδ(α∧⋆γβ)=⟨12(α⌟β)γ−1−α⊗β|ˉδγ⟩ω
where by γ−1 I actually denote the inverse metric of γ (namely, whose elements are (γab) in world-sheet coordinates). This formula can be used to perform the variation of the Einstein-Hilbert action in differential forms which I commented in another question here and is based on Thirring's course on mathematical physics. I have also seen it been derived in Göckeler & Schücker and in this paper. If someone is really interested, I can provide a more easier derivation of this result upon request. Anyway, applying this to our equation gives:
ˉδγS[X,γ]=−∫Σ⟨12(dxμ⌟dXμ)γ−1−dXμ⊗dXμ|ˉδγ⟩ω
and the equation of motion completely fix our (auxiliary) world-sheet metric as
12(dXμ⌟dXμ)γ−1=dXμ⊗dXμ
By recalling that hab=∂aXμ∂bXμ is our induced metric on the world-sheet, the above equation just becomes the well-known formula
12γcdγabhab=hcd
that you may grab your copy of Polchinski and see Eq.(1.2.16) (with slight changes in index positioning). So, to summarize, in differential forms, the equations of motion for the bosonic Polyakov action are just
◻Xμ=012(dXμ⌟dXμ)γ−1=h