First, let's try and clean up the MTW expression a little bit by getting rid of the manifestly coordinate variant stuff, by introducing a proper contraction, a proper integral over the surface at infinity, and the unit normal $r^{a}$ to the surface at infinity:
$$M_{ADM}={16\pi}\oint\sqrt{\gamma}\,\,d^{2}x \gamma^{ab}r^{c}\left(\gamma_{ac,b}-\gamma_{ab,c}\right)$$
This is only valid, however, not just only for in asymptotically flat spaces, but actually only for asymptotically Cartesian coordinates. You can prove this to yourself by calculating the ``ADM Mass'' of the flat 3-metric in polar coordinates:
$\begin{align*}
16\pi M_{ADM}&=\lim_{r\rightarrow\infty}\oint r^{2}\sin\theta \gamma^{ab}\left(\gamma_{ra,b}-\gamma_{ab,r}\right)d\theta d\phi\\
&=\lim_{r\rightarrow\infty}4\pi r^{2}\left(\gamma^{rr}\gamma_{rr,r}-\gamma^{ab}\gamma_{ab,r}\right)\\
&=\lim_{r\rightarrow\infty}4\pi r^{2}\left(0-\frac{4}{r}\right)\\
&=-\infty
\end{align*}$
Obviously, this is wrong, and just an artifact of the way that spherical coordinates behave at infinity. To fix this, we need to always subtract the divergence from the ADM mass of flat spacetime from the ADM mass of the coordinate system in question. So, that's the origin of the $H$ and $H_{0}$ terms. The $H$ term is the extrinsic curvature of the space in question, while $H_{0}$ is the extrinsic curvature of flat spacetime. Now, it's just a matter of showing that the expression inside the integral is equal to $H$.
Typically, I define the extrinsic curvature as $\gamma^{ab}\nabla_{a}r_{b}$. So, let's go on an adventure:
$\begin{align*}
H&=\gamma^{ab}\nabla_{a}r_{b}\\
&=\gamma^{ab}\left(\partial_{a}r_{b}-\Gamma_{ab}{}^{c}r_{c}\right)\\
&=\gamma^{ab}\partial_{a}r_{b}-\frac{1}{2}\gamma^{ab}\gamma^{cd}\left(2\gamma_{ad,b}-\gamma_{ab,d}\right)r_{c}\\
&=\gamma^{ab}\partial_{a}\left(\gamma_{bc}r^{c}\right)-\gamma^{ab}r^{c}\left(\gamma_{ac,b}-\frac{1}{2}\gamma_{ab,c}\right)\\
&=\delta^{a}{}_{c}\partial_{a}r^{c} + \gamma^{ab}r^{c}\gamma_{bc,a}-\gamma^{ab}r^{c}\left(\gamma_{ac,b}-\frac{1}{2}\gamma_{ab,c}\right)\\
&=\partial_{a}r^{a}+\frac{1}{2}\gamma^{ab}r^{c}\left(\gamma_{ac,b}-\gamma_{ab,c}\right)
\end{align*}$
Now, we note that in any coodinate system adapted so that the coordinate $r=constant$ that determines the surface is chosen for one of the coordinates, we have, necessarily have $r^{a}=(1,0,0)$, we note that $\partial_{a}r^{a}=0$, and we thus conclude that the two expressions for ADM mass are equivalent.
This post has been migrated from (A51.SE)