I don't think that there is a known nice mathematical explanation in the sense I guess you would like, i.e. different from simply computing both sides and checking that they match. To support this point of view: at the beginning of the following video
Kontsevich explains that the large N asymptotics of the volumes of the unitary groups U(N) can be naturally expressed in terms of Euler characteristics of moduli spaces of Riemann surfaces and claims that there is no known intrinsic satisfactory explanation. Such relation is a special case of the Gopakumar-Vafa correspondence and of course Kontsevich is aware of that and so what he is asking for is something more and I don't think that someone knows the answer.
Nevertheless, for someone not comfortable with the AdS/CFT point of view, they are maybe more approachable explanations in the physics context. For example, in this paper:
http://arxiv.org/abs/hep-th/0205297
Ooguri and Vafa give a physical derivation of the correspondence from the worldsheet point of view whereas in this paper:
http://arxiv.org/abs/hep-th/0011256
Atiyah, Maldacena and Vafa give a physical derivation from the "dual" point of view, i.e. from the spacetime point of view. This paper tries to reduce the correspondence to a geometric statement (a flop in 7 dimensions) and so is maybe more accessible to a mathematician.