I don't think that there is a known nice mathematical explanation in the sense I guess you would like, i.e. different from simply computing both sides and checking that they match. To support this point of view: at the beginning of the following video
Kontsevich explains that the large $N$ asymptotics of the volumes of the unitary groups $U(N)$ can be naturally expressed in terms of Euler characteristics of moduli spaces of Riemann surfaces and claims that there is no known intrinsic satisfactory explanation. Such relation is a special case of the Gopakumar-Vafa correspondence and of course Kontsevich is aware of that and so what he is asking for is something more and I don't think that someone knows the answer.
Nevertheless, for someone not comfortable with the AdS/CFT point of view, they are maybe more approachable explanations in the physics context. For example, in this paper:
http://arxiv.org/abs/hep-th/0205297
Ooguri and Vafa give a physical derivation of the correspondence from the worldsheet point of view whereas in this paper:
http://arxiv.org/abs/hep-th/0011256
Atiyah, Maldacena and Vafa give a physical derivation from the "dual" point of view, i.e. from the spacetime point of view. This paper tries to reduce the correspondence to a geometric statement (a flop in 7 dimensions) and so is maybe more accessible to a mathematician.