Question

Why is the symmetry group of the $A_{N-1}$ theory the $Osp(2,6|4)$? And how do we extract the fact that the bosonic subgroup is $Spin(2,6)\times Spin(5)$?

The $A_{N-1}$ theory is the resulting 6d SCFT made out of a stack of $N$ coincident $M5$-branes. The $Spin(5)$ is supposed to be the rotation group in the transverse space of the $M-5$-branes and gives the $R$-symmetry but this is the double cover of $SO(5)$ and I do not understand why the $R$-symmetry is not given by the $SO(5)$. A reference is this one.

Answer 1

The $A_{N-1}$ theory is a $N=(2,0)$ 6d superconformal theory. So its symmetry group is the $N=(2,0)$ 6d superconformal group. Let $G$ be this group and let us try to determine $G$. Let us first work at the level of Lie superalgebras. Let $\mathfrak{g}$ be the Lie superalgebra of $G$. As in any superconformal algebra, the bosonic part of $\mathfrak{g}$ is made of the ordinary conformal algebra and of the $R$-symmetry algebra.

It is well known that the conformal algebra of the $d$-dimensional Minkowski spacetime is $so(2,d)$ (this result plays for example a key role in $AdS/CFT$ because $so(2,d)$ is naturally the Lie algebra of the isometry group of $AdS_{d+1}$). In particular, in $6$ dimensions, it is $so(2,6)$. Remark that the 6d conformal algebra $so(2,6)$ naturally contains the 6d Lorentz algebra $so(1,5)$.

To go further, one has to understand the fermionic part. The 6d Lorentz algebra has Weyl spinors with $8$ real components. The $N=(2,0)$ supersymmetric algebra contains 16 real supercharges transforming in two copies of the same chirality of this Weyl spinor. Remark that there is an exceptional isomorphism of Lie algebras $so(1,5)=sl(2, \mathbb{H})$, where $\mathbb{H}$ is the field of quaternions, and the 8 real spin representation of $so(1,5)$ is simply the fundamental representation $\mathbb{H}^2$ of $sl(2, \mathbb{H})$.

The ferminonic part of $\mathfrak{g}$ contains not only these 16 real supercharges of the standard supersymmetry algebra, which square to a spacetime translation, but also conformal supercharges which square to special conformal transformations. Conformal supercharges form another copy of two Weyl spinors with respect to $so(1,5)$. In conclusion, the full fermionic part of $\mathfrak{g}$ has $16.2=32$ real components and is the copy of two Weyl spinors of the conformal algebra $so(2,4)$.

 $R$-symmetry rotates the $N=1$ superalgebra inside the $N=2$ superalgebra. As the supercharges are in quaternionic representations, the $R$-symmetry algebra is $usp(4)$ i.e. the algebra of automorphisms of $\mathbb{H}^2$. There is an exceptional isomorphism of Lie algebras $usp(4)=so(5)$ (this in clear at the level of Dynkin diagrams: $B_2$ and $C_2$ are the same).

Finally we have found that the bosonic part of $\mathfrak{g}$ is $so(2,6)\oplus usp(4)=so(2,6) \oplus so(5)$. This superalgebra is called $osp(2,6|4)$: the "$o (2,6)$" notation signals the presence of $so(2,6)$ in the bosonic part and the "$sp( |4)$" notation signals that the $R$-symmetry is $usp(4)$.

To go from Lie algebras to Lie groups, one has to see what are the representations used. The superconformal algebra $so(2,6)$ acts on the supercharges through the spinor representation so the relevant group is the full $Spin(2,6)$. Similarly, the $R$-symmetry acts on the supercharges through the fundamental representation $\mathbb{H}^2$ of $usp(4)$ which is a spinor representation of $so(5)$ so the relevant group is the full $Spin(5)$.

Much more information can be found in these notes

http://www.physics.rutgers.edu/~gmoore/FelixKleinLectureNotes.pdf

by Moore. The section 3 is particularly relevant.