Hi. I am going to lay out for you how to calculate the spectrum for mixed boundary conditions. I hope this helps because you might have some confusions on the zero point energies.
Let us take a specific example which you will be able to alter. Let us consider the D3-D7 brane system and consider open RNS superstrings. We can have four possible sets of boundary conditions which are NN, ND, DN, DD. N signifies Neumann and D signifies Dirichlet boundary conditions. It can be shown that the zero point energy in the R sector is zero, \(a_R=0\) . Also, it can be shown that for the NS sector we get \(a_{NS}=1/2\). Now, you must be familiar on what happens for NN and DD boundary conditions so I am not going to repeat it here. Instead, let's look at the mixed cases. As I said above there are two possibilities, ND and DN. It is possible to find the following contributions to the normal ordering constant \(a\)from half-integer or integer bosonic and fermionic modes. Note that all possibilities must be taken into account.
\(a_{bi}=\frac{1}{24}, \,\, a_{bh}=\frac{1}{48}, \,\, a_{fi}=-\frac{1}{24}, \,\, a_{fh}=\frac{1}{48}.\)
Where b,f stand for bosons and fermions respectively, while i and h stand for integer and half-integer respectively. Now, one requirement that must always be satisfied is that the zero-point energy of the R-sector is zero. Let us now consider that out of the d-2 polarization modes, ν of them are either ND or DN with half-integer bosonic modes. Then the contributions of the bosons to the zero-point energy is \(\frac{2(d-2)-3\nu}{48}.\)Therefore, the fermions in the R-sector should contribute \(\frac{3\nu-2(d-2)}{48}.\)The fermions which are polarized alonf d-2-ν NN or DD directions have integer modes and contribute \(-\frac{d-2-\nu}{24}.\)The fermions which are polarized along ν DN or ND directions therefore must have half-integer modes and contribute \(\frac{\nu}{48}.\)Then, in total, we get the required contribution \(\frac{2\nu-2(d-2)}{48}.\)This allows us to compute the zero point-energy for the NS sector which we find to be
\(a_{NS} = \frac{1}{2}-\frac{\nu}{8}.\)
Remember that \(a_R\)is always zero and the ground state of a massless string is always in some fermionic representation. For \(\nu=4\) we get \(a_{NS}=0\) and therefore the mass of the lowest NS state is zero. This state is not projected out by the GSO so that we have an equal number of massless bosons and fermions building up the vector supermultiplet.
Now after these details that are needed to convince yourself you are calculating stuff right we can move onto the spectrum for D3-D7 branes. The cases of NN and DD must be known to you. In the NS sector of D3's you get the \({\bf{8}}_V\)giving a gauge boson and 6 scalars in 4d. In the R sector you get the \({\bf{8}}_C\)getting four 2-spinors in 4d. For the NS sector of the D7's you get the \({\bf{8}}_V\)again (which has a gauge boson with six degrees of freedom and two scalars) and in the R sector you get the \({\bf{8}}_C\), a \(SO(8)\) spinor.
Now, for the mixed part. Let us denote the coordinates as numbers. E.g. \(x_1 \equiv 1\)etc. In 0,...,3 we will have NN boundary conditions, in 4,...,7 we will have DN and in 8,9 we will have DD. Now, you would have to perform the oscillator expansions etc in order to be able to calculate the spectrum and I am not gonna do this here (but you can find help in Zweibach's book). Once you get the mass formulae you are able to compute the spectrum. For the NS sector we get three states which transform in the \({\bf{1}}, {\bf{4}}_C, {\bf{1}}\) under SO(4) respectively and give us 4 complex scalars. In the R sector we get two states transforming in the \({\bf{4}}_C\) under SO(4) and \({\bf{2}}_C\) under SO(2) giving an SO(2) spinor.
Now, the D3-D7 intersection preserves 1/4 of the total susy of type IIB, i.e. 8 out of 32 supercharges and has an SO(4)xSO(2) isometry in the directions transverse to the D3-brane. The SO(4) rotates in the 4,...,7 and the SO(2) rotates in the 8,9. Now, take a look at the moddings
\[\begin{array}{ l c c r } \text{Directions} & 0,1,2,3 & 4, 5, 6, 7 & 8,9 \\ \hline \text{Bosons} & \alpha_n & \alpha_{n+1/2}& \alpha_n \\ \text{NS}& b_{n+1/2} & b_n & b_{n+1/2} \\\text{R} & d_{n} & d_{n+1/2} & d_{n} \\ \hline \end{array}\]
Try to consider the states now. For the R-sector we get a SO(3,1) spinor with two degrees of freedom since Weyl spinors have two complex degrees of freedom and going on-shell cuts them in two, transforming under SO(2), in the \({\bf{2}}_C\) giving another 2 fermionic degrees of freedom and a 2d spinor transforming under SO(2) for the directions 8,9. The latter is only being seen as internal degrees of freedom for the R sector thus giving 4 degrees of freedom in total. By considering the 7-3 string we get another 4 degrees of freedom, thus in total we have 8 sermonic degrees of freedom. For the NS sector we get a scalar for 0,..., 3$and another scalar for 8,9. For 4,..., 7 we get a 4d spinor transforming under SO(4) with four degrees of freedom which will act as internal degrees of freedom. Finally we get another scalar from 8,9 so in the end multiplying out the total degrees of freedom we get 4. Now, we have to also take into account the 7-3 string just like we did for the fermions and not just the 3-7. Thus we get another 4 scalars and in the end the summation of total degrees of freedom yields 8. The previous result comes into agreement with the fact that the 3-7 string (together with the 7-3 in order the theory in order to be complete) saves 8 out of the 32 super symmetries and we see that the requirement \(N_B = N_F\) is satisfied indeed. Finally note that the states of the 3-7 string transform in the \({\bf{N}} \times {\bf{\bar{K}}}\) (and for the 7-3 \({\bf{K}} \times {\bf{\bar{N}}}\)) where the the D3-D3 system has the gauge group \(U(N)\) and the D7-D7 system has gauge group \(U(K)\). The bars above come from the fact that the Chan-Paton indices live on the fundamental and anti-fundamental of the gauge group.
Now, its almost straight forward to generalize to other systems as well.
I hope this helps a little bit. If you need more help ask here (maybe someone else can explain better than me) or try except for the classic textbooks the one by Ibanez and Uranga.