Essentially for SU(N) you find that tenors can be written in terms of upper and lower indices (except for SU(2) which can be written solely in upper or lower indices due to the group being pseudo-real). In order to pick out the irreps using the highest weight procedure one finds that a tensor, VH which we describe by (n,m) n being the number of upper indices and m the number of lower, has two properties when acted one by the generators of the group
- VH is symmetric in the upper indices and symmetric in the lower indices
- VH is traceless
Thus, in order to count the dimensionality of the tensor, we find that we can use simple counting. For the upper indices, that go from 1 to N, the number of ways to arrange them is
(n+N−1N−1)=(n+N−1)!n!(N−1)!
Again for the lower indices
(m+N−1N−1)=(m+N−1)!m!(N−1)!
Thus the total number of ways to have symmetric upper and lower indices is
B(n,m)=(n+N−1)!n!(N−1)!(m+N−1)!m!(N−1)!
However, the tensor is also required to be traceless. This says that the object
we get by taking the trace vanishes, and it is symmetric in n−1 upper indices
and m−1 lower indices. Thus this imposes B(n−1,m−1) constraints, so the
total is
D(n,m)=B(n,m)−B(n−1,m−1)
Using, this equation for the case of N=3, n=3, m=0, you find that D = 10, as you expected.
I would highly suggest learning the Young Diagrams method that was listed in the comments, for it is extremely quick to calculate tensor products of irreps in SU(N). The method I described above was found in Howard Georgi's Book Lie Algebras in particle physics, chapter 10.
EDIT: I just wanted to add that B(n,m)≡0 if one of the terms is negative. This is just to say that one does not have any traceless constraints if there is nothing to contract.