Essentially for $SU(N)$ you find that tenors can be written in terms of upper and lower indices (except for $SU(2)$ which can be written solely in upper or lower indices due to the group being pseudo-real). In order to pick out the irreps using the highest weight procedure one finds that a tensor, $V_H$ which we describe by $(n,m)$ $n$ being the number of upper indices and $m$ the number of lower, has two properties when acted one by the generators of the group
- $V_H$ is symmetric in the upper indices and symmetric in the lower indices
- $V_H$ is traceless
Thus, in order to count the dimensionality of the tensor, we find that we can use simple counting. For the upper indices, that go from 1 to $N$, the number of ways to arrange them is
\[\binom{n + N -1}{N-1} = \frac{(n + N -1)!}{n!(N-1)!} \]
Again for the lower indices
\[\binom{m + N -1}{N-1} = \frac{(m + N -1)!}{m!(N-1)!} \]
Thus the total number of ways to have symmetric upper and lower indices is
\[B(n,m) = \frac{(n + N -1)!}{n!(N-1)!} \frac{(m + N -1)!}{m!(N-1)!} \]
However, the tensor is also required to be traceless. This says that the object
we get by taking the trace vanishes, and it is symmetric in $n-1$ upper indices
and $m-1$ lower indices. Thus this imposes $B(n-1, m-1)$ constraints, so the
total is
\[D(n,m) = B(n,m) - B(n-1,m-1)\]
Using, this equation for the case of N=3, n=3, m=0, you find that D = 10, as you expected.
I would highly suggest learning the Young Diagrams method that was listed in the comments, for it is extremely quick to calculate tensor products of irreps in $SU(N)$. The method I described above was found in Howard Georgi's Book Lie Algebras in particle physics, chapter 10.
EDIT: I just wanted to add that $B(n,m) \equiv 0$ if one of the terms is negative. This is just to say that one does not have any traceless constraints if there is nothing to contract.