You want to write down the most general $G$-invariant polynomial constructed out of $\Phi$ and/or $\Phi^*$. All invariants can be identified by treating the product of several fields as a tensor product of representations, say $\Phi_{ij}\Phi^{*kl}\Phi_{mn}\Phi^{*op}$, and projecting out the singlet component. This is done by contracting all indices with $G$-invariant tensors. For example, the only (algebraically independent) invariant tensors of $SU(N)$ are $\delta^i_j$, $\epsilon_{ijk}$ and $\epsilon^{ijk}$. This tells you immediately that an invariant constructed out of a single rank-2 tensor (whether symmetric or antisymmetric) field $\Phi$ is necessarily a function of $t_n\equiv{\rm tr}(\Phi^\dagger\Phi)^n$, $\det\Phi$ and $\det\Phi^\dagger$. This already solves the problem how to write down the most general $G$-invariant function. However, the number of independent parameters in this function can be reduced by noting that the above invariants are all not independent. Since $\Phi^\dagger\Phi$ is a Hermitian positive-semidefinite matrix, all $t_n$ depend just on its $N$ real non-negative eigenvalues and thus only $t_n$ with $n=1,...,N$ are independent. They can be obtained from the generating function
$$
f(z)\equiv{\rm tr}\log(1+z\Phi^\dagger\Phi)=\sum_{n=1}^\infty\frac{(-1)^{n+1}}nt_nz^n.
$$
In terms of the eigenvalues $\lambda_k$, this reads $f(z)=\log\prod_{k=1}^N(1+z\lambda_k)$ which can in turn be expressed in terms of $t_n$ with $n=1,...,N$ using a variant of Viete's formulas. The same strategy can be used for a field $\Phi$ in an arbitrary tensor representation as well as for other groups, taking into account the appropriate invariant tensors. Additional relations among the different invariants can occur if the tensor has some symmetry or satisfies some constraint. For example, for a traceless tensor in the adjoint representation of $SU(N)$ one has ${\rm tr}\Phi^4=\frac12({\rm tr}\Phi^2)^2$ for $N=2,3$. Of course, if you are only interested in renormalizable interactions, you will need just a small part of this machinery.
This post has been migrated from (A51.SE)