a) For a rotation of axis $\vec n$ and angle $\theta$, the transformation for a spin $1/2$ representation is :
$$\psi \to e^{i \frac{\theta}{2} \vec\sigma. \vec n}\psi = G(\vec n,\theta)\psi\tag{1}$$
where $\psi$ is a 2-component row complex spinor. Here $G(\vec n,\theta)$ is a member of $SU(2)$. The action of a matrix on a 2-component row object is always at the left of the object.
b) Beginning with : $\vec S=\frac{1}{4}tr(\Psi^\dagger \vec \sigma\Psi)$, suppose we have a transformation $\Psi \to \Psi G$. As you noticed, it would not change the value of $\vec S$, because :
$$tr(G^\dagger\Psi^\dagger \vec \sigma\Psi G)=tr(\Psi^\dagger \vec \sigma\Psi GG^\dagger) = tr(\Psi^\dagger \vec \sigma\Psi) \tag{2}$$
But $\vec S$ is a vector (it is in the fundamental or vectorial representation of $SO(3)$, or, if you prefer, in the adjoint representation of $SU(2)$), so it has to change under a rotation. So the transformation $\Psi \to \Psi G$ is non-valid and irrelevant.
This post imported from StackExchange Physics at 2014-03-09 08:43 (UCT), posted by SE-user Trimok