Unitary gauge in $Z_2$ lattice gauge theory with matter field

Question

A $Z_2$ gauge theory with Ising matter field on a 2-dimensional square lattice has the Hamiltonian

                       $\begin{equation} \begin{split} H=&-t\sum_{\vec r,j}\sigma_j^x(\vec r)-g\sum_{\vec r}\sigma^z_1(\vec r)\sigma^z_2(\vec r)\sigma^z_3(\vec r)\sigma^z_4(\vec r)\\ &-\lambda\sum_{\vec r}\tau^x(\vec r)-\mu\sum_{\vec r,j}\tau^z(\vec r)\sigma_j^z(\vec r)\tau^z(\vec r+\hat e_j) \end{split} \end{equation}$       
where $\sigma$'s are Pauli operators for the gauge field living on the links and $\tau$'s are the Pauli operators for the matter field living on the sites. $\vec r$ denotes the position of a site and $j$ denotes the link attached to the site which is in the $\hat e_j$ direction. The local gauge transformation is induced by 

$$\begin{equation} \tau^x(\vec r)\sigma^x_{\hat e_x}(\vec r)\sigma^x_{-\hat e_x}(\vec r)\sigma^x_{\hat e_y}(\vec r)\sigma^x_{-\hat e_y}(\vec r) \end{equation}$$

In chapter 9.10 of the book *Field theories of condensed matter physics* by Fradkin, he said we can choose a unitary gauge defined by

$$\begin{equation} \tau^z(\vec r)=1 \quad\forall\ \vec r \end{equation}$$
so that the last term becomes
$$\begin{equation} -\mu\sum_{\vec r,j}\sigma_j^z(\vec r) \end{equation}$$ On the other hand, gauge invariance implies the Hilbert space we are considering is the one that consists of vectors which are invariant under the local gauge transformation, so the third term becomes
$$\begin{equation} -\lambda\sum_{\vec r}\sigma^x_{\hat e_x}(\vec r)\sigma^x_{-\hat e_x}(\vec r)\sigma^x_{\hat e_y}(\vec r)\sigma^x_{-\hat e_y}(\vec r) \end{equation}$$
so the Hamiltonian becomes 
                     $\begin{equation} \begin{split} H=&-t\sum_{\vec r,j}\sigma_j^x(\vec r)-g\sum_{\vec r}\sigma^z_1(\vec r)\sigma^z_2(\vec r)\sigma^z_3(\vec r)\sigma^z_4(\vec r)\\ &-\lambda\sum_{\vec r}\sigma^x_{\hat e_x}(\vec r)\sigma^x_{-\hat e_x}(\vec r)\sigma^x_{\hat e_y}(\vec r)\sigma^x_{-\hat e_y}(\vec r)-\mu\sum_{\vec r,j}\sigma_j^z(\vec r) \end{split} \end{equation}$      

My question here is: after choosing a gauge, why do we still have gauge invariance so that the gauge transformation generators act trivially? In this example, the question is that after choosing the unitary gauge $\tau^z(\vec r)=1$, why do we still have $\tau^x(\vec r)\sigma^x_{\hat e_x}(\vec r)\sigma^x_{-\hat e_x}(\vec r)\sigma^x_{\hat e_y}(\vec r)\sigma^x_{-\hat e_y}(\vec r)=1$?