Suppose for simplicity we have an abelian gauge theory. We want to quantize it in terms of 4-potential $A_{\mu}$.
If we impose the covariant gauge (like $\partial_{\mu}A^{\mu} = 0$) on the physical state $|\Psi\rangle$, namely
$$\langle\Psi_{1}|\partial_{\mu}A^{\mu}|\Psi_{2}\rangle = 0,$$
then it turns out that the physical state contains an equal number of time-like and longitudinal photons; they are associated with creation-destruction operators attached to $A_{0}, \mathbf{A}(p)\cdot\mathbf p$ respectively, and the former (time-like) has negative norm. But the latter (longitudinal) provides that the norm of the total state, with equal number of time-like and longitudinal photons, is zero. In the literature this procedure of the quantization is called the Gupta-Bleuler quantization.
In this gauge the relation between unitarity and gauge invariance is thus obvious: by breaking the gauge invariance we thus break the unitarity, since the negative norm states enter the game.
Suppose now we want to quantize the theory in a gauge $A_{0}= 0$. There is a remaining set of time independent gauge transformations related to longitudinal degrees of freedom, for which the Gauss constraint
$$
G(x) = \nabla \cdot \mathbf{E} - \rho
$$
is a Noether current. Precisely, the hamiltonian $H$ of the theory commutes with $G$. To eliminate such degrees from the physical state, we have to require
$$
\tag 1 G(x)|\Psi\rangle = 0
$$
The breaking of gauge invariance in general means that $(1)$ is violated with time.
My question is following. In the gauge $A_{0} = 0$ I don't see why the presence of longitudinal photons causes the violation of the unitarity in this gauge. Precisely, in this gauge the situation is similar to the massive QED, where longitudinal degree of freedom is present, as I think. Could you clarify where the unitarity violation is hidden in the $A_{0}=0$ gauge?