Once you uniquely fix the gauge for the gauge field, any two mathematically distinct configurations of the matter field are also physically distinct, they aren't gauge related. Changing a gauge changes both matter and gauge field, so that making sure that the gauge field is constrained to a surface which intersects each gauge orbit once and only once implies that any matter field configuration is different from any other, because you no longer have freedom to change the gauge.
What that means is that you can fix the gauge using a constraint on the gauge field or a constraint on the matter field, but not both. The thing that is going on can be explained using coordinate transformations. If you have three points in the plane, you can describe their relative configuration using one of the points as an origin, and another point on the x-axis, then the location of the third point is arbitrary pair of real number cartesian coordinates, and each mathematically different pair of points is a different configuration (ignoring reflection symmetry). You could also fix the coordinates by putting the third particle on the y-axis, but then you can't simultaneously insist that the second particle is on the x-axis.
When you fix the using a condition on the matter field, there is an issue, at least for linear representations: a zero matter field doesn't transform under gauge transformations at all, so the usual vacuum for matter fields doesn't work to fix a gauge! But in the case where there is a nonzero VEV, the VEV can be used to fix a gauge, the unique gauge where it is real and pointing in one particular direction in the representation space.
When you choose this matter gauge fixing, any two mathematical configurations of the gauge field are now physically distinct, and the action is now a physical action path-integrated over all the configurations of $A$. The matter field kinetic term contributes a part quadratic in $A^2$, and this action describes a collection of massive vector mesons interacting according to the gauge interaction, in other words, because of the VEV, you have a Higgs mechanism. This is the usual treatment of the Higgs mechanism, using the matter field to define the gauge condition.