I actually dealt this model some in my thesis work. (Shameless plug: Stability of spherically symmetric solutions in modified theories of gravity.) The basic picture in the Dolgov-Kawasaki paper is that they start with a background solution where the Ricci scalar is equal to the trace of the stress-energy tensor, R0=T (inside a star, say). This solution should match up to some exterior solution with R0≈μ4 (one of the cosmological models that were motivating Carroll, Duvvuri, Trodden, & Turner); however, if μ4<0, this turns out to be impossible. (Basically, the Ricci scalar tends to diverge at large distances from the star.)
Having dispensed with the μ4<0 case, they look at at the μ4>0 case. In this case, one can obtain a well-behaved solution in which R0≈T inside the star and R0≈μ4 asymptotically. The next step is to look at the behavior of perturbations about this solution; the linearized perturbation equation is eq. (5) in their paper. The equation of motion for the first-order perturbations grows exponentially with time, since the last term on the left-hand side of eq. (5) is so large in magnitude and negative.
I should mention that while Dolgov & Kawasaki's proof isn't iron-clad (see my comments at the start of Section III.B in the paper I linked to above), it does persist in a more rigorous analysis. You might also take a look at The Large Scale Structure of f(R) Gravity, which obtains a similar result.