What is meant by fractional scaling dimension is exactly what is says: Given a dilatation $x\mapsto\lambda x$, the field/operator $\mathcal{O}(x)$ behaves as
$$ \mathcal{O}(\lambda x) = \lambda^h\mathcal{O}(x)$$
with $h\in\mathbb{R}$ a possibly fractional or even irrational number.
The prime example of quantum field theories in which a fractional scaling dimension appears is for conformal field theories, which are always scale invariant because the scaling is just on of the conformal transformations. It is a bit unusual for an interesting theory to be scale and not conformally invariant, actually. What the author means with "scaling by fractional dimensions" is simply that quantum theories need not have integer $h$. Here's a "simple" example:
Consider a 2D theory of a Majorana fermion $\psi$ on the cylinder $\Sigma = S^1 \times \mathbb{R}$. The action is
$$ S[\psi] = \int_\Sigma \bar\Psi\gamma^\mu\partial_\mu\Psi$$
with $\Psi = (\psi \;\bar \psi)^T$. There is a conformal map to the complex plane such that
$$ S[\psi] = \int_\mathbb{C} \psi\bar\partial\psi + \bar\psi\partial\bar\psi$$
where the integration measure is in both cases already chosen invariant under conformal (and other) transformations. This theory is scale invariant iff under $z\mapsto\lambda z$,$\bar z \mapsto \bar\lambda\bar z$1 the fields behave as
$$ \psi(\lambda z,\bar\lambda\bar z) = \lambda^{1/2}\psi(z,\bar z)\text{ and } \bar\psi(\lambda z,\bar\lambda\bar z) = \bar\lambda^{1/2}\psi(z,\bar z)$$
where $\frac{1}{2}$ is clearly a fractional "scaling dimension". In the full quantum analysis, however, it turns out that there is a third independent state (which, by the state-field correspondence of CFTs, means there is a third independent field) that has scaling dimension $\frac{1}{16}$. This is essentially due to the possibility to choose anti-periodic boundary conditions for the spinor fields.
1It is an annoying convention to write $\bar\lambda$ for the factor of the second dilatation although it is not the complex conjugate of $\lambda$, just as $\bar h$ is not the complex conjugate of $h$ in the following
This post imported from StackExchange Physics at 2015-05-31 13:12 (UTC), posted by SE-user ACuriousMind