Anomalous dimensions and operator mixing can be best explained using the Wilson RGE and operator language (the discussion and notation mostly follows Chapter 23.6.2 of this book). To obtain the Wilson RGE one looks at the infinitesimal change of the action (or Lagrangian) due to integrating over an infinitesimally thin shell $d\Lambda$ of energy/momentum under the constraint that the generating functional (that describes the physics of the system)

\[Z[J] = \int\limits^{\Lambda}\mathcal{D}\phi\exp\left(i\int d^4x\sum\limits_nC_nO_n(\phi)\right)\]

does not changes by this infinitesimal RG transformation

\[\Lambda\frac{d}{d\Lambda}Z[J] = 0\]

This leads to a system of non-linear differential equations for the Wilson coefficients

\[\Lambda\frac{d}{d\Lambda}C_n = \beta_n(\{C_m\},\Lambda)\]

Linearizing this system of equations around a fixed point gives the linear differential equations (also called continuous RGEs)

\[\Lambda\frac{d}{d\Lambda} C_n = \gamma_{nm}C_m\]

where the matrix $\gamma_{mn}$ contains the so-called anomalous dimensions. They describe how the dimension of an operator deviates from its classical dimension $d_n$ (sometimes also called engineering dimensions).

Diagonalizing the matrix of anomalous dimensions $\gamma_{mn}$, the operators can be classified into irrelevant ($\lambda_n > 0$), relevant ($\lambda_n < 0$), and marginal ($\lambda_n = 0$) operators. Following the RG flow towards longer distances, the irrelevant operators lose their importance (they span the basin of attraction of the fixed point considered), the relevant operators grow in importance (they lead away from the fixed point around which the linearization is done), whereas the marginal operator neither grow not increase in first order (they potentially allow for cyclic behavior of the RG flow).

In principle, the classification into relevant, irrelevant, and marginal operators is only valid if the analysis of the fixed point is done in the eigenbasis of the matrix of anomalous dimensions. Otherwise, the non-zero off-diagonal elements lead to operator-mixing. This can lead to situations, where an operator classified as irrelevant by its classical scaling dimensions becomes important for the IR behavior of the system due to getting mixed with other relevant and marginal operators.

Also, if an eigenvalue of the anomalous dimension matrix has multiplicity >1, diagonalization may be impossible and the operators of the corresponding block remain mixed even after block-diagonalization.