Actually even the quantum fields support ("are") representations of Poincaré group acting on a suitable vector bundle whose sections are, in fact, the considered fields:
$$\phi^A(x) \to S^A_B(\Lambda) \phi^B (\Lambda x + t)$$
where $(\Lambda, t)$ is the generic element of the Poincaré group. Notice that $S$ sees only the Lorentz part of the Poincaré group and defines a vector representation on its own right.
When you fix an event in spacetime, i.e., you deal with a fiber of the vector bundle only, the Lorentz group part of the semidirect product you wrote acts on that fiber, by means of the representation
$$\phi^A(x) \to S^A_B(\Lambda) \phi^B (x)$$
leaving the fiber fixed. $S$ is a finite dimensional representation of the Lorentz group, since the fiber has finite dimension as a vector space (the range of the index $A$ if finite). However this fiber does not admits a Hilbert space structure invariant under that representation, for this reason the representation is not unitary and it can be finite dimensional.
It is fundamental to notice that, in QFT, $\phi^A$ is also an operator in the Hilbert space of the theory, and Poincaré group is a continuous symmetry of the physical system: It leaves the transition probabilities invariant. Essentially due to Wigner theorem, one has that this symmetry can be implemented unitarily in the Hilbert space of the theory by means of a (strongly) continuous unitary representation $U_{(\Lambda, t)}$.
It is natural tu assume that, under this unitary representation of Poincaré group acting in the Hilbert space, the quantum fields viewed as operators, transform covariantly with respect the other representation acting in the spacetime:
$$U_{(\Lambda,t)} \phi^A(x) U_{(\Lambda,t)}^\dagger = S^A_B(\Lambda) \phi^B (\Lambda x + t)\:.$$