A unitary representation of a group G is a mapping U from G to the algebra of unitary linear operators on a Hilbert space V. This space is called the representation space, and one says that G acts unitarily (or is unitarily represented) on V. Thus for every g∈G, the representation defines a mapping ψ→U(g)ψ with the natural compatibility properties. In particular, unit vectors are mapped to unit vectors. In quantum mechanics, the elements of V (or only the unit vectors, depending on the author) are referred to as the states, and the thing that acts on the states is the group element.
Saying that states of a certain form form a representation is loose talk for saying that all states of this certain form form a Hilbert space (with inner product taken from the context) on which G acts unitarily (in the obvious way, or in the way defined by the context).
Finding a representation means finding a Hilbert space V and the action of G on it. Typically, one pieces V together from constituents already known. This defines the states of interest. When the construction is elegant, the notation used for the states is such that the group action is obvious; otherwise the group action has to be defined explicitly and one must prove that products behave correctly.
Matrices are just the special case when V is the Hilbert space of complex column vectors. The representation of states as column vectors is appropriate for an N-level system and the unitary group U(n), but for other groups it is usually preferable to use a different notation for the states that is adapted to the group.