To describe anyons, it is said that category theory is useful. In my understanding, a category suitable for describing anyons is a modular tensor category, which has a rigid structure. Physically, the rigid structure means that there exists a conjugate type of anyons a* for each type a, which yields a trivial charge when a and a* are fused.
However, I wonder why there should be a conjugate type for each type. What is the physical reasoning or observation for this requirement?