A necessary condition is that your Hamiltonian H (I assume self-adjoint) is such that
(1) it has pure point spectrum,
(2) it has spectrum bounded below,
(3) every eigenspace has finite dimension.
This is because if e−tH (I assume t>0) is trace class, it must be compact and compact operators have bounded spectrum made of proper eigenvalues, with 0 as the only possible accumulation point and only possible point of the continuous spectrum.
If you work in L2(R), Hamiltonians like −d2dx2+V(x) generally have a continuous part unless V diverges as |x|→+∞. In the former case e−tH cannot be trace class, the latter has to by studied case by case.
If, instead, you work in compact regions (with suitable boundary conditions), as in the case you are considering, the spectrum is well behaved (in view of known properties of the heat kernel of the Laplace-Beltrami operator corrected with a regular potential). Thus you only have to check if ∑+∞n=0e−tEn converges. To this end there is the celebrated Weyl's asymptotic formula ED/2n∼Cn,
where
D is the dimension of the manifold
D=1 in our case,
C>0 is a constant and
Ek+1≥Ek.
Thus En∼Cn2 and ∑ne−tEn∼∑ne−tCn2
always converges for
t>0.
The final answer is: Yes e−tH, for t>0, is trace class for the potential cosx in L2([0,2π]) with periodic boundary conditions.
Compactness of the region plays a crucial role here. This is the reason why a quantum gas in a finite box can be treated by means of the standard QM (states are positive trace class operators with unit trace) whereas infinitely extend systems need the algebraic formalism and the KMS condition to be mathematically rigorously handled (if one is not content with the standard thermodynamic limit procedure).