How to select which differentiable manifold to use to model spacetime?

Question

I've asked this question on SE as well, but haven't gotten a satisfactory answer. Would be really grateful for your help:

I'm studying differential geometry basics for general relativity. I know that spacetime is modeled as a 4-dimensional smooth manifold. Smooth manifold means that we consider a restriction of the maximal atlas such that all charts in it are compatible. A smooth manifold is specified once we choose an equivalence class of compatible atlases. Among all coordinate charts among all of those atlases (in the equivalence class), the differentiability notion is well-defined - a curve that's differentiable in one coordinate system of one atlas will be differentiable in any other coordinate system of any other atlas in the equivalence class.

This much is clear.

I'm watching a lecture on the same topic and the lecturer discusses about an issue here (the link starts at the relevant timestamp and it's just a 1.5 min watch till 24:00). For the 4-D case, there exist uncountably many smooth atlases up to diffeomorphism.

At 23:25 timestamp, he mentions that the differentiability of curves depends on the atlas we're using. So a curve may be differentiable w.r.t. one atlas and not to another (not in the same equivalence class but another smooth atlas nonetheless).
So this amounts to saying that depending on the choice of smooth atlas equivalence class (which amounts to choosing what differentiable manifold to use), a curve may be differentiable in one choice and not in another.

This makes it seem that the differentiability notion is ill-defined. The lecturer talks about the same issue briefly here as well, but doesn't get around to discussing the resolution. How is this issue resolved?