At the non-rigorous/intuitive level, OP's observations are spot on. To facilitate such thinking, physicists often use DeWitt's condensed notation, where a field $\phi^{\alpha}(x)$
is written as $\phi^{i}$, while pretending that $i=(\alpha,x)$ is an index of a local coordinate $\phi^{i}$ for some differential manifold.
The problem is that the space of all field configurations is typically an infinite-dimensional space, while ordinary differential geometry is usually only discussed in the context of finite-dimensional manifolds.
Thus strictly speaking, one would have to master/study/develop an infinite-dimensional mathematical version of differential geometry to make OP's picture precise/rigorous.
This post imported from StackExchange Physics at 2014-03-28 17:12 (UCT), posted by SE-user Qmechanic