I have now scanned a little through Wolfgang Bertram's articles. I may need more time to dig deeper.
From the acknowledgements I see that Bertram has been in contact with Anders Kock, who, following William Lawvere's ideas mentioned above, is the one who had developed synthetic differential geometry (SDG, which is what the formal- and super-smooth sets that I am using in the QFT notes are models of) to a comprehensive textbook theory of differential geometry. Kock has long amplified the groupoid and higher cubical groupoid structure of infinitesimal neighbour points in SDG (see his textbooks or specifically his article "Infinitesimal cubical structure and higher connections" arXiv:0705.4406). On the other hand, these groupoid structures of infinitesimals are also long familiar in algebraic geometry, where Grothendieck called them crystals and which these days are maybe most popular under the name "de Rham stacks". I am saying this because from my superficial scan of Bertram's articles, I am getting the impression that it is these structures that he is after. Currently however I am not seeing clearly what he claims to be adding to these well-established (if maybe not widely enough known) constructions.
In conclusion, I dare to say (subject to possibly having missed something while scanning the articles) that the structures that Bertram is describing are all well reflected in synthetic differential geometry in general, and in the model of "formal smooth sets" that I am referring to in the QFT notes, in particular.
The point amplified at the beginning of these notes is that in fact (formal) smooth sets accomplish a bit more than just providing explicit infinitesimals that serve to streamline PDE theory. Namely another important aspect is that they form a topos (the "Cahiers topos") which means that they provide well-behaved smooth mapping spaces and their variant: smooth spaces of smooth sections. (Bertram hints at wanting to proceed in this direction of "cartesian closed categories", such as toposes, in the outlook section of his article above. ) This is a key requisite for properly dealing with the spaces of field histories in physics, as my notes mean to make clear.
Often authors in mathematical QFT who are ambitious to be more sophisticated than the traditional informal discussion invoke the theory of infinite-dimensional manifolds (such as Frechet manifolds etc.) to deal with spaces of field histories properly. This is however on the one hand very technical. On the other hand the "functorial geometry" perspective shows that much of this technicality is not actually necessary: the incarnation of these spaces of field histories as smooth sets (sheaves on the site of smooth test spaces) is first of all much simpler, second it exists even if the infinite-dimensional manifold structure does not (such as when the base space(time) is not compact) and at the same time it does account for everything that one actually wants to do with these spaces, notably it provides for a good theory of differential forms on these spaces (which allows for instance to make easy rigorous sense of things like the de Rham differential of the action functional, in order to rigorously prove the pinciple of extremal action) .
So the existence of good smooth mapping spaces in "smooth sets" toghether with the infinitesimal aspects in "formal smooth sets" are both important in application to field theory. Once we pass to the construction of the covariant phase space, their combination becomes crucial. Here we need to consider the smooth space of sections over an infinitesimal neighbourhood of a Cauchy surface.
Traditional differential calculus is hard-pressed to really deal with such constructions. In "formal smooth sets" it is easy and straightforward. In fact formal smooth sets give a formalization to the way that physicists like to handle these spaces anyway: by local coordinate models. This is really what Grothendieck's "functorial geometry" perspective is saying: the physicist's intuitive way of dealing with generalized geometries (infinite-dimensional, super, etc.) has a good rigorous underpinning.
There would be more to say, but I should stop now. I have written an exposition of these things in various places, see for instance Higher Prequantum Geometry (arXiv:1601.05956).