In order to learn or understand introductory QFT you need to understand, vector algebra, vector calculous, complex numbers, the mathematics of operators acting on a Hilbert space, the mathematics of special relativity and some elements of complex analysis, series expansions, Fourier transformations, group theory etc.
I think the best way one to proceed is by physics and not mathematics textbooks. If you understand most topics in non-relativistic quantum mechanics, special relativity and classical mechanics, the QFT textbooks will take you further and complement any lacking skills (usually). In any case, you start learning QFT and when you find a mathematical subtlety you dont understand you stop and cover it. Learning from pure math textbooks, to me, is not the most helpful way to learn QFT. You say you would like to avoid this way, but I think that you would need a really long time by studying only "pre-requisites" and then going into QFT.