Why do we ignore surface terms in the action in Quantum Field theory ?

Question

I have trouble understanding why do we ignore surface terms. To get the on-shell fields we fix the fields at infinity so these terms do not matter. However , In the path integral , We integrate over all field configurations. So we must integrate over all values of the fields at infinity. So , one shouldn't set terms like $e^{i\oint dn^{\mu}\psi_{\mu}} $ equal to one and they should contribute to the partition function. Is this correct ?

Comments

The only (hyper) surfaces involved are time-like (hyper) surfaces at the ending "points" where the filed is considered known, so no integration over all possible field configurations is done. In other words, the surface integrals are constants (known boundary conditions). IN and OUT fields are considered known, aren't they?

---

In the field theory textbook I use , The path integral is written as $\int D\phi e^{iS}$ So I'm confused. The total derivative terms are evaluated at the boundary of spacetime by gauss theorem. Right ? do we keep the fields at spatial & temporal infinity fixed ? I'm confused by this point. 

---

I also remember reading somewhere that total derivative terms do not contribute to the perturbative series but can have non-perturbative effects. I don't understand what this means but I think it is somehow related.

---

If you remember, the path integral can be introduced in calculation of the amplitude $\langle x(t_1)|x(t_2)\rangle$, $t_2 > t_1$. So the initial and the final "positions" (states) are considered fixed, known. It is written in many textbooks.