A number of questions on this topic already exist. The most detailed one is probably this (http://www.physicsoverflow.org//26751/physics-of-asymptotic-series-and-resummation?show=26751#q26751). However, I still find some things confusing.
Assume that we have a (Gevrey-1) divergent series $\sum_n a_n g^n$ intended to represent a physical quantity. Naturally, we think that the full exact answer must be some function $f(g)$ with this series as an asymptotic expansion $f(g)\sim \sum_na_ng^n$. However, such a function is not unique. For instance $\tilde{f}(g)=f(g)+e^{-1/g}$ has the same asymptotic expansion at $g=0$.
Now, the Borel summation is a method to obtain a particular example of such function $\mathcal{B}f(g)\sim \sum_na_ng^n$. There are cases when the Borel resummation is ambiguous (due to singularities of the Borel transform on the real axis). But let us assume that in the case of interest there is no such problems and the Borel summation gives unique non-ambiguous result $\mathcal{B}f(g)$. In all the physical literature that I've been through it was implicitly assumed that when we are in such in a situation, i.e. when the Borel summation exists and is unique, it is the exact quantity that we are looking for.
My question is: why is it so? Why are we supposed to prefer $\mathcal{B}f(g)$ over, say, $\mathcal{B}f(g)+e^{-1/g}$ when computing 'physical quantities'?
Maybe it is worth having some particular example. Consider integral
$$I(g)=\int_{0}^{\infty}dx\, e^{-x-gx^2}$$
which has the following asymptotic expansion
$$I(g)\sim \sum_{n=0}^\infty g^n \int_0^\infty dx\, e^{-x}\frac{(-1)^nx^{2n}}{n!}=\sum_{n=0}^\infty g^n \frac{(-1)^n(2n)!}{n!}$$
The series is sign-alternating and hence the Borel transform has no singularities on $\mathbb{R}_+$. In this case, the Borel sum $\mathcal{B}I(g)$ is indeed equal to the original integral. Written explicitly, this identity is
$$\int_0^\infty dx\, e^{-x} \sum_{n=0}^\infty g^n \frac{(-1)^nx^{2n}}{n!}=\int_0^\infty dx\, e^{-x} \sum_{n=0}^\infty g^n \frac{(-1)^nx^{n}(2n)!}{(n!)^2}$$
Looking at this particular example I have no temptation to add the non-perturbative term $e^{-1/g}$ as it clearly does not belong here. However, my impression is that this situation is general, relevant for the most 'physical quantities'. The only general argument that I can think of is that any 'physical quantity' can be represented as a some (path) integral for which our example is a toy model. Then the reasoning could be somehow generalized I suppose. However this argumentation seems too vague for me.