In mathematics in general, the expression "moduli space" is often used in a quite vague way. It appears when one wants to classify mathematical objects and when the set of objects that one wants to classify is not only a set but is also endowed with some extra "natural" structures, often of geometrical nature. Typically, if one is considering objects depending on parameters, one wants to consider the "space" of all the parameters. In this context, "moduli" is synonym for parameter and a "moduli space" is a "space of parameters".
A "moduli space" should be a geometrical object, the idea being that this geometry probably encodes a lot of information about the objects that one is trying to classify, and that this geometrical object is maybe interesting by itself. In algebraic geometry, a natural geometrical structure is that of algebraic variety. So given a classification problem in algebraic geometry, one can ask: is there an algebraic variety which parametrizes the objects to classify? But what do we mean precisely by that? If one only means that there is a bijection between the points of the variety and the objects to classify, it is an extremely weak condition, which certainly does not specify this algebraic variety in any unique way. Yet, there are many examples, where there is a feeling of what is the precise "moduli space" to consider (elementary examples: projective spaces classifying lines in vector spaces, or more generally grassmannian varieties classifying vector subspaces of a given dimension in a given finite dimensional vector space, already non-elementary example, due to Riemann: the "space" of all smooth projective genus g curves over the complex numbers).
The problem is: how to make the above intuition precise? For example, given a classification problem, what kind of geometrical structure do we expect on the set of objects to classify? How to construct such geometrical structure? In which sense is it "natural"?
One approach could be to say that one should think about the correct notion in each specific case.
The questions makes sense for any kind of classification problem in mathematics. It happens that they are interesting and have been particularly studied in algebraic geometry, which is the context of the question, so I know restrict myself to this case.The remarkable fact, due to Grothendieck, is that, in algebraic geometry, it is possible to give a precise definition of what is a "moduli space". (From now on, when I say "algebraic variety", I should say "scheme"). To do that, one has to realize that an algebraic variety $X$ determines a functor from the category of algebraic varieties to the category of sets: to any algebraic variety $Y$, one associates the set of morphisms of algebraic varieties from $Y$ to $X$. And by general easy categorical arguments (Yoneda's lemma), the variety $X$ is completely determined by this functor. A functor from the category of algebraic varieties to the category of sets which comes in this way from $X$ is said to be represented by $X$. Not every functor is representable, but if one wants to define an algebraic variety $X$, one way is to start with a functor and then try to show that this functor is representable.
What is the relation between this maybe abstract sounding functor story and a classification problem? The point is that a precise way to define a classification problem in algebraic geometry is to give such functor, called the moduli functor, the idea being that to an algebraic variety $Y$, one associates the set of families of objects that one cares about parametrized by $Y$. The precise notion of "family of objects" depends on the specific problem under consideration (for vector bundles or sheaves, the good technical notion is "flat family"). The question of existence of a moduli space is then the one of the representability of this functor: if this functor is represented by an algebraic variety $X$, one calls $X$ the moduli space. Going through the definition, one sees that it has all the expected intuitive properties: in particular, $X$ parametrizes a family of objects and any family of objects parametrized by some $Y$ comes by pullback from a map from $X$ to $Y$, i.e. $X$ is the basis of a universal family of objects.
Remark that the fact that there is a relation between the functorial point of view on algebraic varieties and the problem of moduli spaces is not unexpected: any algebraic variety $X$ is tautologically a moduli space: it is the moduli space of points in $X$.
This points of view was used spectacularly by Grothendieck, eg to show the existence of the "Hilbert schemes", i.e. of algebraic varieties parametrizing subvarieties of a given projective algebraic variety, thus giving a vast "non-linear" generalization of grassmannian varieties.
But what to do if the moduli functor is not representable? One could just give up. Or one could realize that often, there is a natural obstruction to the representability: the fact that the object we want to classify have automorphisms. For example, assume that there is the class of objects that we want to classify is reduced to one object. One would like to say naively that the moduli space is a point. But if this unique object has a non-trivial group of automorphisms, then the point will not satisfy the functoriality properties required to represent the moduli functor. The problem is that in that case, there is no universal object, the problem being that our unique object is not "canonically" defined but only to automorphisms.
Having identified automorphisms has a problem, there is three ways to go. One way is to restrict ourselves to object without automorphisms or to at least kill the automorphisms in some way. For line bundles, the automorphism group is always $\mathbb{C}^{*}$ and rigifying slighlty the moduli functor, one can obtain a representable functor and so a moduli space for line bundles, usually called the Picard variety. For higher rank vector bundles, the situation is worse because the automorphism group can change and can be as big as $GL_n(\mathbb{C})$ (eg for the rank $n$ trivial vector bundle) and a simple rigification does not work. One way to proceed is to introduce a notion of stability (depending on some extra data in general) and to restrict ourselves to stable vector bundles, which happen to have a moduli space.
Another way is to give up on the representabiliy and to ask for an algebraic variety $X$ which is "as close as possible" to represent the moduli functor. The precise version of this idea is the notion of "coarse moduli space". For example, it is possible to construct a coarse moduli space for semistable vector bundles, which is a bigger class than the stable vector bundles.
Finally, one can give up the notion of algebraic variety and look for a more general notion of "space" in algebraic geometry. The correct notion is the notion of "algebraic stack". An algebraic stack is locally something which looks like the quotient of an algebraic variety by the action of an algebraic group. It is the right kind of objects which remember the information about automorphism groups and many moduli functors naturally define an algebraic stack which is then called "moduli stack". For example, it can be shown that there exists moduli stacks classifying vector bundles or coherent sheaves on an algebraic variety.
You often need stacks if you care about "all" objects of a certain type, eg all vector bundles, and then the moduli stack can have behaviour weirder than an algebraic variety. (Exercise: try to draw a "picture" of the stack of rank 2 vector bundles on the complex projective line). You care about coarse moduli spaces if you consider "semistable" objects in a certain sense. You care about moduli spaces (in the strictest sense) if you consider "stable" objects. Remark that these stability notions often match with physical notion of stability (eg "stable versus unstable particle") in some physical contexts.
There is no big differences between vector bundles and coherent sheaves: in each case, there is the trio: algebraic stack for all, coarse moduli for semistable, moduli for stable. The main difference comes from compactness property. Over varieties of dimension >1, moduli spaces of vector bundles are not compact and you need to include more general coherent sheaves to have compact moduli spaces (a physics example of this phenomenon: think about what happens when a 4d instanton shrinks to a point).
Q&A (4871)
