What are some critiques of Jaynes' approach to statistical mechanics?

Question

Suggested here: What are the justifying foundations of statistical mechanics without appealing to the ergodic hypothesis?

I was wondering about good critiques of Jaynes' approach to statistical mechanics. @Yvan did a good job in pointing out a couple of them, but I would like to have them fleshed out a bit, especially by someone who is not biased towards Jaynes.

As I think Jaynes' thoughts on this matter shifted a bit through the years, let me summarise what I think his position is:

Logically, one must start with two things:

1. There is a microscopic theory of the phenomenon under consideration --- for the moment that can be the existence of a (quantum or classical) hamiltonian formulation, which then ensures the existence of a preferred Louiville form. Thus it makes sense to discuss probabilities of trajectories (ensemble), independently of where those probabilities come from.

2. There exists a viable macroscopic, coarse-grained description, which is only the case if an experiment says so --- the key is what Jaynes sometimes calls "reproducibility". If a phenomenon is not readily reproduced then clearly one has not gained sufficient control over enough variables --- e.g. it was an experimental fact that controlling temperature and volume of a gas was sufficient to determine its pressure.

Then it is logically true that one could develop a quantitative theory/relationship of the macroscopic degrees of freedom or observables, and the claim is that one should set up an ensemble over the microscopic trajectories subject to the constraints of the macroscopic observations and a unique one is chosen by maximising entropy.

With the ensemble in hand, one could then proceed to make predictions about other observables, failure to then observe them means either your microscopic theory is wrong, or the set of variables chosen is not correct, and the circle of Science is complete.

Importantly this makes no reference to ergodicity, and in fact this works out of equilibrium --- equilibrated systems just tends to help with experimental reproducibility. Personally I see it as morally dimensional analysis, writ large.

Yvan pointed out that there is a problem with classical hamiltonian systems because they have an (uncountable) underlying configuration space, and there are technical problems with defining entropy on them. My opinion is that this does not matter, because all physical hamiltonians are really quantum, and those come with a canonical choice. (Field theories need regularisation, both UV and IR as always, to render the number of degrees of freedom finite to be "physical".) However, I'm probably just being naive and I would certainly welcome education on this.

My preferred reference (never published, I think): Where do we stand on maximum entropy?

This post has been migrated from (A51.SE)

Comments

+1 for the link to Jaynes's article. I didn't know it and it is wonderfully written. I'm enjoying it tremendously. Thanks!

This post has been migrated from (A51.SE)

---

@JoséFigueroa-O'Farrill: I personally find Jaynes' continuous diatribe against detractors of his brand of probability theory annoying; for instance here it seriously dilutes what is otherwise a great set of applications of the methodology, which I believe stands on its own without the abstract dialectic. I can strongly recommend his suggestion of going through the exercise of using the Gibbs Algorithm on the precessing spin --- it really is informative about where the various bits of physics enter into the final result.

This post has been migrated from (A51.SE)

---

I'd really like to draw attention to his derivation of Kubo relations --- it is surely the best I've ever seen in any textbook, and in fact generalises to the high order, non-linear elements!

This post has been migrated from (A51.SE)

---

I don't mind the diatribe, which is not to say that I agree with Jaynes. I just like what I've read so far about his article on the maximum entropy. He clearly thought deeply about these issues and has an engaging style, his opinions notwithstanding.

This post has been migrated from (A51.SE)

---

Relevant Jaynes quote 1: "We are not puzzled by 'irreversibility' because (one of those important results which has been in our equations for over a Century, but is still invisible to some), given the present macrostate, the overwhelming majority of all possible microstates lead, via just those evil, deterministic mechanical equations of motion, to the same macroscopic behavior; just the reproducible behavior that we observe in the laboratory. So what else is there to explain? There would be a major mystery to be explained if this behavior were not observed."

This post has been migrated from (A51.SE)

---

Relevant Jaynes quote 2: "Recalling that ergodic theorems, or hypotheses, had been actively discussed by other writers for over thirty years [prior to the publication in 1903 of Gibbs’ Statistical Mechanics], and recalling Gibbs’ extremely meticulous attention to detail, I think the only possible conclusion we can draw is that Gibbs simply did not consider ergodicity as relevant to the foundations of the subject. Of course, he was far too polite a man to say so openly."

This post has been migrated from (A51.SE)

---

@José Figueroa-O'Farrill : my favourite Jaynes' paper is [this one](http://bayes.wustl.edu/etj/articles/gibbs.paradox.pdf), which I think should be compulsory reading to all stat.mech students, and most writers of textbooks on this subject. It is not directly related to the topic of this question, but it cannot hurt to make some proselytism. He gives the clearest (and only correct) discussion of the "mixing paradox", showing that it has nothing to do with indistinguishability of particles in quantum mechanics (the usual, but wrong argument).

This post has been migrated from (A51.SE)

---

@genneth: I think that the reference you give has been published in [Maximum Entropy Formalism](http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=6188), R. D. Levine and M. T. (Eds), The MIT Press (1978).

This post has been migrated from (A51.SE)

Answer 1

I think that, for many physicists, one major criticism of this approach is the feeling, justified by the previous successes of reductionism, that statistical mechanics ought to be derivable logically from the underlying microscopic theory (say, classical or quantum mechanics) . Of course, this is impossible, at least without additional assumptions, since the latter describes the evolution of the system but does not tell you anything about the initial conditions...

One main strength (and limitation) of Jaynes' approach is that it applies regardless of the underlying dynamics (at least as a foundation of equilibrium statistical mechanics). The cost is that, seen from this point of view, equilibirum statistical mechanics is not a fundamental theory of physics, but "only" a special case of statistical inference (in particular, it describes our knowledge about a system and not the system itself).

This post has been migrated from (A51.SE)

Comments

I really don't get this: "...statistical mechanics ought to be derivable logically from the underlying microscopic theory...". The way I understand it, statistical mechanics is the framework by which we may hope to derive thermodynamics from the underlying microscopic theory. Or am I missing something?

This post has been migrated from (A51.SE)

---

@JoséFigueroa-O'Farrill : The only information one retains from the micrscopic theory in Jaynes' approach is the Hamiltonian, basically. In his approach, the precise form of the microscopic dynamics (say, Hamiltonian evolution) does not play any role. In this sense, he does not derive Statistical mechanics from the apriori more fundamental microscopic theory. [to be continued]

This post has been migrated from (A51.SE)

---

[part 2] In particular, the interpretation of probabilities in Jaynes' approach is as a description of the state of knowledge of the observer. One might, in principle, hope to give a more mechanical meaning to these probabilities (that's exactly one the other approaches, based on mixing properties of the dynamics) are trying to achieve. [To be continued]

This post has been migrated from (A51.SE)

---

[part 3] As I said above, this can't work without some further assumptions, because all provable statements will require taking initial conditions outside a set of measure zero w.r.t., say, Liouville measure. But there is no reason a priori to define typicality in terms of the Liouville measure (and this can't follow from the Hamiltonian evolution, as the latter has nothing to say about initial conditions).

This post has been migrated from (A51.SE)

Answer 2

The MAXENT approach works perfectly well, but with the caveat that it can only be justified when the statistical evolution (e.g., convergence to equilibrium) is deterministic. However this is not much of a restriction: we can’t very well do much predictive physics at all without recourse to at least statistically deterministic dynamics. On the other hand, given the recipe for statistical evolution of a system, the question of whether or not MAXENT methods are necessary at all arises, as Grad said:

[MAXENT] has not yet been connected in any way with dynamics. It can only be looked upon as an ad hoc recipe whose accuracy must be empirically determined. Either [the MAXENT] hypothesis is correct, in which case it is unnecessary, or it is incorrect and should not be used.

[Grad, H. “Levels of Description in Statistical Mechanics and Thermodynamics”. In Delaware Seminar in the Foundations of Physics, Bunge, M., ed. Spinger, New York (1967).]

The empirical determination that Grad refers to is basically identical to determining the relevant macrovariables and some idea of their dynamics. In practice these are usually unknown, and the MAXENT hypothesis is technically incorrect, but a good implementation of the principle will successfully approximate the set of relevant macrovariables and their dynamics based on some reasonable Ansatz. So we amend Grad’s objection that MAXENT “is incorrect and should not be used” to “it is only an approximation of the actual dynamics in and should only be used with care”.

This post has been migrated from (A51.SE)

Comments

+1; though as a proponent of the method I feel that its inability to be magic is not that serious...

This post has been migrated from (A51.SE)

---

I am actually a fan of Jaynes' approach to statistical physics, and hold the man's work in high esteem. But I am much less enamored of the whole probability theory catfight (I would like to understand it better).

This post has been migrated from (A51.SE)

Answer 3

I would recommend anything by Cosma Shalizi on this subject, e.g. as a starting point:

Bayes, Anti-Bayes (blog posts)

Maximum Entropy Methods

This post has been migrated from (A51.SE)

Comments

As far as I can tell, he is mostly interested in the issue of Bayesian statistics (i.e. something I don't really understand) vs. something I really don't understand. The only one which makes direct reference to stat. mech is [this](http://cscs.umich.edu/~crshalizi/weblog/270.html). His point about boiling water by becoming ignorant is a bit bizarre, and I'm unsure how to respond in the same language. I cannot speak to his credentials on statistics, but I hope it is clear that his knowledge of physics is not sufficient to be considered a serious commentator.

This post has been migrated from (A51.SE)

---

To put it into context, I feel that Jaynes is a physicist first (he did real physics with quantum optical systems), and I actually feel that his work on statistics is almost a blemish. Perhaps I have simply benefited from his proselytising without knowing the exact intellectual environment in which it was done. But in any case, I want to know what's wrong with the approach as physics, not as statistics --- the key difference is the _experimental_ reproducibility.

This post has been migrated from (A51.SE)

---

genneth, Cosma Shalizi's post you link includes a link to his arxiv paper: http://arxiv.org/abs/cond-mat/0410063. This contains a clear statement of his result and section IIB in particular contains arguments about how applying a "maximum entropy" procedure doesn't reproduce the expected results of stat mech. Although he's now a statistician and not a physicist, he does have a PhD in physics and it looks like a serious argument to me. (I haven't read enough Jaynes to say how closely Shalizi's assumptions model Jaynes's ideas, but I think you should at least give his arxiv paper a close read.)

This post has been migrated from (A51.SE)

---

@Matt Reece : But apparently, Section IIB is only a criticism of Jaynes' elementary approach to the second principle of thermodynamics. It does not seem to give any argument against MaxEnt as a logical foundation for equilibrium statistical mechanics.

This post has been migrated from (A51.SE)

---

@Matt: I did read it, but concluded that it failed to appreciate the reproducibility criterion. If some relevant macroscopic observable is not already predicted then the experiment is simply not fully reproducible; i.e. I think a correct application of the method would show a constant entropy.

This post has been migrated from (A51.SE)

---

@Yvan: I think the max ent approach is more generally applicable than just equilibrium. Thus I think the question is a good one, but the answer to be satisfactory.

This post has been migrated from (A51.SE)

---

@genneth: Yes, but as a satisfactory theoretical framework for "non-equilibrium statistical mechanics" does not exist at the moment, discussing its foundational issues seems less important ;) . (I am not talking about linear response theory and such, obviously.)

This post has been migrated from (A51.SE)

Answer 4

The main criticism: The maximum entropy principle works (i.e., gives a correct description of a physically system) if and only if

the knowledge of the observer is of a very special kind, namely consisting precisely of the expectation values of at least those extensive quantities that are important for a thermodynamic description of the system in question, and the prior is chosen correctly, consistent with the well-known principles of statistical mechanics.

If one gets the prior wrong (e.g., forgets correct Boltzmann counting), the entropy of mixing doesn't come out correctly, eveen though everything else is done as usual.

If one gets the set of macroobservables wrong - e.g., $H^2$ in place of $H$, or only the total energy when in fact a spatially distributed energy distribution is required for an adequate (nonequilibrium) description - then one gets a meaningless theory inconsistent with observation.

Thus, essentially, Jaynes uses as input what should be a result - namely the correct set of relevant variables, and the correct prior to use. It is a ''derivation'' presupposing the facts, and indeed it was presented only almost a century after the birth of statistical mechanics.

---

I discuss the shortcomings of Jaynes' approach to statistical mechanics (and a number of related problems)

in Section 10.7 of my book Classical and Quantum Mechanics via Lie algebras,

and in various articles in my

theoretical physics FAQ:
• (in Chapter A3)
  What about the subjective interpretation of probabilities?
  Incomplete knowledge and statistics
  Entropy and knowledge
  The role of the ergodic hypothesis

• (in Chapter A6)
  Entropy and missing information
  Ignorance in statistical mechanics
  

This post has been migrated from (A51.SE)