Is the current scoring scheme for reviews appropriate?

Question

It seems that if the accuracy is negative, then the more positive the originality, the lower total score there will be. I'm not sure how "originality" is interpreted by most, but to me a positive originality has a positive connotation. For example, if I give a paper one upvote on originality and one downvote on accuracy, I would be thinking: "interesting idea, too bad it's wrong." It deserves a better score than the case of "bad/cliche/crazy idea, and totally wrong/not even wrong", in which case I would give one downvote on both originality and accuracy, corresponding to a less negative score however.

What do you guys think? 

For reference, here is our current formula of calculating the total score: http://www.physicsoverflow.org/faq#a4807

Comments

It's supposed to work that way, original and inaccurate means wild speculation without mathematical basis/with faulty mathematical basis, commonly associated with the term "crackpot" : )

Answer 1

Hm as I underststand it, far reaching new claims (high originality score), are no good if they can not be backed up by a corresponding accurate (at best mathematical) justification and reasoning.

For example the ideas in the first ER/EPR paper are very original and they are soundly backed up by already accepted mathematical/theoretical knowledge. So I voted (+1,+1) on that one based on Lumo's review.

But far reaching claims, for example to have found a new ToE or being able to generally dispense with the concept of renormaliztion, may be original, but if the argument or calculations the author applies are weak and out of thin air or even worse, contradict established theoretical / experimental knowledge, the final score should IMHO be negative for such  (+1,-1) papers

So in my opinion, the formula behaves as it should concerning papers with positive originality score, as far reaching claims or new ideas (originality +1) are only rated positive if they are soundly backed up and do not contradict established knowledge. To me this seems a good way to punish crackpots, wannabe new Einsteins, crazy surfer dudes, etc who want to overthrow establised knowledge for no good reason for example.

But I noted a strange (possibly wrong behavior) of the formula in another situation. I will further investigate this and then mention it in this thread too....

Comments

" To me this seems a good way to punish crackpots, wannabe new Einsteins, crazy surfer dudes, etc who want to overthrow establised knowledge for no good reason for example." I doubt if such people can receive any upvote on "originality" here.

---

Yes, they should not. And in such cases we should in my personal opinion not too much worry about censorship, as PhysicsOverflow is meant to be a serious physics site.

Maybe the best way to answer this question would be to systematically investigate the behavior of the formula and find out if there are places in the 2d parameter space (originality and accuracy) where it does not behave is ist should. Not sure, if Dimension10 and Ron have already done this and I am the only one who can not do it in my head without using Matlab for example ... ;-P

---

In my case "the accuracy factor "made my score negative, not because I made a mistake, but because my proofs and conclusions support the Dirac's opinion on the reason of renormalization success (a fluke). People do not tolerate such things anymore. In my case it turned out to be a "factor of tolerance".

---

@JiaYiyang They should! I upvoted originality on Vladimir's paper but downvoted the accuracy. If we assume that people vote sensibly, and not downvote everything because they hate the paper, or such, then crackpot works will get punished as they should.

---

@Dilaton I notice a strange problem for unoriginal, accurate papers... Is this the same strange behaviour you encounter?

---

@dimension10, I think we really interpret "originality" differently, it could be just semantics but the word "original" has a strongly positive connotation to me, such feeling makes it very hard for me to vote up something like, proposing a theory based on, say Santa Clause, with almost zero mathematics and claim it's physics. For the record I did not vote up or down Vladimir's paper.

---

The formula behaved rather reasonalble where I thought it would break down...

What is the issue with unoriginal ut accurate papers? They should be rather positively received, as not every paper has to introduce a completely new point of view or framework to positively contribute to research. In fact, most good papers do rather incremental progress, as Lumo often says too.
 

---

@Dilaton To me it seems that downvoting for originality indicates plagiarism or something similar.

---

I don't understand why someone would downvote a paper with accurate calculations. The question of "accuracy" was, I thought, meant to indicate "accuracy within the assumptions of the paper", not "accuracy in describing nature" necessarily, or "concordance with other literature", or anything political. It is to assess whether the calculations or results are soundly obtained. The question of agreement with experiment is separate.

---

I thought for a paper to be "accurate", the calculations have to be in accordance with the assumptions and with the conclusions/claims issued too? I mean presenting a good calculation that is valid in certain situations and under specific assumptions and then issue far reaching claims based on this (that are not justified by the derivation in the paper) is not good (or not "accurate")?

---

If you have a beef with the conclusions, then comment on the paper. Getting hung up on the wording of introductions and conclusions is a total waste of time, it's usually arguing over philosophy. If the deductions are accurate, upvote accuracy, and originality, but if there are overreaching conclusions from these accurate deductions, just say so in your review, and your review will get upvoted. The voting and review system is designed to complement the literature, not enforce political voting.

I don't think it is a good idea to restrict the conclusions people are allowed to come to by politics. If Schwinger calculates the magnetic moment of the electron, and then says "this calculations shows that fundamentally, electrons really love the number pi, and they made friends with Archimedes", who cares? Just upvote the calculation, upvote the originality, and say "Schwinger is totally off base about the conclusions at the end there" in the content of the review, problem solved. A positive upvote can come with a review critical of the conclusions, the two activities, voting and commenting, are separate acts unlike in a journal refereeing process.

---

This has nothing to do with policy, censorship, or whatelse ...

Throwing out too far overreaching claims (not to confuse with hypotheses, or interesting further issues to study) that are not backed up by the beaf of the paper (or other acceptable sources) seems unscientific to me.

Did I get this wrong when thinking that the votes should be accompanied by a (partial) review or at least a comment, which can then be upvoted by other people who agree with the point stated, etc ...?

---

If you guys are talking about my Toy model, I want to add two obvious things. First, my model is easily promoted to QM by inserting its Lagrangians into the path integral. Next, my finite corrections correspond (to a great extent) to finite corrections in QFT with cut-off. So my conclusions, although obtained (on purpose) with a classical mechanics problem, have their akin counterparts in QFT.

---

The problem they have is not with the model itself, but with the conclusions that you draw from it regarding other people's calculation methods. The quesiton is whether these conclusions, which are in this case inflammatory and really unjustified, mean that the paper deserves a downvote on accuracy. I don't think so, I think that people are allowed to express their opions, the work's technical accuracy in the model is what should matter for a review. I would upvote your paper on accuracy, except I haven't gone through it in detail yet.

---

Thanks, Ron. I would love to listen what is inflammatory and unjustified too. Will be waiting for your review. Take your time, though.

Answer 2

Based on the answers by Dilaton and Ron, I think we do need a instruction message in every review post on what originality and accuracy mean. In fact, my definition of "original" disagrees with Dilaton and dimension10's, and my definition of "accurate" disagrees with Ron's. Once the instruction is ready, I will follow what the instruction says. 

Comments

Maybe we should also say there that people are free to decide about their votes as they see fit from their best knowledge and judgement, as any votes can be explained in reviews or comments as needed. We should avoid the impression that we want to patronize the votes of our members by bureaucracy and policies, as it can be observed on other physics Q&A sites. So I think you do not necessarily have to wait for official definitions before voting in the reviews section, if you feel familiar enough with the topic and paper at hand ... ;-)

---

Well, I think it's important to have a common definition of what the votes mean, because they will be aggregated. People can happily disagree on whether a paper is "original" or not, but the meaning of "original" should be the same for everyone.

---

I think the point is that it might be best though to leave it nebulous until the community congeals on a common definition. For me, the definition of "original" is the degree to which a paper is unrelated to other work, with completely new papers with no relation to previous work scoring very high, papers extending previous work scoring medium, and plagiarized or duplicative papers scoring negatively. The "correctness" to me means that the calculations, experiments, deductions, are accurate within their context. A false deduction leading to an overbroad conclusion is also a failure of accuracy, but not as bad as totally made-up equations with totally false calculations.

Answer 3

I don't think that a good idea that's wrong deserves a downvote on accuracy, so long as the predictions of the idea are accurately worked out, and honestly compared with experiment. If a theoretical paper presents an original idea, with accurate calculations, it should be upvoted on both counts, even if it disagrees with experiment.

The point of these categories is to make sure that people vote on accuracy, not on politics. If a paper is original, accurate, and disagrees with experiment (for example, steady state model, or Brans-Dicke theory), this paper should be upvoted. The accuracy votes are designed to reflect accurate calculations and deductions, not agreement with experiment.

For an experimental paper, on the other hand, accuracy means that the results are measured well, accurately, and the originality means that the experiment is of a novel kind, and reveals something interesting about nature.

Comments

Hi Ron, are you sure that Wilson's more general view of systematically derive appropriate EFTs does not allow to correctly deal with the classical self field either? This is what I am not sure about and Vladimir always talks about the rather limited and dated point of view while ignoring Wilson' s insights completely...

---

He doesn't care about Wilsonian field theory. The Kadanovian Fisherite Wilsonian theory is just a more careful, nonperturbative, version of what Feynman, Dyson and Schwinger and all the others did in the 1950s. It just gives you a way to understand what's going on at short distances. At long distances, the problems of definition remain. I'll explain the issue--- what is an electron state? The bare electron is trivial, it's a free particle. But a dressed electron comes with a 1/r field which extends out to infinity, and if you just tap it slightly, it creates a shower of infinitely many soft photons at enormous wavelengths to fix-up the long-distance field.

A proper treatment of a dressed electron seems like it needs to consider the field as part of the particle. Wilsonian approaches are local, and only fold in the self-mass and so on in a local approximation, and ignore the field at long distances, where "long" mean "longer than the compton wavelength of the electron". This is not a solution to the long-distance issue that you don't have a good definition of a single electron state, and no definition for the S-matrix. These problems are relatively involved compared to renormalizing QED.

The models VK advocates are all about using new variables to treat the electron+self-field as a unit somehow. This is interesting for this problem. Feynman et al had a different approach, which was to just give the photon an infinitesimal mass, thereby cutting off the infrared divergences. But you have to be careful here, and the story with the infrared divergences is generally forgotten, precisely because it is not important for the more fundamental Wilsonian issues.

VK doesn't agree with Wilsonian renormalization, and he's totally off base here, but his point of view makes it that he is trying all sorts of different approaches to the long-range problem of infrared divergences.

Also, to respond to a hidden comment, I agree with not telling anyone how to vote. My opinions  on how to vote are solely mine, and I can't impose them on anyone else in any way.

---

@Rom Maimon: Thank you, Ron, for your being devoted to science.

By the way, you write: "what is an electron state? The bare electron is trivial, it's a free particle." To me, a free particle is a particle whose kinetic energy is such that the interactions do not change it noticeably. I do not know how one can imagine a true free particle, a particle without interaction. Can we be sure it obeys a free equations if we are unable to make sure of it? It can be any equation or no equation at all. So, a free Dirac equation is an equation for a physical interacting particle whose kinetic energy is the main term. It has the physical mass and charge by default (by definition, to be exact). In other words, our equations are phenomenological, for physical particles, and we never write equations for non observable (bare) things. When we forget it, we fall in error.

---

I personally solve this problem by considering the single electron state in the theory of massive QED with an infinitesimal photon mass. A single particle exact eigenstate obeys a free Dirac equation with no assumptions, just by Poicare symmetry and consideration of spin. So this eigenstate's motion in response to an external slow electromagnetic field can be described by a free field theory to an approximation, and in the massive photon case, the approximation is that the frequency of external perturbations is not sufficient to create even one photon. The "phenomenological equations", meaning the free equations, then describe the single electron state, and the mass in these equations is the free electron mass. Making the electron interact with the quantum field just involves choosing the field perturbation to come with local counterterms that deal with the short distance renormalization of the mass in response to the self-field. This short distance thing is separate from the problem of the long-range field, the long-range field of the electron only makes a negligible (about 1/alpha) contribution to the electron mass. The only problem of principle that remains is how to describe the electron states in the limit of zero photon mass, when every change in momentum is accompanied by infinitely many soft photons. For this problem, it is useful to consider variables which include the self-field around the electron in some way, but I don't know how exactly, because it is the difficult problem.

Your complaints are not with the methods that people use, but with the philosophical words that they use to justify these methods. Ignore the words, and look at the calculations themselves, they can be justified properly, without the words that tick you off.

---

@Ron: Thanks, Ron. I voted your comment up. But I see a conceptual difference in our approaches. It is not a phylosophy, but physics. There may be non-interacting particles, in your approach. It is physically meaningless. As well, there may be self-interacting particles in your approach. To me, it is a bullshit. I am sure there may only be interacting particles. The rest is our human (or bull) shit (whatever).

---

Let me try to understand what you're saying about "accuracy within the postulates of the paper". So even if the paper's postulates are completely ridiculous, one should still upvote it for accuracy because the results are accurate within the postulates of the paper?

---

I almost agree with you, but what in my opinion does not deserve any upvotes, are papers of the kind "I have found a better method to do/calculate X under the assumtion A in Situatatuon B AND other people who use the established method Y generally are wrong/deluded/etc ...". Papers that do the first part before the AND are perfectly good and may deserve many upvotes on accuracy if done correctly, while the unjustified overreach, often done by crackpots paired with attacks against established physics/physicists, I would rather downvote on accuracy.