What should be the standard/level of the Reviews section of PhysicsOverflow?

Question

The short answer I think is that the reviews section should be research-level.

But as some of the recent discussions have shown, that there are additional points to clarify:

1. What are acceptable sources of submissions apart from the ArXiv?
2. Under what conditions do we reject  reject a submission creation request?
3. What makes submissions potentially closable (from a technical point of view they are just questions)?
4. Should author summaries be more detailled than the ArXiv abstract (which could for example be added when the submission is created by default) ?
5. Concerning the reviews, we already said that "partial reviews" are welcome on PhysicsOverflow, but are there any standards they should comply with?

The list might be incomplete, so please feel free to add in new points.

This post is bey no means intended to overbureaucratize PhysicsOverflow, but it should help to keep up the level of the Reviews section too, which seem a bit less straight forward than for the Q&A section.

To keep things ordered and easily migratable into the FAQ as soon as things are settled (and also for and also to allow for bijunique voting), each point should get adressed and discussed in a separate answer.

Answer 1

To the first point, I consider it to be ridiculous to discriminate based on the source of the submission. You could have a perfectly legitimate paper, but you make the mistake (yes, mistake, ViXrA was made with good intentions, but has detoriated into a crackpot repository) of submitting it to ViXrA. So does that mean that PhysicsOverflow should reject the submission, just because it is also available on ViXrA? I don't think so.

Any source for submissions should be acceptable, judge the paper based on its content, not its source.  

Answer 2

To the second point, I think that this should be our accept/decline policy for submissions (click for larger version):

It's completely transparent and objective, and filters out the worst of the crackpot papers, without censoring any legitimate physics at all.

Comments

Yep, thanks for this answer, very concise and down to the point.

---

Sorry, there's a typo in the second rhombus. It should read "Is it in the scope of a mainstream theory". I'll fix it soon. 

---

I upvoted this with some reservations. I would add two more boxes to this flow-chart, if it fails at all points, "is the submission honest and sincere?" and "is there a community of people who would read the reviews?" These are more political, but they can prevent rejection when there is something half-baked and yet still potentially interesting in the future. An example would be a detailed simulation of a completely wacky cellular automaton model a-la Wolfram, with false associations to particle physics, but which is interesting for other reasons having nothing to do with physics.

There are interesting models which are not particularly mathematically or experimentally justified, but are interesting. I am not crying tears over the rejection of the "Theory of everything" paper, but I am worried about the future, when less scrupulous people will have the opportunity to misuse the rejection policy to exclude honest research on the margins.

---

@RonMaimon But how are we supposed to assess the honesty of a submission? Go to the author's house and do a lie detector test? Of course there is a "community of people who will read the reviews" for everything - the group of people who will click on the submission list + the group of people who will find it from a search engine or something + the group of people who randomly entered the submission ID into the URL bar.

---

I do understand the fear that in future, the policy may be misused, but I think the policy is quite clear and objective. I can't really comprehend exactly what you're saying, could you provide an example of an "interesting model without mathematical or experimental justification"?

---

In addition to what Dimension10 said, the scheme he designed will never by blind and mindlessly applied. If a submission is wringly rejected first and you for example point out why it is interesting after all, the issue can certainly be resolved.

But we really have to be careful to not leave a weird impression or look like an unmoderated crackpot forum. It might be detrimental at this early stage of the site ...

---

Right,  that is why I've changed my mind. We must attract great physicists in here.

---

It is a physical forum, so I would love to see a rhombus with "Is it physically justified?" question too. Thanks.

---

I gave the example: a random cellular automaton model usually has zero mathematical or physical or experimental justification. But they are good to study, because they can give insight into physical models eventually, as for example, the lattice fluid model eventually got a mathematical justification.

---

@RonMaimon That already passes the first criterion, it isn't in conflict with established physics.

@VladimirKalitvianski Iff it's physically justified, it would also be either mathematically or experimentally justified.

---

@dimension10: You are hopelessly naive here, you have no experience with human politics, or the language that people use when they come up with a new model or a new idea. They are often very grating to establishment ears.

When a person proposes a cellular automaton model, it often comes along with absurd completely unjustified reasoning for how this little model is actually what nature is doing underneath everything. This reasoning is completely crackpot, and totally off base, but it is this reasoning that motivates a lot of people to study such models, and such a motivation, while completely unjustified mathematically or experimentally, sometimes leads a person to consider interesting models.

When you institute such a flowchart, you will get people grabbing a sentence here or a sentence there in the motivation or conclusion and saying "This sentence is not physically or mathematically justified" and therefore rejecting the paper from review, even when it contains a good model inside. This is the danger, instituting political censorship.

The paper you rejected, the "theory of everything" has no content at all, it has no model, just a few ridiculous equations copied from elementary physics books without understanding. I can understand deleting it as an attack on the site, but I am wary of attempting an objective criterion list to separate crackpot from non-crackpot work, as such a criterion list will be used for exclusion eventually, guaranteed. The only way to prevent this is to allow veto, so that if, say, two +500 score people insist on a review then it must be accepted for review.

Remember, accepting for review is not the same as giving weight to the document. If it is crap, it will be torn to shreds. The barrier for a submission must be as minimal as possible. Although, at this early stage, I don't mind if only the serious submissions are accepted, I hope that the censorship policy can be relaxed at a later date, when the system is fully working and the import of all serious papers is automatically done. Then the crackpot papers will be lost in the noise, unless we are subject to a deliberate spam duplication attack.

---

@RonMaimon The veto can be used frivolously. How about the vetoer having to pay a ransom of 40 rep, or something?

I am fine with relaxing the policy later on.

---

@Dilaton: You don't understand what will happen once there is a large crowd here. When there is a new thing, people will scream bloody murder, and you will feel pressure. The only "dark side" is censorship, crackpots are harmless. although in this case I understand that when we start out, we want to stay professional.

I hope that later on we will apply a policy of benign neglect regarding submissions that are crazy, just to bend over backwards to make sure that there is no censorship whatsoever. Papers of the "theory of everything" sort we got can be extra-judicially removed, as they were in this case, as there is no controversy over them.

---

@Dilaton No, I meant the person who vetoes the rejection. And by "pay" I mean lose reputation points. But that would mean the person becomes less reputable because he felt that something is not crackpot, so no, I take that back.

Vetoes can be used frivolously, I prefer the idea of dropping the policy once we graduate from Reviews I.

---

Ok...

Yes, I hope too that as soon as a large enough good crowd is here (maybe we should not graduate before this happens ...?), things will become easier and that the minimal standard outlined in the flowchart will be agreed upon by the majority of people without speaking (and the crackpots and cranks in a negligible minority), such that it will no longer have to be explicitely demanded and written down...

But if we give a wrong, unprofessional, rather crackpotty and weird impression now, there will never by any good large enough crowd here ...

Also, I do not really fundamentally disagree with @RonMaimon, as I think we have both seen enough bad stuff happening and things turning to the dark side elsewhere, to know that it is important to avoid such things here by all means ...

Answer 3

I think the only standard we need is that there is a significant body of legible text which is on topic for the forum, meaning about physics or related mathematics, or related methods from other fields, or borrowing from physics into other fields.

If you have an open review, anyone can get a quick review of material, with pointers toward related literature. Any attempt to keep some people out (other than presenting significant legible on-topic original text) will create a political struggle for the keys to the forum, and the result will be that some group of censorious editors will succeed in shutting up a bunch of other people, using political lables.

There is no need for a paper to have any pedigree, in my opinion. Crank papers are the least of our concern, they do no harm, as they are downvoted immediately, and require no special attention afterwards.

Comments

You expressed the concern that "nobody is ominscient" somewhere, which I take as "crackpottery is subjective". I agree with that, but the worst of crackpottery is quite objective. See my answer to the second point for a criterion I consider quite objective.

Answer 4

To point 3, I think that submissions shouldn't generally be closed, for now at least. Closing a submission just prevents reviews, that's it. Reviews I is only a temporary phase, of course, so what we should really think about is the reviews section after graduation.

If PhysicsOverflow ever decides that not all ArXiV physics papers are on-topic for PhysicsOverflow, then we could "close" bot-submitted submissions (if they have interesting reviews, or else we could just delete them, they were submitted by a bot, after all). But for now, no.

Answer 5

To point 4, yes of course they should!

I mean, it should be allowed for users to just copy the abstract (as long as it's not in violation of some copyright), but we should not grant authorship based on that. Anybody can copy the abstract, or paraphrase it, but that doesn't mean that they wrote the paper.