How should papers be introduced into the Reviews section?

Question

The review section is a great idea, as is Ron's idea of linking two questions to each reviewed paper. But how should papers be introduced into the review section in the first place?

Comments

presumably a bot will do this. do you mean something else?

---

No, physicsnewbie, you don't want to have people put them in by hand, you need to use a bot. The bot doesn't vote, it just makes the space for review, so that people don't have any barrier to posting comments whenever they feel like it.

My "two questions" is not a great idea, it's a stupid hack.

---

@physicsnewbie If I'm not wrong, polarkernel's idea of the OAI PMH allows even the older papers.  

---

@physicsnewbie Huh? First of all, your comment has nothing to do with the reply or anything else, but fine. I can't find a single place where he has said so, in fact he clearly states that a user can post a paper not on the arxiv, and that we could later even try importing from other databases.  

---

Even more, I hope that when the reviews section is stable, we can host contributions locally, so that they can get a professional review, even if they don't appear on arxiv, or in a journal. For example, transferring a really interesting answer to a question with a new calculation, to see where it fits in the existing literature (one has to be careful here to exceed the content quality of typical arxiv papers when it first happens).

---

@ryan People could do the posting of papers themselves, rather than relying on a blind bot.

---

@dimension10 Ron appears to be suggesting that the bot does all the downloading, and it isn't possible for a user to manually suggest a review for some obscure paper from the past, that predates Arxiv, that's been ignored by the physics community.

Answer 1

The questions within the category "Reviews" and its subcategories will be a special type of questions in any case (two related questions or only one). As dimension10 already mentioned, we should not start his category before this new type is defined and available. This to avoid later ugly handmade synch corrections. This type should fit somehow in the Q2A framework to avoid a completely new development. In order to approach a more detailed specification of your requirements, I try to formulate some more detailed explanations and questions.

1) To ease the implementation I would prefer to have only one question per paper. I already made a proposition on how such a question could look like. To keep the information concentrated, I copy my proposition here again:

The text includes a link to the paper on ArXiV and the papers title. Two voting inputs are possible, one for originality and one for accuracy/quality. Authorized users may give one vote on each of them. A score value is computed from the total of each vote criteria and is displayed within the question.

2) The asking user (in the Q2A sense) in the bottom line will be the bot for papers imported automatically or a real user, if imported manually.  I have grayed the usual votes on the top left in my proposition, because it is unclear to me, who should earn the points from such a question. The bot? The editing user? Nobody?

3) Should there be a mechanism to change this asking user, if a question imported by a bot is edited by a real user? What if multiple users edit such questions?

4) Who exactly earns the votes for originality and quality and/or the score value and how? The authors of the papers? How would you "link them individually to authors by hand when the authors join the site"? By direct access to the database? Q2A does not provide any means to distribute votes to multiple users.

5) What shall the semantics of answers and comments to review questions be? At least earning votes is easy there, we can use the normal Q2A behavior.

6) I assume that in any case, we need a plugin to import papers into review questions manually. OK?

You help best for a fast implementation of your demands, if you provide precise requests to me.

Changing the voting system or the tagging system in Q2A is not only a core-hack, it's a hardcore hack! The related code is located deep in the framework and percolated over many files. If somebody is interested in details, you may download the code here, it is open source. 5 minutes installation time mentioned there are true, but not for a reviewing system. Propositions are welcome.

Comments

+1 Well this answers my question, since you're providing both a bot and a manual input which I'm OK with.

---

+1, and Thanks for posting an answer here too.  

1.   
2. I agree, it is sort of unclear who should get the reputation from the upvotes.
   
   Ron proposed that the author who comes to defend his work gets the reputation; however, this is difficult, first of all, because, it may not be possible to verify that the "author" is really the real author of the paper. ArXiV does provide a "view email" tool, but this can only be viewed manually, and if a user's email has changed, and it is not updated on ArXiV... Furthermore, what if more than one author defends the paper? What if their response "defence" is just a "Thank you for your comments" or something? Also, the fact that an author did not visit Physics Overflow does not necessarily mean that they did not contribute to the paper, or that their name is only for political reasons.              
   
   The points should not be obtained by the editor. The editor only summarises the paper, which the voters don't vote on. They only vote on the paper itself (by the way, I think that at the end of the post, there could be text saying "Summary:", so that users only write under the summary, which will prevent them from confusion the summary with the abstract of the paper, or whatever.  
   
   Therefore, I think that correctly assigning the rep is implausible, and that the rep should just be gained by the bot or nobody. Maybe the bot, and it could be called "Authors", or something. 
3.  No, for the reason mentioned in the last-but-1 paragraph on point #2.  
4.  The bot, or nobody, for reasons mentioned in the last paragraph of (2).
5. It can just be like any other ansswer or comment. Rep gained goes to the original writers of the answer or comment. I doubt there is an issue of "'originality' and 'accuracy'" for reviews and comments.
6. Yes, but I don't understand how this plugin could be made. I assume you are talking about databases other than ArXiV, then one would need a huge list of such databases, and separate mechanisms for each of them. 

By the way, there should be text at the end of each post saying "Use comments for citation suggestions, and answers for reviews."         

---

The issue of assigning reputation to authors should be by splitting it equally between all authors who come here and respond cogently to criticism. If you have a political author who is able to discuss cogently, there's nothing you can do, give the author the rep. The author's name appears, and if the paper is badly refereed it will impact this author's reputation also, so they should reap the benefits. You should trust the names that appear, they are usually accurate, aside from rare cases.

---

Wouldn't it be simpler if a pseudo-user is generated for each review, and given a name that references the paper such as arXiv:1403.3361 or user: number:date etc? At least with this, the paper is given the points.

---

Another ridiculous idea! The WHOLE POINT IS TO GIVE THE AUTHOR REPUTATION!!! So that a person with a long career doesn't need to do anything to get respect on the site, and writing papers will give rep the same as answering questions, much more so if the paper is original and accurate.

---

Sorry, posted anonymously by accident, the above comment is by me. But the reputation has to go to the author, they wrote it!

---

@ron OK, after thinking about this, i agree that the points have to go to an author because their name isn't going to be associated with one paper. Hmm, I wonder what on earth I was thinking?

Answer 2

You need to use a bot. All physics papers on arxiv must be here.

For non-arxiv literature, you should probably do it by hand, although it is also possible, if the site is successful, to link to other databases of older literature. But arxiv is comprehensive for the past ~20 years, and should be enough to get the ball rolling.

The importing of papers is very simple--- the bot makes ~100,000 questions with a link, and two votes apiece, owned by "community" and link them individually to authors by hand when the authors join the site. There's only 60,000 physicists or so, it won't be too hard. You can even write this kind of bot in user-land! If I left a computer attached to this website with a paper-introducing bot, it would run out of papers in about a month.

Comments

By "All papers on arxiv", you mean "All physics papers on arxiv", right? 

---

The idea of creating a review for every physics paper from Arxiv dating back to 1995 isn't realistic. It means you'd have to do it silently on the main page, to prevent this process swamping everything else. To make matters worse, people would eventually no longer pay attention because it would be like sifting through the mud for a diamond. It needs to be actively done by members here to ensure a high probability of something interesting being publicized for review.

---

@physicsnewbie Of course, the imports themselves can be hidden from the main page and activity. But the comparison of reviewing to "mud" isn't fair.      

---

Automated download from arxiv is not permitted. http://arxiv.org/help/robots

---

Maybe we should contact them? As ArXiv is (as I understad it) rather meant to be a repository for papers than a review site, I see no reason why they should be against our intention to get the papers for reviewing and discussing them ... (?).
 

---

@user10001 read polarkernel's answer.   OAI - PMH is designed to avoid automatic downloads. 

---

Automated downloading from an arxiv mirror, and daily updates using a SINGLE download of the feed. There is no barrier to downloading the indexes, it just doesn't like you downloading all the pdfs, the reason is that (at least in the past) it stored the tex, and a computer had to process it into a pdf, and if you downloaded a lot at the same time, the computer would thrash. I don't even know if they do this anymore, they probably store all the pdfs, all of arxiv wouldn't even be a blip in modern storage.

Answer 3

Read point 4 of PolarKernel's answer here . It should answer your question.     

Comments

The link doesn't work

---

@physicsnewbie, sorry, fixed.  

Answer 4

Instead of using a bot to automatically download proposals from Arxiv, I think this should be left to members only: to post a link to any paper they wish. It should be then left to the community to upvote the proposal, which is then put into the accepted level if it's score reaches a minimum threshold. Only then are two questions allocated to it by a moderator.

This prevents the site from being swamped by auto-downloads, and introduces an initial reviewing phase from someone who at least has read the paper, and finds it interesting. Making sure that members are only allowed to make a proposal also ensures that authors don't misuse the site to promote their own work for the sake of publicity.

Comments

Do you mean something like a "Request for import votes thread" for papers to be reviewed? In my opinion, this would be unnecessarily restrictive. If PhysicsOverflow will work as expected and become a high-level site mostly populated by researchers, good students, and otherwise knowledgeable enough people who know what they are talking about, crackpots trying to promote genuine nonsense will not have much fun here: their nonsense papers will quickly get downvoted and debunked from a physics point of view by the community.

---

@Dilaton OK, I agree that voting for proposals before they're posted, is too restrictive..

---

Physicsnewbie: you have requested a feature, voting to import, which can have absolutely no purpose other than wrecking the usability of the site.

---

@ron I agree now that voting to import is pointless, which is why I gave a second answer

---

Don't hide the answer, even if it doesn't represent your POV any longer. Having the answer helps others understand the discussion better. I have reshown it.   

Answer 5

Some kind of a middle ground between importing all ArXiv papers automatically and restricting the papers discussed in the Reviews section to the ones approved by the community by means of some kind of "Request for import votes" thread for papers, would be to simply let people post their (or other's) new (or older) papers they find interesting and like to be discussed.

If PhysicsOverflow mostly attracts people who know what they are doing and talking about, the danger of attracting crackpots who try to promote genuine nonsense would be rather low, as their posts would be quickly downvoted and debunked from a physics point of view.

(Of course, which method is best to import how many papers depends on the question of our Review section is meant to replace or complement journal peer reviewing at the end, ...)

Comments

I think blindly importing all Arxiv papers from HEP, say, is a terrible idea. People aren't generally going to be interested, and those that are will have gone to Arxiv to look at the latest additions anyway. You need to leave it to members to post a paper they think is worth discussing, which then shows up on the main page as new activity.

---

I think blindly linking all arxiv papers is the only right idea. WHAT POSSIBLE REASON COULD THERE BE TO WAIT FOR SOMEONE TO DO IT??? It's absolutely ridiculous! Just leave them there, empty. It will also bring people to this site, when they search for specific paper titles.

Answer 6

This is an additional answer, after reading Dilaton's comment for my previous one.

I think it should be left to members to post the two questions for a paper, and not some blind auto-downloading procedure, because it will trivialize reviewing as some blind auto-process. Have a look at papers on http://www.vixra.org/hep/0703 dating back to 2007, and you'll find the comments are generally empty.

If it's left to members to suggest a paper, then at least they've gone to the trouble of reading and posting it in the Reviews section in the first place.

Comments

@dilaton I would be happy with the best of both worlds, where a bot downloads papers from Arxiv, and users can manually create one. The download from Arxiv must be done silently, but activity on the main page must show up for one created by a user. To get the site going, we only need a manual procedure to start with and user Dimension10 has given a good demonstration of this ./?qa=refereeing/hep-ex 

---

Importing ArXiV papers by a bot is not difficult (see here, point 4), it's just some work to do and will take (not that much) time. I will implement what the community finally decides, no problem. However, to raise my motivation and understanding, I would be happy if someone could explain to me what the purpose of importing all papers since the beginning shall be. In my feeling, we end up with a statistics at the right side panel like this:

100,822 questions
1,252 answers
5,281 comments
1,245 users

1.2% answered questions, very competent site, isn't it? So please help me to understand.

---

Don't worry about those statistics, anyway, the reviews doesn't need to count. Ignore questions in review when making the statistics of the site overall.

Anyway, the reviews section has the potential to get filled up with phase-transition suddenness, as there is an instability--- once enough people comment on papers, it will be a free-for-all, as everone in the world comes to comment on everyone else's papers.

It is also easy to link outside commentary, so all the blog posts regarding papers will get a link here (the bloggers are usually ahead of the curve), and hopefully a short summary of the main refereeing points made in the blog, without the usual bloggy blah-blah-blah.

The referee section though needs to be organized heirarchically, like arxiv, so that people can find the paper easily, and comment on it. This is the main sticking point. Making physics categories and sub-categories is serious work, and requires a physicist who knows what work is in what field. I can do this manually for about 5,000 papers in a short time, but for 100,000 papers, it will take months by hand. I am hoping that once there is a seed hierarchical tag system, it can be done by the community quickly, so that the tag system will be quickly useful, to find all arxiv papers relating to PT symmetric quantum mechanics without a search, or all papers relating to certain GUTs and so on.

---

To physicsnewbie: the idea of manually created papers showing up on the main page, while the bot papers are silent, this is a good idea. I am sorry for the "rude rhetoric", but I was suspicious! I see now you are sincere and helpful, thanks for the good idea, and sorry for the hostility. Blunt discussion is necessary to avoid falling into the trap of making an average of positive and negative, an average which always is zero.

---

@dimension10 According to PolarKernel, a plugin is taking care of manual input, which is ideal IMO. We can discuss how it should work, and what it generates on another thread.

---

@physicsnewbie I know that, however I am concerned about how that can be done. Different journal databases may have different formats. 

---

Also, the demonstration is currently outdated, and anyway it is a demonstration of the questions that the bot should generate.