I am adding an answer, because there are about 4 people here, so you don't want a vote in this case, and votes are meaningless anyway. You know who everybody is, and what they think, so you should just achieve consensus through discussion.
The idea for "two questions" is simply to have two separate votes, and two separate locations for the two different kinds of (significant) editoral comments, so that you segregate comments about previous related work from comments about the paper itself.
The comments about the paper itself should strictly consists of positive and negative reviews regarding the physics content, which are not duplicative of the paper's contents. A positive review should consist of a nontrivial application or extension, not meriting a separate paper, but interesting enough to give perspective on the importance of the work, or perhaps a more pedagogical explanation of an obscure point, which explains the applications of the result, and gives confidence that the calculation is correct, because it explains a hard step.
A negative review can consist of a counterexample to a claimed theorem, a flaw in an argument, an experimental incogruency, a naturalness comment, a misunderstood thing in the paper, anything that would normally go on a negative referee report. Both types of refereeing answers can be voted on normally. These things go in the "review" section as answers.
For the "originality" question, the idea is to simply aggregate the claims of prior art in one place, and use these claims to come up with an originality score. This is produced by community consensus, by people voting +1 when they feel the paper's score is too low, and voting "-1" when they feel it is too high (or voting 0). This is the normal process on these types of sites, and the community usually converges to good scores.
Then the reputation is the weighted sum of the accuracy score by the originality score, something like exp(originality/5) times accuracy, or whatever reputation function people think is fair (I think perhaps exp((originality/5)^{1/3}) times accuracy--- with sign of course--- will be fair).
Part of the idea is to provide authors reputation simply from their published work, so if they come here, they automatically get reputation from publications, they don't need to do mickey-mouse stuff on the site to get reputation. But also, it is designed to be a reasonable substitute for journal refereeing which does not require anything else. It's sufficient to produce a number which is roughly similar to the number journals use when deciding whether to publish a paper or not to publish, minus the political aspects.
There is no third criterion, originality and accuracy are completely sufficient to produce a reasonable outcome for all published papers.
A straight "vote up/down" on papers doesn't work well at all, as the vote mixes up originality/quality and therefore it can also be "I don't like the author", "I don't like the institution", "I don't like the subject matter", "I don't like the font", or whatever other superficial criteria people use. It will simply reinforces dogma regarding papers, and in those cases where you have accurate original papers, you need to make sure that people show you why it is not original specifically, or why it is not correct specifically. If they don't have to do this, they will make up reasons for upvoting/downvoting, which are different from these two criteria, which are the only ones you should use. Period.
I am sure that if you don't separate the criteria like this, into two separate votes, with two separate kinds of comments, papers which are both original and correct will get political downvotes, without any specific criticism, simply because they are unpopular. They will be downvoted, without anyone showing a mistake, and without anyone showing prior art. This is the tragedy of politics that requires high-power editors at major journals. The high-power guys are just there to simply ignore all aspects of criticism except originality and correctness.
With specific criteria questions, you will ownvote on originality if you think there is a missing citation, or the result is folklore. You will downvote on content if you think there is a serious mistake deserving a downvote, and hopefully you will also say what that mistake is! The separation essentially requires you to be specific about why you are downvoting the paper (or upvoting), the natural thing is that you downvote a paper's accuracy and also upvote a specific criticism, or you upvote to upvote a specific missing citation, and downvote the paper's originality.
Any general comments of a non-refereeing nature, like explanation of the results, can go into the QUESTION for the accuracy section. You can summarize the whole content of the paper itself in the question, feel free! The question text is otherwise just a link. But if you have a neutral comment, neither a criticism nor an extension/significant-reworking, it should just be added to the question community wiki style, or as a comment, so it doesn't give you rep, and it has no bearing on the refereeing content--- it's just explanation of the paper. This also provides a way to summarize old nonfree literature.
The result in this way can be extremely similar to a (good) journal's refereeing, if people are careful about how they vote. If people are careless about how they vote, however, you might need to strictly enforce downvoting on published literature, so that every downvote comes with an associated upvote of one or more specific criticism of the paper, and each downvote on originality comes with an upvote on a missing citation. This type of thing can make it so that if a criticism is retracted (for example, someone posts a sincere mistake, and later it is shown to be wrong), then all the downvotes which came from this criticism are automatically retracted when the criticism is deleted. This is a major project, and it might not be necessary, it can happen just by asking people to vote with sincerity and downvote with respect, and generally, I trust the community on this. People who are free usually do a good job on evaluating literature.
The originality/quality separation is absolutely essential. Without it, you will never get original material, you get quora-like or stackexchange-like rehashes or old crap. Rehashes get cheap upvotes, original material gets hardly anything, because hardly anyone can evaluate it an be sure it is correct. Rehashes are next to zero effort, while original material is extremely hard to produce. Nobody will bother with original material unless original material is rewarded much more. This is the only way to get professional level contributions, to separate out original contributions and reward them more.
The two criteria really are the entire content of the refereeing process, when it is done right. When it is done wrong, there are a million other things to consider, like "does this suit my journal's audience?", "What will be the impact factor?", "How many people care about this topic?" and so on and so on. These are all bogus criteria, and journals do best when they are not a part of the refreeing process. All you need is ORIGINALITY/ACCURACY, and that's it.
If you are under the impression that crackpots score high on originality, don't be. It is extremely rare that you here an original crackpot idea. The only crackpot ideas that score high on originality are those that have an extremely good chance for not being crackpot at all.
Q&A (4870)
