Does this statistical inference make any sense?

Question

I am trying to measure a quantity from two data sets which were from identical experiments except that the data were taken at different times. The measurement from the first data set gives \(a\pm\sigma_a\); the second data set gives  \(b\pm\sigma_b\) where  \(\sigma_a \sim \sigma_b \sim \sigma\) is the statistical uncertainty. But it turned out that  \(a-b > 3\sigma\), which means t-test shows that the two results are highly unlikely to be from the same distribution. If I combine the two results in the usual way by weighting them by the inverse of the squares of their uncertainties and calculate the final statistical uncertainty likewise, I will get  \((a+b)/2 \pm \sigma/\sqrt{2}\).  But \((a+b)/2 \pm \sigma/\sqrt{2}\) is not a satisfactory interpretation of my measurement. From a Bayesian statistics point of view, the result from the first data can be treated as the prior probability distribution of the parameter being measured. Updating the prior probability distribution by the posterior probability distribution, i.e. from the second data set, gives me entirely different final probability distribution of the parameter than the one described by \((a+b)/2 \pm \sigma/\sqrt{2}\) in the usual Frequentist 's way. We are exhausted trying to find the systematic variation between the two data sets which were supposed to give us the same result. What is the correct inference of the final result in both the ways, Bayesian and Frequentist? Or, how to present the final result in the case? Any kind of opinion in the form of comment or answer will be highly appreciated. Thanks.

Comments

As an experimenter, if I got a 3sigma difference between two supposedly identical initial conditions experiments I would question the "identical" first, i.e. check very carefully what has changed between the two times of taking data. If I could find no conceivable input change I would take a third data set . Playing with probability methods would not be my choice. Maybe the "error" is in the first data set.

---

Dear Ana, it seems like you did not read the complete question. I know what you mean but there is this important part in the question "We are exhausted trying to find the systematic variation between the two data sets". That means we have no alternative data as well. That's why it is an important question to ask in POF. Could you please comment on the whole question. 

---

You are saying there is no possibility of getting a third data set by a new experiment and you have to publish? I would treat the two data sets independently with their statistical and systematic errors and publish both values  in a similar way that the particle data group presents the values of different experiments , for example , the mass of the Z and let the reader decide.

Alternatively and maybe in parallel I would combine the two sets assuming that the 3 sigma is a fluke of statistics, stating clearly the matter . After all three sigma significance resonances have disappeared before. That is why we set the 5 sigma as definitive. ( I am using "resonances" as an example)
 

Answer 1

There are two possibilities: Either (1) you have seen a rare 3-sigma event, or (2) you have underestimated the uncertainties in your experiment. Making an accidental systematic change between the two measurements would count as an example of (2), i.e. a source of uncertainty that you didn't take into account.

Experience suggests that (2) is by far the likelier possibility.

Ideally you would figure out how and why you underestimated the uncertainty -- what aspect wasn't controlled properly or whatever. But if it's not terribly important and you're in a hurry and you can't do a third and fourth experiment, you can just say there is an additional source of uncertainty $\sigma_{other}$ which accounts for the "unknown unknowns".

Now your two measurements are:

\(a \pm \sigma_a \pm \sigma_{other}\)

\(b \pm \sigma_b \pm \sigma_{other}\)

You can guess \(\sigma_{other}\) by setting it to a value that makes the two measurements 1 sigma apart or so (a reasonably probable value). After you do that, you can say that your best guess for the real answer is something like \((a+b)/2 \pm (\sigma_{other}/\sqrt{2})\) (since I gather that \(\sigma_{other}\) is the dominant source of uncertainty). But you can't treat that expression too literally, it's just a very rough guess.