[Scientastic SUNDAY] Researchers overturn landmark study on the replicability of psychological science

Given that their new result was consistent with all 100 studies being true and you would expect at least ~5% of them to be false

No, the situation is not symmetric. If you take 95% confidence levels, then it means that, in a situation where no effect exists, there is a 5% chance of false positive (that chance alone gets you a non-negligible departure from the null hypothesis). It does not mean that, when an effect exists, there is a 5% chance that a study shows no effect (or within error bounds of the null hypothesis).

In other words: if smoking does not cause cancer, then a study can still show a non-trivial correlation 5% of the time (false positive). But if smoking does cause cancer, then a reasonably-powered study can show this effect very reliably (with almost 100% probability).

I guess that around 5% of studies, by chance alone, would yield a significantly different magnitude for the effect, but that's not the same thing as showing no effect, and it does not count as not replicating.

By the way, I hope that's not what the authors meant by:

"And that was just the beginning," King said. "If you are going to replicate a hundred studies, some will fail by chance alone. That's basic sampling theory. [...]"

Because I can't understand it differently, and it's a freshman-level mistake.

~~~~~

On a side note: the authors also harp about the differences in the conditions of replications. Even if they are somewhat different, I don't think I would trust a finding which is not a little robust with respect to (reasonable) changes in operational conditions.

/r/FeMRADebates Thread Parent Link - sciencedaily.com