"my message is that this noisy, N = 41, between-person study never had a chance. The researchers presumably thought they were doing solid science, but actually they’re trying to use a bathroom scale to weigh a feather—and the feather is resting loosely in the pouch of a kangaroo that is vigorously jumping up and down.
To put it another way, those researchers might well have thought that at best they were doing solid science and at worst they were buying a lottery ticket, in that, even if their study was speculative and noisy, it was still giving them a shot at a discovery.
But, no, they weren’t even buying a lottery ticket. When you do this sort of noisy uncontrolled study and you “win” (that is, find a statistically significant comparison), you actually are very likely to be losing(high type M error, high type S error rate).
That’s what’s so sad about all this. Not that the original researchers failed—all of us fail all the time—but that they never really had a chance.
On the plus side, our understanding of statistics has increased so much in the past several years—no joke—that now we realize this problem, while in the past even a leading psychologist such as Kahneman and a leading journalist such as Gladwell were unaware of the problem."
In biology, this happens when you use a small number of mice (as can happen when you're maybe setting up some kind of secondary confirmation experiment where you take the mice you already experimented on and that had a certain result and you do a further experiment and a few will die or someone will drop a sample or a computer glitch will ruin a data file and you only have usable days for like 5 mice). You can end up with a really strong effect size that, in later experiments, turns out to be much more minor. Or, because mouse lines are supposed to be genetically identical and raised in the sane environments from lab to lab but it turns out that they can randomly have very different microbiomes and epigenetics that can skew results.
No comments:
Post a Comment