Discussion about this post

User's avatar
James Smoliga, DVM, PhD's avatar

As somebody that works in (and debunks) research within the biological and medical sciences, I find it fascinating how much overlap I see between what you describe and what I find in very different disciplines.

With that first decoy example having a correlation of r = -0.29... Even if the study was well performed, a correlation <0.30 is weak at best. And, I would argue, it is often the result of statistical noise that is bound to be found when one does a sufficient number of analyses. Indeed, I have shown this in my own work debunking the concept of "digit ratio" (a field in most correlations are in the 0.2 to 0.3 magnitude).

So many other examples of weak science (or even theory) being supported as facts.

This is why, as somebody who values science and the scientific method, I am skeptical anytime I see research quoted or a headline proclaiming "new study finds" about any topic. I want to trust science and scientists, I really do. But there are so many incentives to confirm hypotheses, rather than rigorously test them, that a healthy sense of skepticism is necessary.

We need to actually dig into the methods and results ourselves to critically evaluate them, otherwise it's hard to know if what is being summarized accurately reflects the research itself. That includes evaluating the authors' own conclusions to determine if they are directly supported by the data or reaching too far.

Luke Conway's avatar

An absolutely brilliant piece that, I think, correctly highlights the main problems we face as social science researchers. Our problem is not that results can't be replicated. It's that they CAN be replicated...but the methods were bad to begin with.

I did not agree with everything in here, but I will be recommending this piece to anyone who wonders why so many of us academics believe the social sciences are going off the moorings; or anyone interested in real facts about poverty and capitalism.

The opening transgender/microaggressions reversal was freakin' genius. I hope psychologists in my own field are listening! Thanks for a terrific piece.

7 more comments...

No posts

Ready for more?