SoylentNews is people
SoylentNews
This latest result is "pretty damning," says University of Maryland, College Park, cognitive scientist Michael Dougherty, who was not involved with the research. "Citation counts have long been treated as a proxy for research quality," he says, so the finding that less reliable research is cited more points to a "fundamental problem" with how such work is evaluated.
[...] University of California, San Diego, economists Marta Serra-Garcia and Uri Gneezy were interested in whether catchy research ideas would get more attention than mundane ones, even if they were less likely to be true. So they gathered data on 80 papers from three different projects that had tried to replicate important social science findings, with varying levels of success.
Citation counts on Google Scholar were significantly higher for the papers that failed to replicate, they report today in Science Advances, with an average boost of 16 extra citations per year. That's a big number, Serra-Garcia and Gneezy say—papers in high-impact journals in the same time period amassed a total of about 40 citations per year on average.
And when the researchers examined citations in papers published after the landmark replication projects, they found that the papers rarely acknowledged the failure to replicate, mentioning it only 12% of the time.
Well, nobody likes a Debbie Downer, do they?
Journal Reference:
Marta Serra-Garcia, Uri Gneezy. Nonreplicable publications are cited more than replicable ones [open], Science Advances (DOI: 10.1126/sciadv.abd1705)
(Score: 2) by Socrastotle on Friday July 02 2021, @06:31AM
I'm not just poking at the probabilistic nature of things. Quantum mechanics, for instance, is inherently probabilistic - yet few would call it littered with fake assertions. The issue is that if you tell me that you flipped a coin 10,000 times and it came up heads 7,000 times then if I repeat your experiment, and you were honest, then I'm also going to get extremely close to 7,000 heads when I do your experiment - with some slight range for variation. The problem we're running into in the social sciences is that, instead, people are getting 2,300 heads far more often than not. And that means that the original experiment was invalid. The reason for that invalidity can be many things, but one of the biggest concerns is p-hacking. [nih.gov]
p-hacking being one of many practices, but one of the most obvious is after-the-fact data dredging. Imagine you measure a large number of variables about something. And then you run 10,000 trials. As your number of variables increase, you're going to find more and more patterns in the data that mean absolutely nothing. For instance US spending on science is strongly correlated with suicides by hanging. Of course that's obviously a spurious correlation [tylervigen.com] but it's only obvious because those two variables "obviously" (another danger, but that's another topic) have nothing to do with one another. In fact that correlation is far stronger than most published correlations with the ever-implied-but-not-explicitly-stated suggestion of causation - it's a 99.79% correlation rate.
But any good social scientist who wants to engage in proper p-hacking will only be measuring variables that sound viably related to their study. And so when they find these completely unrelated variables that *sound* possibly related: Boom - patch up that hypothesis a bit, and publish. Of course the next scientist who tries to carry out your experiment will find no such pattern, but you published, got some grants, and padded out your CV - so who cares? So poking at astrology again, imagine an astrologer observes a correlation that when people born when Venus was closest to Earth and also blocking out Mars had children who had a fertility rate 37% higher than average. It's pretty easy to see how you can now spin this into an causal astrological effect, where you are using past data (which was clearly just correlational) to make future predictions.
And with enough hand-waving you'll be able to show it again in the future. Perhaps if it doesn't work out one way or another, just add "Ahh! Of course, we also need consider the relationship of Jupiter in the picture." Or perhaps it now has to do with with these two events and their relationship to the Equinox. Just make your model more and more complex so that it always shows what you want it to show, even when it is all based on a completely spurious correlation. This in general is also why the introduction of computer models and computer aided data digging have undoubtedly had a hugely negative impact on science, even though they, on the surface, sound like things that would instead be unimaginably positive for scientific pursuits. Now with computers and computer generated models, find a new spurious correlation can be done at the press of a button. And that's before we even get into "AI"...