SoylentNews is people
SoylentNews
This latest result is "pretty damning," says University of Maryland, College Park, cognitive scientist Michael Dougherty, who was not involved with the research. "Citation counts have long been treated as a proxy for research quality," he says, so the finding that less reliable research is cited more points to a "fundamental problem" with how such work is evaluated.
[...] University of California, San Diego, economists Marta Serra-Garcia and Uri Gneezy were interested in whether catchy research ideas would get more attention than mundane ones, even if they were less likely to be true. So they gathered data on 80 papers from three different projects that had tried to replicate important social science findings, with varying levels of success.
Citation counts on Google Scholar were significantly higher for the papers that failed to replicate, they report today in Science Advances, with an average boost of 16 extra citations per year. That's a big number, Serra-Garcia and Gneezy say—papers in high-impact journals in the same time period amassed a total of about 40 citations per year on average.
And when the researchers examined citations in papers published after the landmark replication projects, they found that the papers rarely acknowledged the failure to replicate, mentioning it only 12% of the time.
Well, nobody likes a Debbie Downer, do they?
Journal Reference:
Marta Serra-Garcia, Uri Gneezy. Nonreplicable publications are cited more than replicable ones [open], Science Advances (DOI: 10.1126/sciadv.abd1705)
(Score: 1, Insightful) by crafoo on Thursday July 01 2021, @12:35PM (10 children)
Social Sciences are Fake News. Well, fake science at least. Cult ideology and propaganda cloaked in the trappings of the scientific process.
(Score: 0) by Anonymous Coward on Thursday July 01 2021, @01:54PM
Damn straight! The only real sciences are the Antisocial Sciences that make your brain hurt when you study them.
(Score: 3, Interesting) by HiThere on Thursday July 01 2021, @02:05PM (8 children)
If you said "...often fake news..." I'd agree with you. And very often weak science. Some of it's good, though. And a lot of chemistry isn't all *that* good.
This one looks like an example of "weak science". 80 isn't a large number of papers. (OTOH, I wasn't interested enough to more than read the summary.)
To be good science they'd need a much larger number of papers, and they'd need to stratify "cites" into categories of at least "agreed with" and "disagreed with". But their point that "cite counts" is a very poor measure of paper quality it probably correct.
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.
(Score: 3, Informative) by Socrastotle on Thursday July 01 2021, @02:35PM (7 children)
Statistics can often be somewhat paradoxical. You can, assuming you have a random and representative sample, derive conclusions with extremely small samples.
80, for the purposes here, is huge. One simple way to start to play with numbers a bit is to use a binomial calculator [stattrek.com]. So let's say we believe that 70% of studies are, on average, replicable. That's already abysmal, but I want to lowball things here to emphasize the point. And so we do a replication study of 80 different studies that we believe are representative of the sample we're considering, and are randomly selected. I'd normally expect to get around 56 "successes" (80 * 0.7) but instead I only got 40. That's "only" 16 away, and the sample size is relatively small. So if I assume that the real rate is indeed 70%, how often should only 40 or fewer replicate?
The answer is
. And emphasizing, that is a percent, so in other words this would be expected to happen, by random chance, less than
times. So you can say with an exceptionally high degree of certainty that the real rate is not 70%. If you don't follow how to use the calculator:
Probability of success = 0.7
Number of trials = 80
Number of successes = 40
And we're looking at P(X = x).
(Score: 0) by Anonymous Coward on Thursday July 01 2021, @02:59PM (1 child)
People who have done replication stidies found 90% were not replicable.
(Score: 2) by Socrastotle on Thursday July 01 2021, @03:19PM
I'm not familiar with any that low, but I do know that social psychology in particular has a replication rate of around 25%.
I was speaking in terms of ideals. In general what percent of our science would we like to be "valid"? Of course 100%, but that's impractical. So what is practical? Perhaps something like 97%. But with such a high figure it becomes intuitively obvious that hitting e.g. 50% over 80 samples is not just noise. But what about hitting 50% when you're only aiming for 70%? It doesn't, intuitively, seem so implausible, especially over "only" 80 samples.
So the point is that even if you want to set our "real" figure far lower than anybody would ever actually want or think (again - 70% is an abysmal replication rate for idealized "science"), 80 samples is far more than enough to draw some extremely high probability conclusions.
(Score: 1) by shrewdsheep on Thursday July 01 2021, @05:05PM (3 children)
I guess your intention is to calculate a P-value for the observation of 40 successes for 80 replications under the null hypothesis of a success rate of .7. To get a meaningful P-value, you would have to calculate the probability P(X ≤ 40). The probability of a simple outcome (i.e. P(X = x)) is almost always meaningless, as a matter of fact it is always zero for continuous distributions. Also for the binomial it tends to zero for any outcome and success probability as N tends to infinity. The P-value is the probability of the event of observing our outcome plus all more extreme outcomes.
(Score: 2) by Socrastotle on Thursday July 01 2021, @05:14PM (2 children)
It is the P(X <= x) of course. I got HTML'd with the less than sign.
(Score: 2) by Anti-aristarchus on Thursday July 01 2021, @09:02PM (1 child)
But shouldn't it be:
One must take prior probabilities into account, whether frequentist or subjectivist.
(Score: 2, Informative) by shrewdsheep on Monday July 05 2021, @07:19AM
As a frequentest your prior would be uniform (and might therefore be improper), if you are an empirical Bayesian you would again have a uniform (improper) prior, but on the hyperparameters, and as a full Bayesian, well, the full fudging would start.
(Score: 2) by HiThere on Thursday July 01 2021, @05:26PM
From the summary, it seems clear that the sample was not "random and representative sample" as what they did was "So they gathered data on 80 papers from three different projects".
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.