SoylentNews is people
SoylentNews
Statistician Valen Johnson and 71 other researchers have proposed a redefinition of statistical significance in order to cut down on irreproducible results, especially those in the biomedical sciences. They propose "to change the default P-value threshold for statistical significance for claims of new discoveries from 0.05 to 0.005" in a preprint article that will be published in an upcoming issue of Nature Human Behavior:
A megateam of reproducibility-minded scientists is renewing a controversial proposal to raise the standard for statistical significance in research studies. They want researchers to dump the long-standing use of a probability value (p-value) of less than 0.05 as the gold standard for significant results, and replace it with the much stiffer p-value threshold of 0.005.
Backers of the change, which has been floated before, say it could dramatically reduce the reporting of false-positive results—studies that claim to find an effect when there is none—and so make more studies reproducible. And they note that researchers in some fields, including genome analysis, have already made a similar switch with beneficial results.
"If we're going to be in a world where the research community expects some strict cutoff ... it's better that that threshold be .005 than .05. That's an improvement over the status quo," says behavioral economist Daniel Benjamin of the University of Southern California in Los Angeles, first author on the new paper, which was posted 22 July as a preprint article [open, DOI: 10.17605/OSF.IO/MKY9J] [DX] on PsyArXiv and is slated for an upcoming issue of Nature Human Behavior. "It seemed like this was something that was doable and easy, and had worked in other fields."
But other scientists reject the idea of any absolute threshold for significance. And some biomedical researchers worry the approach could needlessly drive up the costs of drug trials. "I can't be very enthusiastic about it," says biostatistician Stephen Senn of the Luxembourg Institute of Health in Strassen. "I don't think they've really worked out the practical implications of what they're talking about."
They have proposed a P-value of 0.005 because it corresponds to Bayes factors between approximately 14 and 26 in favor of H1 (the alternative hypothesis), indicating "substantial" to "strong" evidence, and because it would reduce the false positive rate to levels they have judged to be reasonable "in many fields".
Is this good enough? Is it a good start?
OSF project page. If you have trouble downloading the PDF, use this link.
(Score: 2) by looorg on Wednesday July 26 2017, @01:27PM (2 children)
If this gets accepted as the new standard I'm afraid that the new methodology will just include a lot more data-massaging or p-value-hacking. Gathering, (re-)defining and manipulating your data to reach the golden standard of a "proper" result at a p-value of 0,005. After all it will be cheaper then doing actual proper studies. I gather they will just make pre-studies to find out the limits and distribution of the sample and then for the real thing they'll just to stuff as many data-points as possible into the "good" limits/range as they possibly can, all other data-points will be discarded as irrelevant to the study and not counted in the total number of observations. This will naturally be different from field to field. It might make more sense for drug- and medical research then other fields that use the same p-standard.
I do wonder if this will have any kind of impact on the pamphlet they include in the little box of medicine. The side effect parts is some really scarey reading; sure I'd like some massive headaches, potential strokes and a high risk of anal leakage with my pills. Wait wasn't this what the pill was supposed to cure?
Then there is the question of what do you do with all the old "bad research", if a p of 0,005 is the new standard will it invalidate all previous research at p 0,05? Invalidate in the sense of will it be usable, trusted and quoted? Nobody is going to redo all of it to find out. The data for older research might not even exist anymore in any relevant form. If anything I wish they would just then take the opportunity to move away from this kind of thinking or p-validation all together where things are written and then just concluded with a p 0,0x and because of that it must all be significant, true and great.
(Score: 1) by Virindi on Wednesday July 26 2017, @02:22PM (1 child)
If only the full raw data was included with every paper published...
Or at least archived somewhere where anyone can get it. But no, raw data has to be locked away and forgotten. We only care about results!
(Score: 0) by Anonymous Coward on Thursday July 27 2017, @08:35AM
If you're seeking your own funding, there's no time or reward for looking at someone else's data. Ditto for checking reproducibility. You need NOVEL SHINY WHIZZBANG to get grants. The university does not care about your results or your science. Bring in the grant money because the Dean of Science and the Associate Dean of Ethics in Science and the Sub-Dean of Scientific Regulation and the Associate Principal of Science and Society need your 54% overheads pay for their offices and staff.