Humanity would understand very little about cancer, and be hard-pressed to find cures, without scientific research. But what if, when teams recreated each other's research, they didn't arrive at the same result?
That's what the Reproducibility Project: Cancer Biology of the Center for Open Science is attempting to do—redo parts of 50 important cancer studies and compare their results. They released their first five replications today, and it turns out that not all of the data is matching up. At least once in every paper, a result reported as statistically significant (the way scientists calculate whether an effect is caused by more than chance alone) was not statistically significant in the replicated study. In two of the cases, the differences between the initial and replicated studies were even more striking, giving the Center for Open Science researchers cause for concern
"I was surprised by the results because of all that homework that we did" to make sure the studies were being reproduced accurately, Tim Errington, Project Manager at the Center for Open Science told Gizmodo. "We thought we were crossing every T and dotting every I... Seeing some of these experimental systems not behave the same was something I was not expecting to happen."
(Score: 0) by Anonymous Coward on Saturday February 04 2017, @11:20AM
I wish they would see whether the results were similar or not rather than use statistical significance as the metric for reproducible.
(Score: 1) by khallow on Saturday February 04 2017, @02:16PM
I wish they would see whether the results were similar or not rather than use statistical significance as the metric for reproducible.
They are. The original research used statistical significance so a faithful reproduction of that work also has to use it.
(Score: 1) by shrewdsheep on Saturday February 04 2017, @02:28PM
The analysis should follow through in this way, sure enough. However, reproducibility should not be defined by comparing the significance of p-values of the studies. If the addition of the second study strengthens the conclusions of the first paper, that should count as reproduction and this is what OP meant, I believe. Also keep in mind the multiple testing problem involved in the whole enterprise.
(Score: 1) by khallow on Sunday February 05 2017, @12:44AM
However, reproducibility should not be defined by comparing the significance of p-values of the studies.
Sorry, those p-values are big parts of the studies. And where's the funding to aggressively push forward on a hundred different studies? Merely replicating the studies (which is what's being done here), is pretty hard on its own.
(Score: 3, Insightful) by looorg on Saturday February 04 2017, @11:53AM
Instead, they thought that our scientific culture incentivizes clean, sexy results ...
It sure does. Nobody wants reports or papers that conclude that there is no certain result. They all want clear quantifiable results. That is how you get, future, grant money.
Iorns was surprised by the personal reactions from some of the scientists on seeing their results called into question ...
Shouldn't be that surprising, those people might have built their entire career on that paper or research and if that is now called into question that could be very bad for them. Titles, position, status and future grant money could just go byebye.
---
I gather it's to much to hope for that this is another nail in the coffin of the p-value significance hoax, or p-value hacking or whatever you like to refer to it as. P-values was just such a sweet and simple thing, you didn't have to explain things anymore. Just run your little test and the statistical software said it was all good and then you could go and do other things.
But then perhaps cancer is just tricky, after all cells start dividing in a manner not intended. What is to say that that would follow the same identical path over and over again.
(Score: 2) by bradley13 on Saturday February 04 2017, @01:48PM
...except the way people understand them. The lazy see a p-test and think: "here's a result". This is, of course, wrong.
If you see a result p really means is: Hey, look - this might be something interesting. We should look into this further.
Everyone is somebody else's weirdo.
(Score: 0) by Anonymous Coward on Saturday February 04 2017, @01:55PM
It depends how you use it:
https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theorytestingparadox.pdf [umn.edu]
http://www.biorxiv.org/content/biorxiv/early/2016/12/20/095570.full.pdf [biorxiv.org]
(Score: 2) by TheRaven on Saturday February 04 2017, @04:25PM
sudo mod me up
(Score: 3, Insightful) by mhajicek on Saturday February 04 2017, @05:53PM
Perhaps it should become standard to publish the full data set, so that anyone cam do their own statistical analysis.
The spacelike surfaces of time foliations can have a cusp at the surface of discontinuity. - P. Hajicek
(Score: 2) by deimtee on Saturday February 04 2017, @10:57PM
I agree that publishing all the data would be best, but in human studies it is possible you will run into patient confidentiality problems. At the very least there would be extra work involved in anonymizing patient data. (however, any animal based studies should be regarded as very suspect if they don't make the full dataset available. )
No problem is insoluble, but at Ksp = 2.943×10−25 Mercury Sulphide comes close.
(Score: 0) by Anonymous Coward on Saturday February 04 2017, @09:56PM
I think I've only ever seen one use Welch's t-test, which actually does account for this.
The t-test that R uses by default is the Welch's t-test. I think there is not even an useful case where to use a Student's t-test above the Welch's t-test.
In many biological cases one would do an ANOVA anyway, instead of multiple t-tests.
(Score: 0) by Anonymous Coward on Saturday February 04 2017, @08:43PM
What you stated is semi true. It is why this gives me a chuckle every time. https://www.youtube.com/watch?v=RjzC1Dgh17A [youtube.com]
(Score: 0) by Anonymous Coward on Saturday February 04 2017, @11:57AM
Thank you greedy journals!
(Score: 1, Funny) by Anonymous Coward on Saturday February 04 2017, @02:01PM
This can't be right. We all know that its only the "soft" sciences that aren't reproducible!
Because they aren't even real science!
(Score: 0) by Anonymous Coward on Saturday February 04 2017, @08:54PM
Medical research is soft science.
(Score: 0) by Anonymous Coward on Saturday February 04 2017, @09:37PM
You are soft science.
(Score: 0) by Anonymous Coward on Saturday February 04 2017, @03:28PM
This is why we need businesses to run our science labs. Government can't do anything right. Reproducible cancer treatments? Pffft give me 10000 flavors of chewing gum and a squeegee mop that holds more water ANY DAY OF THE WEEK. Sad. Disgusting.