An ambitious project that set out nearly 5 years ago to replicate experiments from 50 high-impact cancer biology papers, but gradually shrank that number, now expects to complete just 18 studies.
"I wish we could have done more," says biologist Tim Errington, who runs the project from the Center for Open Science in Charlottesville, Virginia. But, he adds, "There is an element of not truly understanding how challenging it is until you do a project like this."
[...] Costs rose and delays ensued as organizers realized they needed more information and materials from the original authors; a decision to have the proposed replications peer reviewed also added time. Organizers whittled the list of papers to 37 in late 2015, then to 29 by January 2017. In the past few months, they decided to discontinue 38% or 11 of the ongoing replications, Errington says. (Elizabeth Iorns, president of Science Exchange, says total costs for the 18 completed studies averaged about $60,000, including two high-priced "outliers.")
One reason for cutting off some replications was that it was taking too long to troubleshoot or optimize experiments to get meaningful results, Errington says. For example, deciding what density of cells to plate for an experiment required testing a range of cell densities. Although "these things happen in a lab naturally," Errington says, this work could have proceeded faster if methodological details had been included in the original papers. The project also spent a lot of time obtaining or remaking reagents such as cell lines and plasmids (DNA that is inserted into cells) that weren't available from the original labs.
[...] The project has already published replication results for 10 of the 18 studies in the journal eLife. The bottom line is mixed: Five were mostly repeatable, three were inconclusive, and two studies were negative, but the original findings have been confirmed by other labs. In fact, many of the initial 50 papers have been confirmed by other groups, as some of the Reproducibility Project: Cancer Biology's critics have pointed out.
http://www.sciencemag.org/news/2018/07/plan-replicate-50-high-impact-cancer-papers-shrinks-just-18
Interesting that it seems to be much easier for labs to do "secret protocol replications" than the "open protocol replications" aimed for by this project.
(Score: 0, Interesting) by Anonymous Coward on Tuesday August 07 2018, @03:34PM (8 children)
This is particularly the case when so much research is funded by taxation.
In contrast, you can't get away with nearly as much nonsense when you're funded solely by voluntary exchange.
(Score: 4, Interesting) by ikanreed on Tuesday August 07 2018, @03:38PM (5 children)
No, then you just die of cancer because all the basic biomedical research is trade secrets not be divulged just in case one of your competitors uses the information to actually make a treatment.
There's no joke. I mean, other than your ideology.
(Score: 1, Interesting) by Anonymous Coward on Tuesday August 07 2018, @03:47PM (3 children)
It sounds like you're the one with an ideology.
(Score: 3, Interesting) by ikanreed on Tuesday August 07 2018, @03:51PM (2 children)
I mean I am.
It's ideological to recognize the inherent flaws of one of the dumbest fucking belief systems in the modern world. That's true. I do not contest that I am affected by my ideology. Just... goddamn is yours dumb as hell.
(Score: 1, Funny) by Anonymous Coward on Tuesday August 07 2018, @04:00PM (1 child)
Yeah. I'm the one with the dumb ideas.
(Score: 3, Interesting) by ikanreed on Tuesday August 07 2018, @04:12PM
Man, I thought after that first post, I'd be arguing with you, but you just keep making true statements.
(Score: 2, Interesting) by Anonymous Coward on Tuesday August 07 2018, @04:27PM
Pretty sure its the worst of both worlds in the US. You pay for it via taxes so its really inefficient AND all the good info gets locked up or kept secret. It wouldnt surprise me if somewhere in the 20 trillion dollars that the DoD cant account for theres some cure for cancer. I mean chemo and chemical weapons are pretty much the same thing.
(Score: 2) by DeathMonkey on Tuesday August 07 2018, @07:44PM (1 child)
In contrast, you can't get away with nearly as much nonsense when you're funded solely by voluntary exchange.
Capitalists would never fudge the numbers just to make more money!
(Score: 0) by Anonymous Coward on Wednesday August 08 2018, @02:29PM
Not only will consumers report to each other whether some service is working as well as expected, but the lack of forced funding tends to make consumers require increasingly drastic means by which to prove validity.
Capitalism is an iterative process.
(Score: 4, Funny) by ikanreed on Tuesday August 07 2018, @03:35PM
The replication tumor is in remission. With continued cashmotherapy, we think we can reduce the amount of metastatic verification to trace levels.
(Score: 3, Interesting) by FatPhil on Tuesday August 07 2018, @03:37PM (6 children)
=> Paper not replicable.
Next!
(can I have $60000 please?)
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 3, Interesting) by SubiculumHammer on Tuesday August 07 2018, @04:53PM (5 children)
It does not mean that at all Phil. Science is hard. There are many variables. Some methodological nuances are hard to describe in detail and are learned by practice, limitations on number of words of methods sections, and/or took expertise beyond the capabilities of the lab trying to replicate.
(Score: 3, Interesting) by Bot on Tuesday August 07 2018, @06:53PM (2 children)
Sure thing, but then it's NOT SCIENCE.
Too complex for protocols? Age of information: grab a webcam and DOCUMENT EVERYTHING from the setting up of the instruments to the procurement of needed shit. If you cannot replicate it, at least you document it.
We are talking about cancer, you know, those 30000$ a month treatments. You can spare 200$ for 4tb of hd video.
Account abandoned.
(Score: 0) by Anonymous Coward on Tuesday August 07 2018, @07:45PM (1 child)
How long of an appendix will that make when printed?
(Score: 2) by Bot on Wednesday August 08 2018, @02:47PM
You don't need to print everything, we are discussing the proof it does not work.
Account abandoned.
(Score: 2) by FatPhil on Tuesday August 07 2018, @11:12PM
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 1) by khallow on Wednesday August 08 2018, @05:53AM
And you base that on what? Turns out as a result of conveniently not having those methods, that research couldn't be falsified by the project.
Fraud can be hard too.