In February, the White House announced its plan to put $1 billion toward a similar objective—a "Cancer Moonshot" aimed at making research more techy and efficient. But recent studies of the research enterprise reveal a more confounding issue, and one that won't be solved with bigger grants and increasingly disruptive attitudes. The deeper problem is that much of cancer research in the lab—maybe even most of it—simply can't be trusted. The data are corrupt. The findings are unstable. The science doesn't work.
In other words, we face a replication crisis in the field of biomedicine, not unlike the one we've seen in psychology but with far more dire implications. Sloppy data analysis, contaminated lab materials, and poor experimental design all contribute to the problem. Last summer, Leonard P. Freedman, a scientist who worked for years in both academia and big pharma, published a paper with two colleagues on "the economics of reproducibility in preclinical research." After reviewing the estimated prevalence of each of these flaws and fault-lines in biomedical literature, Freedman and his co-authors guessed that fully half of all results rest on shaky ground, and might not be replicable in other labs. These cancer studies don't merely fail to find a cure; they might not offer any useful data whatsoever. Given current U.S. spending habits, the resulting waste amounts to more than $28 billion. That's two dozen Cancer Moonshots misfired in every single year. That's 100 squandered internet tycoons.
Related Stories
U.S. Vice President Joe Biden has released another cancer progress report/wish list:
Vice President Joe Biden today released his vision for doubling progress against cancer over 5 years. It includes numerous policy recommendations and a laundry list of projects by the National Cancer Institute (NCI) and other federal agencies that would require additional funding. Biden and his wife, Jill, have met with thousands of experts and patient advocates, they explain in a 17-page strategic plan submitted to President Barack Obama, who asked Biden in January to lead the effort. "We sought to better understand and break down the silos and stovepipes that prevent sharing of information and impede advances in cancer research and treatment, while building a focused and coordinated effort at home and abroad," they explain.
The Bidens' wish list ranges from giving patients more control over their medical data to launching "a national conversation" about cancer drug pricing. They also want to see more high-risk research funding at NCI and changes to the institute's intramural research program to focus more on emerging science and major public health challenges.
An accompanying 29-page report from Biden's federal moonshot task force lists what agencies have done so far and their plans to address five strategic goals. The first goal, "catalyze new scientific breakthroughs," contains several items that "align with" the 10 research areas recommended last month by an NCI blue ribbon panel, the report says. For example, it describes Department of Defense (DOD) efforts to develop new imaging technologies for detecting early molecular changes in cells that may lead to cancer.
The Cancer Moonshot is part of the War on Cancer:
The mission of this Cancer Moonshot is not to start another war on cancer, but to win the one President Nixon declared in 1971. At that time, we didn't have the army organized, didn't have the military intelligence to know the enemy well, and therefore didn't have the comprehensive strategy needed to launch a successful attack—now we do. Because of the progress over the last 45 years we have an army of researchers and oncologists, the powerful technologies and weapons, and immense public support and commitment to action.
Related:
Biomedicine Facing a Worse Replication Crisis Than the One Plaguing Psychology
The White House Announces $121 Million Microbiome Initiative
"Cancer Moonshot" Releases Blue Ribbon Panel Report
Microsoft to "Solve the Problem of Cancer" Within Ten Years - Scientists are Skeptical
(Score: 2) by aristarchus on Thursday April 21 2016, @08:10AM
The data are corrupt. The findings are unstable. The science doesn't work.
"Where have you gone, Joe DiMaggio? The nation turns its lonely eyes to you!"
Have we had enough of the "Science Bad" posts yet? What is next? Irreproducibility in Climate Vaxing!!! Horrors.
(Score: 2, Insightful) by khallow on Thursday April 21 2016, @09:45AM
Have we had enough of the "Science Bad" posts yet?
Perhaps for me, but clearly not for thee. Where in this story does it imply (since it is not said!) "science bad"? Perhaps you should pay more attention?
I think the take home message from studies like this is that we are dealing with a fundamentally broken approach to scientific research. It's not that somehow it would be bad to cure cancer or any of the other noble goals that scientific research claims to strives for and frequently does.
I think we're seeing the results of lack of accountability in research, particularly, but not exclusively from publicly funded research. Too often the appearance of doing or funding research is more important than the outcome.
(Score: 2) by Immerman on Thursday April 21 2016, @01:11PM
I suspect it's a problem even more specific (at least in degree) to the medical fields. Doctors being notorious for having a poor grasp of physics and scientific rigor. Not that that's terribly surprising - they're not trained to be scientists, they're trained to be mechanics for probably of the most sophisticated and poorly understood mechanisms in the realm of human concern. To hold massive reams of data in their head of the many, many things we know can go wrong with the human body or mind, with at least enough detail to be able to offer a decent preliminary diagnosis from often poorly described symptoms, and be able to locate more detailed information, and then apply it to hopefully deliver an effective cure in a squishy inconstant world where you can practically guarantee your patient will ignore at least half of your orders.
It's a valuable skillset, just not one with all that much relevance to performing reproducible science.
(Score: 1) by khallow on Thursday April 21 2016, @03:15PM
I suspect it's a problem even more specific (at least in degree) to the medical fields.
Seems an appropriate time to bring up ITER [wikipedia.org]. The problem here isn't just the irreproducible or the fraudulent research (both which occur in the hard sciences too), but that research is not outcome driven. ITER will have some relevance to what we actually need, a commercially viable fusion power plant. But so did construction of the Taj Mahal have some relevance to the design and construction of low income housing in India of the time. Projects like ITER are first and foremost, status signalling with the scientific value of the research coming in a distant second.
What should be noted here is that we've gone from psychology, where the science has been notoriously weak, to biomedicine which is considerably more grounded. Still the problems manifest strongly. I think it's foolish to dismiss this as a problem of a field which has weak scientific experience. No field is above such corruption.
(Score: 2) by Immerman on Thursday April 21 2016, @10:26PM
I don't see the connection. Yes, science is largely status drive, has been for far longer than formal science existed.
ITER though - that's pretty much what's needed to advance fusion research along what has become the widely accepted idea of what is required for fusion. After the Farnsworth Fusor design was abandoned after long failures to coax it to unity, tokamaks were the most promising alternative. Other techniques have since been developed on the fringes, but science has always been a popularity game, and once the majority accept something as The Way Things Are, it usually takes a generational shift to move on. Especially as investments grow large, nobody wants to be the one to admit that all those resources were likely wasted, and we should abandon them to direct future funding elsewhere. It's damnably hard to even convince folks to spend a tiny fraction of available funds in other directions to hedge their bets. The Polywell folks had to creep forward on a shoestring Navy budget for years, and probably only got that because tokamaks will be far too large to be suitable for most military purposes.
(Score: 1) by khallow on Friday April 22 2016, @12:31AM
ITER though - that's pretty much what's needed to advance fusion research along what has become the widely accepted idea of what is required for fusion.
And obviously, I disagree. The key problem here is that because the designers of the reactor have well over ten billion USD to throw at it, they have chosen a number of expensive options that a commercial reactor would have to avoid in order to keep costs down. A key one is using helium-cooled superconductors (increased costs through both much higher cooling costs and very custom parts). A better approach IMHO would have been a larger spatial scale vacuum chamber combined with nitrogen-cooled high temperature superconductors. They also have combined two significantly different research goals, research into plasma physics with sinking a lot of resources into a particular prototype. Among other things, it limits what they can do (the reactor is expensive enough that they can't reconfigure it as another geometry, such as a stellarator, without blowing their budget).
Another thing is that such massive fusion projects have blocked for decades similar research projects such as the Polywell reactor you mentioned or thorium fission reactors. As to your complaint that you "don't see the connection", how can you explain how tokamak fusion ended up with only one large project and a few measly alternatives? I believe it is readily explained by my earlier posts. They are just as subject to these problems and temptations as anyone else. I doubt, for example, that you'll lose money betting that ITER research will show the need for another one-off, expensive fusion reactor with no practical application to commercial electricity production.
(Score: 3, Insightful) by Immerman on Friday April 22 2016, @04:14AM
For the first paragraph I will say that, in general, when building *research equipment* you want to deal with "incidental" technologies that are as well-tested and fault-tolerant as possible in order to avoid unexpected problems. Perhaps there are high-temperature superconductors now that meet those criteria, I haven't really followed the field closely, but I know when I was a technician involved in superconducter research ~15 years ago, pretty much everything was helium-cooled, nitrogen was only used for pre-cooling and in the outermost layers.
Also, cost optimization, like performance optimization for programming, is something generally better done *after* you have something that actually works. Especially if someone else is footing the bill :-/
And of course in academia, as in most bureaucracies you also have the "spend all your budget or it will be cut next funding interval" mentality - but frankly I don't see any way to get away from that unless/until the people controlling the purse strings stop being such short-sighted idiots. You can hardly blame people doing the actual work for trying to avoid the incredible vanishing budget (And fusion research has a long history of vanishing budgets)
As for tokamak being the only large project - large projects are expensive, and it's often difficult to secure sufficient funding for even one of them. There was only one Manhattan Project for example (well, per side). Only one US Moon landing strategy attempted, etc. First prove it can be done, *then* worry about doing it well. As for the smaller projects - once you get people acclimated to the idea that "doing X costs lots of money", anyone claiming to be able to do essentially the same thing for orders of magnitude less tends to look like a kook. Would you pay someone $5,000 to build you a prototype car that they claimed would compare favorably to a $500,000 supercar and be ready in a few years? Probably not unless you had money to burn. If it were possible, someone else would already be doing it, right? It's not entirely rational, but that's human reasoning for you.
And yeah, the tokamak folks are probably partly to blame as well - their egos and careers are wrapped up in tokamaks being the solution - a lot of them are obviously going to trash-talk the competition. A lot of them probably even believe it.
(Score: 1) by khallow on Friday April 22 2016, @04:51AM
For the first paragraph I will say that, in general, when building *research equipment* you want to deal with "incidental" technologies that are as well-tested and fault-tolerant as possible in order to avoid unexpected problems. Perhaps there are high-temperature superconductors now that meet those criteria, I haven't really followed the field closely, but I know when I was a technician involved in superconducter research ~15 years ago, pretty much everything was helium-cooled, nitrogen was only used for pre-cooling and in the outermost layers.
Point is that handling liquid helium is more costly, difficult, and dangerous than handling liquid nitrogen (which as you note, you'll need to handle anyway). In addition, you'll need exotic materials and techniques that can operate under those conditions or intermediate, possibly fluctuating conditions.
Also, cost optimization, like performance optimization for programming, is something generally better done *after* you have something that actually works. Especially if someone else is footing the bill :-/
We already know fusion works. We have the Sun. But building a prototype that is so cutting edge that you'll have to abandon most of the secondary technologies just to get a commercially viable reactor, isn't helpful. Not every power plant operator is going to want to spend a few billion just to use large amounts of liquid helium when they could be using much cheaper cooling technologies.
unless/until the people controlling the purse strings stop being such short-sighted idiots.
Keep in mind that a primary deliverable for a politician supporting ITER are the press releases and the distribution of the funding to the right parties and the right locales. The actual success or failure of the program is beyond their planning horizon. Businesses aren't the only ones with short term thinking here.
We don't see fraud or irreproducibility with efforts like ITER, but we do see similar outcome-disconnect problems such as a remarkably low expectation for the success of the project and the schedule is similarly remarkably diffident (they're not even going to turn ITER on till 2020).
Would you pay someone $5,000 to build you a prototype car that they claimed would compare favorably to a $500,000 supercar and be ready in a few years? Probably not unless you had money to burn. If it were possible, someone else would already be doing it, right? It's not entirely rational, but that's human reasoning for you.
It helps that a) I could evaluate the design in question, and b) would seek to complete the project successfully in a reasonable time frame. If I were a politicians not so inclined, then well, why spend money more than once? I won't get more votes than I'm already getting. The result is quite rational, if more than a bit unscientific.
In summary, this is yet another example of scientific research with little incentive to produce useful scientific or engineering output. And it shows in unusable design choices and disinterest in reasonable costs or deadlines. IMHO that makes ITER a good example of what happens when you disconnect research from accountability or evaluation. It becomes theater with minimal impact on our future.
(Score: 2) by Immerman on Friday April 22 2016, @03:30PM
>point is that handling liquid helium is more costly, difficult, and dangerous...
Irrelevant unless high-temperature superconducters exist with a proven track record of providing the performance and stability needed (which I can't don't know). It's a far, far bigger waste of money if you build the thing based on more affordable but unproven technology, and then have to rip it all out and re-engineer the whole damned thing because your magnets can't get the job done. It's not like superconductor technology is standing still - by the time we get around to building commercial reactors high-temperature superconducters will have matured considerably from where they are today.
>We already know fusion works. We have the Sun.
Yes. But we can't do it that way, because we don't have mindbogglingly immense gravitational containment and compression systems. We can't even do the same sort of fusion (H-H) because the energies required are thousands of times greater than we can currently hope to generate. We're having to fake it with magnets, easier reactions(D-T), and clever trickery instead. And even then the field strengths we need are near the limit of what we're currently capable of creating.
Yes, there's lots of the secondary technologies that will probably need to be improved before commercial applications become feasible, but they're mostly improving independently of ITER. Plus, we don't actually *know* what exactly will be needed to get stable net-gain fusion working, all we have is theory and imperfect simulations. So we overbuild the test reactor so that we can almost certainly make it work, even if we underestimate just what it will take. And then we use that excess design capacity to explore the limits, and hopefully find ways to leverage excess capacity in cheaper systems to allow for reduced capacity in the more expensive ones. This is an experiment pushing the bounds of human knowledge and capability, it's designed to be tinkered with to explore the problem space, NOT to do things as cheaply as possible.
I won't argue the politics side of things. Politics tends to be stupid, wasteful, and destructive. I think we can agree on that. The scientists involved do the best they can to accomplish their goals* under the political restraints imposed upon them. * (build a working reactor - you don't go into science for the piss-poor salaries - if you're smart enough to be working on one of the worlds leading-edge scientific projects, you could be doing far better financially in any number of other careers with far less accountability. The project itself may have poor accountability, but the careers of the individual scientists tends to rise and fall on their accomplishments)
>It helps that a) I could evaluate the design in question, and b) would seek to complete the project successfully in a reasonable time frame
Except that, as a politician making decisions where fusion is concerned
a) you can't meaningfully evaluate the design. You spent your life learning politics, and have, at best, a hobbyist's knowledge of nuclear physics, and your eyes probably glaze over as soon as anyone starts talking about the details. Instead you can only base your decision on the advice of your advisors (whose eyes probably also glaze over), and the consensus claims of the nuclear physics community.
b) The project is large enough that the "reasonable time frame" required is liable to be measured in decades. Consider just how much time and money it takes to design a modern military aircraft, and then consider that a fusion reactor is far, far more complicated, difficult, and poorly understood.
(Score: 1) by khallow on Friday April 22 2016, @04:00PM
Irrelevant unless high-temperature superconducters exist with a proven track record of providing the performance and stability needed (which I can't don't know). It's a far, far bigger waste of money if you build the thing based on more affordable but unproven technology, and then have to rip it all out and re-engineer the whole damned thing because your magnets can't get the job done. It's not like superconductor technology is standing still - by the time we get around to building commercial reactors high-temperature superconducters will have matured considerably from where they are today.
Don't spend so much on it and then it doesn't become a big deal if you have to start from scratch. Further, who is going to do those necessary experiments with high temperature superconductors, if not ITER?
Yes, there's lots of the secondary technologies that will probably need to be improved before commercial applications become feasible, but they're mostly improving independently of ITER. Plus, we don't actually *know* what exactly will be needed to get stable net-gain fusion working, all we have is theory and imperfect simulations. So we overbuild the test reactor so that we can almost certainly make it work, even if we underestimate just what it will take. And then we use that excess design capacity to explore the limits, and hopefully find ways to leverage excess capacity in cheaper systems to allow for reduced capacity in the more expensive ones. This is an experiment pushing the bounds of human knowledge and capability, it's designed to be tinkered with to explore the problem space, NOT to do things as cheaply as possible.
And I've already stated my opinion of things that push bounds of human knowledge that aren't useful to us rather than bounds of human knowledge that are.
Except that, as a politician making decisions where fusion is concerned
a) you can't meaningfully evaluate the design. You spent your life learning politics, and have, at best, a hobbyist's knowledge of nuclear physics, and your eyes probably glaze over as soon as anyone starts talking about the details. Instead you can only base your decision on the advice of your advisors (whose eyes probably also glaze over), and the consensus claims of the nuclear physics community.
b) The project is large enough that the "reasonable time frame" required is liable to be measured in decades. Consider just how much time and money it takes to design a modern military aircraft, and then consider that a fusion reactor is far, far more complicated, difficult, and poorly understood.
And c) You don't care. ITER's success or failure will happen after you are long gone. A sexy experiment like ITER buys votes and campaign contributions in the now.
(Score: 2) by Immerman on Friday April 22 2016, @04:40PM
>Don't spend so much on it and then it doesn't become a big deal
Except it's not like you can just go in and replace the magnets, they're tightly integrated into the rest of the system, and replacing them means would mean tearing the whole damned thing apart - starting over from scratch would probably be faster and cheaper. And I really doubt the cooling system is anywhere near the majority of the total reactor cost. And don't forget the timescales involved - start over and some of the most important researchers may be dead or retired before they even begin testing the thing. How might Polywell research have progressed if the driving vision of Dr. Bussard hadn't been lost just as they were gaining real momentum?
As for who else is going to do the high temperature superconductor experiments - LOTS of people - they'd be incredibly useful for lots of different things. MRI machines for example - there you've got a well-understood and reasonably mature technology typically using helium cooled superconductors, moving to high temperature superconductors would be an incremental upgrade, rather than basing an already massively expensive long-term project on unproven technology. Conduct one experiment at a time.
>And I've already stated my opinion of things that push bounds of human knowledge that aren't useful to us rather than bounds of human knowledge that are.
How is better understanding the problem space of fusion reactors not useful? Perhaps tokamaks aren't the way to go, but as long as we're pursuing them at all, we should do it right.
>c)
Sure. That's politics. Like I said, I agree it's stupid and wasteful. But better to waste $4B on an overbuilt fusion reactor research project that is fairly certain to at least teach us some interesting things about, than $400B on an overbuilt and increasingly useless looking F-35.
(Score: 1) by khallow on Friday April 22 2016, @07:54PM
Except it's not like you can just go in and replace the magnets, they're tightly integrated into the rest of the system, and replacing them means would mean tearing the whole damned thing apart - starting over from scratch would probably be faster and cheaper. And I really doubt the cooling system is anywhere near the majority of the total reactor cost. And don't forget the timescales involved - start over and some of the most important researchers may be dead or retired before they even begin testing the thing. How might Polywell research have progressed if the driving vision of Dr. Bussard hadn't been lost just as they were gaining real momentum?
Eh, parasites are frequently hard to remove. ITER will happen no matter what. But maybe by 2025-2030, when the next generation to ITER is being started, we'll have commercial alternatives available to short circuit this destructive cycle.
Sure. That's politics. Like I said, I agree it's stupid and wasteful. But better to waste $4B on an overbuilt fusion reactor research project that is fairly certain to at least teach us some interesting things about, than $400B on an overbuilt and increasingly useless looking F-35.
Sure, it can always get worse. IMHO, the US military procurement mess is where publicly funded research thoughout the developed world is heading. They have accountability processes and clear goals yet things still got massively messed up. Incidentally, ITER supposedly is up to $14B these days though I imagine that depends on the exchange rate for the dollar.
(Score: 2) by aristarchus on Sunday April 24 2016, @04:22AM
"Coo coo kachoo, Mrs. Robinson,
Jesus loves you more than you will know.
God bless you, please, Mrs. Robinson.
Heaven holds a place for those who pray,
Hey, hey, hey."
Simon and Garfunkel
(Score: 0) by Anonymous Coward on Thursday April 21 2016, @10:43AM
Just because someone is doing research doesn't make it science. If you have no useable methods section I don't see how that could be science...
(Score: 0) by Anonymous Coward on Thursday April 21 2016, @08:24AM
Just get paid to publish an endless series of papers "reviewing the literature" with different co-authors each time. You could make a career out of it!
(Score: 5, Insightful) by moondrake on Thursday April 21 2016, @08:33AM
Its not only biomedicine. I think the same issue having effect on most of todays science, including in my own field.
Ironically, it is exactly these "Cancer Moonshots" multi-million dollar projects that fuel the problem. Its easy to blame the people, and its completely correct that it is the scientist themselves who are being sloppy or simply not honest. However, it is the atmosphere created by the way funding is handled that the people behave this way. No amount of "ethics" classes and projects (lots of $$$ being wasted there) for young scientist is going to help. Ironically, politics tries to "fix" the problem by making more funding available for "excellent" scientist working on "key" ("key" usually meaning its an over-hyped project with supposedly direct benefits for society or of direct importance for it). By doing this, they automatically select for people that are good at selling their idea, and producing the most papers in high-ranking journals. Most of them will do this by promising golden mountains and being sloppy or simply lying.
If there are any other good scientist that work years on a not over-hyped project that might change the world in 50 years, they are simply without funding in the short time and without a job in the long term.
Every time you read in the paper that important Prof X got Y million for his very important project on saving the world, you know that millions got wasted an lots of young PhD students are without a job in a few years because there is no money to pay them.
(Score: 0) by Anonymous Coward on Thursday April 21 2016, @08:39AM
If there are any other good scientist that work years on a not over-hyped project that might change the world in 50 years, they are simply without funding in the short time and without a job in the long term.
the bad news is science has peaked.
but the good news is there's gigabit wifi and plenty of porn to watch.
so to hell with the world.
(Score: 0) by Anonymous Coward on Thursday April 21 2016, @08:41AM
This is why science is best when done as an application of R&D for a business that depends on its results.
(Score: 2, Insightful) by Anonymous Coward on Thursday April 21 2016, @08:53AM
When science is done by business, no one will ever go to the moon, because money is made by pointing spy satellites at the earth for targeted advertising.
(Score: 1) by khallow on Thursday April 21 2016, @09:10AM
When science is done by business, no one will ever go to the moon, because money is made by pointing spy satellites at the earth for targeted advertising.
As opposed to the half dozen times when it was tried the other way? Doesn't sound like an improvement to me. When someone speaks of "moonshots", they really are speaking of incredible waste to achieve a goal which is abandoned as soon as it is achieved.
(Score: 0) by Anonymous Coward on Thursday April 21 2016, @09:19AM
Well let's see now what was the goal of the space race again? To bankrupt the USSR? Too bad it didn't work as planned. It took Afghanistan to bankrupt the USSR. Afghanistan is getting very good at destroying evil empires. Which evil empire is currently wasting money on a pointless war in Afghanistan right now? It sure is taking a while but the writing is on the wall for the USA.
(Score: 1) by khallow on Thursday April 21 2016, @10:10AM
Well let's see now what was the goal of the space race again? To bankrupt the USSR? Too bad it didn't work as planned.
Was that goal met? Sounds like you imply it wasn't.
(Score: -1, Flamebait) by Anonymous Coward on Thursday April 21 2016, @11:27AM
When science is done by business, no one will ever go to the moon, because money is made by pointing spy satellites at the earth for targeted advertising.
You're an idiot. We only went to the moon because we needed to perfect our ICBM engines, which we also sold to our allies. When that job was done, we stopped going to the moon.
(Score: 2) by CirclesInSand on Friday April 22 2016, @08:55AM
So what if no one goes to the moon? It is a vanity project. The public shouldn't be involuntarily funding a project with no benefit whatsoever. It's like collecting a tax to build a cathedral. If you love it so much, create or join a group dedicated to the effort.
(Score: 4, Interesting) by JoeMerchant on Thursday April 21 2016, @12:50PM
The root cause of the problem is the relative value of "original research," which is too high relative to the value of replication studies.
Debunking studies seem to be mostly funded by people who would suffer from the new science (established industry) while replication studies are funded mostly by the IP holders. It's a fundamentally flawed incentive system.
If we had a community of academics who were funded regardless of the outcomes of their studies, things would be better. It would seem that the rising cost of higher education could more than foot the bill for this, but somehow those profits are being channeled elsewhere. Perhaps "institutions of higher learning" need a new taxing structure that encourages them to grant more tenure, and provide total transparency in the funding of their research. Not more or less tax, revenue neutral, but shift the tax burden to institutions that place more value on their endowment and athletics programs, and less burden on those who employ independent scientists who publish unbiased research.
🌻🌻🌻🌻 [google.com]
(Score: 1) by khallow on Thursday April 21 2016, @04:17PM
If we had a community of academics who were funded regardless of the outcomes of their studies, things would be better.
Which community of academics counts? I'm sure the Scientologists could throw together a community of academics, assuming generously that they haven't already. If you throw money at everyone who claims to be an academic without consideration of the consequences, then I suspect strongly that your money won't go very far.
(Score: 2) by JoeMerchant on Thursday April 21 2016, @10:16PM
If you manage to get a minimum of 5-6 replications of any important studies, those scientists that match the majority - the majority of the time, will be considered of high standing, reflect well on their employing organization, and the tax benefits can follow that.
Junk scientists who spout stuff that nobody else can reproduce even when they try lose standing, funding, etc.
It's not hard to judge, if there are enough respected opinions weighing in. The problem today is that there aren't many opinions left worth respecting. That needs to be restored.
🌻🌻🌻🌻 [google.com]
(Score: 1) by khallow on Friday April 22 2016, @12:33AM
If you manage to get a minimum of 5-6 replications of any important studies, those scientists that match the majority - the majority of the time, will be considered of high standing, reflect well on their employing organization, and the tax benefits can follow that.
What happens when the 5-6 replications by the Scientologist groups disagree with the 1 replication by the non-Scientologist research group?
(Score: 2) by JoeMerchant on Friday April 22 2016, @03:05AM
>What happens when the 5-6 replications by the Scientologist groups disagree with the 1 replication by the non-Scientologist research group?
That's where you start basing reputation on sponsoring organization in addition to individual groups. If the sponsoring organization is frequently an outlier, that diminishes not only their standing, but the standing of all research groups sponsored by them. Transparency will win out, if we actively pursue it and maintain a preponderance of unbiased research in fields that matter, instead of basing policy on the first publication of novel findings and public reaction to the news release.
In your specific case, if a topic only has one "legitimate" group studying it and 5-6 groups from organizations with an agenda to push - hopefully that topic is not of interest outside the organizations with the agendas. The sad state of today's so-called science is that often, industry and other interest groups will attempt, and succeed in saturating a field that should matter to people outside their interest group with biased research, making it very costly to overwhelm that research with a preponderance of unbiased studies.
🌻🌻🌻🌻 [google.com]
(Score: 2) by moondrake on Friday April 22 2016, @08:24AM
Politicians should trust a well organized system a bit. Do you thick the CEO of Google evaluates every janitor in the company? Departments in a company set goals and people are regularly evaluated. The local manager understands that not everybody needs to do the same job and some people have an important supportive job that it necessary for the end product.
A well organized institute or university should be able to hire different categories of people and have funding to do so. In fact, in many European universities this is already the case for education. I have met many people who more or less gave up in science, but got a permanent job because they like teaching. Why can't the same be true for academics. Instead, the current situation in many places is to only hire the "top 5%", the 5% being defined by very questionable statistics.
(Score: 1) by khallow on Friday April 22 2016, @02:48PM
Politicians should trust a well organized system a bit. Do you thick the CEO of Google evaluates every janitor in the company?
Thing is when the trash piles up in the bathroom, even the CEO can see that something is wrong and that maybe that trust has been misplaced somewhere.
A well organized institute or university should be able to hire different categories of people and have funding to do so.
They usually get that funding from their students, alumni donations, various R&D endeavors, sports activities, etc.
Instead, the current situation in many places is to only hire the "top 5%", the 5% being defined by very questionable statistics.
That doesn't sound very trustworthy to me. Everyone and everything has flaws, but in academia I see flaws that have grown over the past few decades which threaten to cripple the mission of academia. For example, there's a recent story [chronicle.com] where a professor agonizes over why they can't give a decent student a recommendation merely because twice they overheard the student speaking about firing or carrying a gun (the first time the student was talking about shooting an AK 47 over winter break and the second time about applying for a concealed carry permit which some US states allow). Further the professor admits that such gun activities are not unusual for region. Yet on that basis alone, the teacher has failed to write a recommendation for the student for a teaching credential program. The professor admits the student deserves the recommendation, but they have simply not done anything without telling the student.
What I found particularly telling about the episode was this:
Last year at some point, Sarah said she was applying to a teacher-credential program and asked me for a recommendation. Initially I said yes because I usually do. I don’t know the exact date she asked, but I am thinking it must have been before the Umpqua Community College shooting last October because that’s when I really started thinking about students and guns. After Umpqua, colleagues and others specifically asked me if I felt safe on the campus and I had to think about that question. Our college’s "shelter-in-place" drills — in which whole buildings practice for an active-shooter situation — have not made me feel safe. I also did not feel safe during a visit to the campus police station where I was offered a free gun-safety lock.
The college here institutionalizes the hysteria of the professor with such things as "shelter-in-place" drills, theater that don't even help people feel safer. Money was spent on that when it could have been spent on bettering their educational services or just simply charging students less. Meanwhile we have a teacher who has no doubt sat in on a lot of anti-discrimination training and yet still is willing to harm a student's future academic career on the basis of two overhead conversations which had no relevance to the student's fitness. This is just a small example. There is so much that is anti-intellectual about modern academics and I think that all ultimately stems from a huge aversion towards accountability and a deeply rooted hypocrisy following from that.
(Score: 2) by HiThere on Thursday April 21 2016, @09:24PM
What is needed is an independent group to do replication studies, and that their funding should not depend on whether they succeeded in replication. The problem is ensuring that THEY do good work. But clearly the original reports should include sufficient data to enable replication to be done. If such isn't provided, credit for the work should be denied. That would be a good first step.
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.
(Score: 2) by JoeMerchant on Friday April 22 2016, @03:09AM
If a topic is important, replication should be attempted a dozen times or more, by geographically, politically, and every other dimension possible, diverse groups. When a lot of groups work on a lot of studies, you can tell the good ones by how well they agree (on average) with their peers, not to mention how their peers rate their methods and publications.
🌻🌻🌻🌻 [google.com]
(Score: 3, Insightful) by bitstream on Thursday April 21 2016, @09:44AM
It's science when steered by careers and money. Instead of curiosity and interest. Something people with the budgets don't like. One could say they want this un-replicable science because they are paying for it. Ie premieres it.
People should really have a look at the persons that make paradigm breaking discoveries. They very often happen because a lot of other reasons than the lightning strike of funding gods.
There's a lot of problems that needs to be solved and having people with deep knowledge go work less is a big waste of time and opportunities for society at large.
(Score: 1, Insightful) by Anonymous Coward on Thursday April 21 2016, @10:56AM
The odd thing is that people defending the current system, who pretty much control all the research funds, aren't scrambling to start actually doing replications. Critics like this author would be shut up quick by all the successful replications, because there's no problem, right? It is almost like the emperor has no clothes, all highly placed stakeholders know how pseudosciency the whole thing has gotten and how their research careers are largely or wholly a sham.
(Score: 1, Interesting) by Anonymous Coward on Thursday April 21 2016, @11:52AM
It doesn't require a conspiracy to get this situation. Suppose you are deciding what project to pursue given some limited amount of funding. You can try something new and possibly carve out a topic no one else has tried before. The upside could be huge yielding years of work and funding to expand on your great new idea. Downside is having to go beg for money again, I.e. what research professors spend most of their time doing. Same old same old.
Alternatively you could take someone else's work and try to replicate what they did. Upside: proving them wrong and maybe getting a publication out of it. Downside, go beg for money again without being able to say you even tried something new. Unless the odds falsifying some data are significantly higher than of finding something new it makes more sense for your career to try the something new.
A rare falsification does not pay for a career and likely costs about as much as the original data. Add to that that when you're reviewing a paper for publication, again there are limited resources (publication space in good journals) so new stuff will get published and the falsification will get a footnote. Unless the falsified finding was very important proving it wrong just isn't that noteworthy.
Ideally larger established well funded research groups should be doing some replications as a service to the community, but a large research group just generally means more mouths to feed.
(Score: 0) by Anonymous Coward on Thursday April 21 2016, @12:38PM
The problem is with those who control the funds and those who are profiting off this situation by remaining. One of the reasons I quit was the extremely low probability anyone would try to independently verify my findings (sure, maybe with some twist added in so it was "new", but not actually try to make sure the findings were stable). What's the point in that case? It isn't science without replication. Assuming I am not alone in that decision, what kind of person remains?
(Score: 3, Informative) by sjames on Thursday April 21 2016, @05:36PM
That's the issue in a nutshell. It's no longer aboput the science, it's about scoring enough points to not find yourself pushed out. There's no glory (funding, career) in being the guy that provided the vital replication and clarification to a dozen studies, even if that clarification was the vital missing piece.
(Score: 0) by Anonymous Coward on Thursday April 21 2016, @01:09PM
Independent verification is often more important than replication. Cancer is complex and the systems to study it are full of caveats so reproducing a study lends less certainty of translation than performing a different study that addresses a similar question:
Initial study - small molecule AKT-signalling modulator reduces HeLa cancer cell growth, in vitro.
Knockdown of PI3K reduces cancer growth and reduces tumor seeding in an immunodeficient mouse.
A broadly nonspecific PI3K inhibitor reduces tumor growth in an immunodeficient mouse.
A broadly nonspecific PI3K inhibitor does not reduce tumor growth in an immunocompetent mouse with spontaneous tumor formation.
Specific PI3K isoform inhibitor reduces tumor growth in a non-spontaneous immunocompetent mouse tumor model.
PI3K-defficient T cells are unable to control tumor growth.
Basal-type breast cancer tumors are more sensitive to PI3K inhibition than luminal-type.
PI3K is not important for tumor epithelial-to-mesenchymal transition.
PI3K inhibitor show no reduction of tumor size in breast cancer patients.
PI3K inhibitor shows survival benefit in patients with triple-negative breast cancer.
PI3K inhibitor reduces tumor metastasis in patients with triple-negative breast cancer but decreases tumor-free survival in patients with ER-positive breast cancer.
(Score: 0) by Anonymous Coward on Thursday April 21 2016, @01:42PM
Can you sum up what you believe has been learned from that list? Also, if we are going to be specific, better include the strain of mouse, dosage, race/age/etc distribution of the clinical patients, the lab that did it, etc. There is no cumulative knowlege being generated, just a set of disconnected observations that lack verification. Is it even worth coming up with a real theory attempting to piece together such unverified info? Not to me, even if I had been responsible for one or two pieces myself.
Essentially, I'd like people to treat areas of research like this with the same skepticism they treat EM drive idea.
(Score: 0) by Anonymous Coward on Thursday April 21 2016, @02:15PM
Can you sum up what you believe has been learned from that list?
Not really but that is the point, which you seem to understand:
better include the strain of mouse, dosage, race/age/etc distribution of the clinical patients
Studies will observe certain facts that were generated under a specific set of conditions. The farther you get away from those conditions then the less certain any conclusions become.
From the previous post, you could attempt to rationalize that: broad PI3K inhibition will reduce tumor growth and immune-mediated control of tumor growth while a specific subtype of breast cancer needs the activity of a specific PI3K isoform, which is not needed by the immune system, to undergo a non-EMT mediated metastasis. Obviously, all the rationalization could be misguided by experimental artifacts and the clinical effect could be caused by a metabolic product of the drug that acts through some unrelated pathway.
(Score: 3, Informative) by shrewdsheep on Thursday April 21 2016, @03:09PM
Being in the field, I do not trust most research findings, nor should I. On the other hand this does not imply the science is broken. There are a lot of reasons why replication fails, low power, publication bias, and confounding being the most important ones. However, keep in mind that most papers are re-utilizing data that has been generated, waste data, if you will. This is dictated by the system of publish-or-perish and those papers should be considered as progress reports of the scientists. This is exploratory research and proper control of statistical properties (such as type I error) is not warranted.
There examples, when it works. Genome wide association studies are one example, where replication arguably does work. Clinical trials are another one and the regulation around clinical trials is a good safeguard for replicable results unless conducted fraudulently (which has happened).
Standards of research could certainly be higher overall that much is certain, however the arguments concerning replicability such as TFA (TL;DR) are most of the time strawman arguments.
(Score: 2, Touché) by khallow on Thursday April 21 2016, @04:21PM
On the other hand this does not imply the science is broken.
Your lack of trust doesn't necessarily imply the science is broken. But a huge amount of irreproducible research does.
(Score: 2) by moondrake on Friday April 22 2016, @08:44AM
I am a bit confused as to the purpose of your post.
>Being in the field, I do not trust most research findings, nor should I. On the other hand this does not imply the science is broken.
As a sibling mentioned, no, trust is not a problem. We should always think critically, but this was not the topic of the discussion. It is claimed that our main source of background information is extremely unreliable. Arguably, we would be able to do our job better if it was more reliable. You can debate whether this is a bug (broken) or a feature-request (this brings tears on many a bugzilla as well). I would really like to see though if it is a regression and whether biological data from 40 years ago was more reliable. Sadly, I am not aware of such a study.
The reasons for failure of replication you name are all valid, and you seem to think better standards are needed. If this is not to improve the replicabilty of results, then what for? Or how do we even judge whether standards have an effect?
(Score: 2) by shrewdsheep on Friday April 22 2016, @09:01AM
My first and main point was that there is not really a problem. There are valueless papers, there is fraud, but undoubtedly there is scientific progress. Exploratory result can be extremely beneficial, too, to generate hypotheses.
My afterthought was that the discussion if often misguided. The real discussion should be IMO about funding. Funding is usually tied to the publication record of applicants. By and large that seems to work as people bending their research are weeded out eventually (most of the time). However, bigger projects or mega projects (solve cancer, simulate the brain) have to be handled better than they are. It takes more than a couple of well received papers to handle such projects and review should be broader (not just publications), out-of-contry/continent, and contingent to fulfillment of milestones. A lot of money gets wasted in these projects but this seems difficult to solve (as other areas show: infrastructure, military, corporate projects, ...).
(Score: 0) by Anonymous Coward on Friday April 22 2016, @01:45PM
"people bending their research are weeded out eventually (most of the time)"
Not from what I've seen, nor as described in this article. Also, besides people bending their research, there is now a huge population doing research who never really figured out science. They just don't know any better nor realize the damage being caused, like puppies wagging their tails and knocking everything off the table.
(Score: 1, Informative) by Anonymous Coward on Thursday April 21 2016, @03:58PM
If anyone is interested, the other paper mentioned in the summary is available open access here:
http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002165 [plos.org]