SoylentNews is people
SoylentNews
Richard Horton writes that a recent symposium on the reproducibility and reliability of biomedical research discussed one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with science (PDF), one of our greatest human creations. The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. According to Horton, editor-in-chief of The Lancet, a United Kingdom-based medical journal, the apparent endemicity of bad research behaviour is alarming. In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world or retrofit hypotheses to fit their data.
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors,. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
[Editor's Comment: Original Submission]
(Score: 4, Insightful) by kaszz on Tuesday May 26 2015, @06:39PM
Could it perhaps be related to that research people are pushed by department heads to publish or be crossed over in the paycheck list? Or that a publication track record may be an advantage when seeking funds?
Perhaps those "blue sky" project style management had something going? and that curiosity driven research has been lost since and become a industry printing publications?
Just some thoughts..
(Score: 4, Insightful) by VLM on Tuesday May 26 2015, @06:56PM
Which fits in as a result of the "graduate ten times as many PHDs are there are PHD level job openings". If you don't want the results of brutal cutthroat competition, then don't make a competitive situation. Academia really is messed up and I'm glad I didn't go down that path.
(Score: 3, Insightful) by Kell on Wednesday May 27 2015, @01:49AM
We here in academia are pushed to graduate more students. It's not that we want to take on inferior or subpar students and train them for a job they'll never get (and aren't capable of, anyway), but rather it's the case that more students bring more income into the university and god help you if you try to reject or flunk a full fee paying student. I've been straight up told that I need more PhD students by the time I get to tenure review.
Scientists ask questions. Engineers solve problems.
(Score: 0) by Anonymous Coward on Wednesday May 27 2015, @08:38AM
Maybe a better model would be to pay universities only if you failed. That would have a twofold effect: Universities would have to balance their failure rates (too little, and they don't generate income; too much, and they won't get any students), thus making passing the exam a meaningful measure against. On the other hand, it would give an extra incentive to the students to be good (because being good, they could save money), and to those who wouldn't have a chance pass anyway, to stay away (and thus improve the situation for those who do have a chance).
(Score: 4, Insightful) by Thexalon on Tuesday May 26 2015, @08:14PM
More to the point, "publish or perish" is all about attempting to quantify something that really isn't quantifiable, so that it can be turned into performance metrics that can then be used as an excuse to cut people and keep the salaries of scientists from getting too expensive from the point of view of college administrators (the administrators' salaries, of course, need to be increased dramatically).
As soon as you create a performance metric, smart people will find a way to fake it, guaranteed.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 0) by Anonymous Coward on Wednesday May 27 2015, @12:35PM
As soon as you create a performance metric, smart people will find a way to fake it, guaranteed.
Indeed and sometimes it really bites you https://en.wikipedia.org/wiki/Cobra_effect [wikipedia.org]
(Score: 0) by Anonymous Coward on Wednesday May 27 2015, @02:08PM
More to the point, "publish or perish" is all about attempting to quantify something that really isn't quantifiable, so that it can be turned into performance metrics that can then be used as an excuse to cut people and keep the salaries of scientists from getting too expensive from the point of view of college administrators (the administrators' salaries, of course, need to be increased dramatically).
As soon as you create a performance metric, smart people will find a way to fake it, guaranteed.
Without a metric, how would you determine which of the thousands of scientists should our limited resources be funneled to? If you agree to use a metric, which metric would be better than the admittedly terrible publish-or-perish model?
Put it this way. A random stranger comes up to you and says, "Give me money to study exothermaldynamics. No, I won't promise you any results or anything else. You can trust me I'm using this money well, I'm a scientist."
Maybe if you know this person personally you would trust them. Maybe if you were proficient enough in exothermaldynamics to be confident to determine a legitimate question from a pure money grab you would give them some money. But neither of those scale at all. So what is the better model we should be using?
(Score: 0) by Anonymous Coward on Wednesday May 27 2015, @03:25PM
Well, in the 1600-1700s science was done by people with wealthy and powerful patrons. The scientists would come over for dinner and entertain the guests. If the science was not interesting enough , the scientists would also get into feuds with each other which would still entertain and contribute to a kind of "science race" amongst the patrons.
I don't necessarily think that worked better, but it is one option.
(Score: 2) by aristarchus on Wednesday May 27 2015, @06:49PM
The scientists would come over for dinner and entertain the guests.
Yes, and it's all harmless fun, until somebody bursts their bladder! https://en.wikipedia.org/wiki/Tycho_Brahe [wikipedia.org]
(Score: 3, Insightful) by Thexalon on Wednesday May 27 2015, @03:31PM
Nobody is suggesting that. There's a fairly good way of vetting somebody who's trying to get grant money:
1. First, you check the random strangers' educational qualifications. For example, an exothermaldynamicist would be expected to either have a doctorate or be working on their dissertation. You would want them to have done well in their coursework, which you can get from their college transcripts.
2. Second, you check their previous work, if any. If they're somebody new working on their dissertation or something like that, then you'll understand them not having much of a record but you'll probably be a bit stingier with the grants.
3. Third, you ask the acknowledged experts of exothermaldynamics to see what they think of the proposal and the person who's proposing it. Answers like "total crackpot!" or "hmm, there might be something to that, it would be worth a try" should give you some good guidance.
4. Fourth, you get opinions on the random stranger from everybody who knows or has worked with them, particularly academic advisors and professors and such.
The fact that there are far more qualified scientists out there than there is funding for them is truly shameful, because it means that we're intentionally holding back the rate of scientific discovery due to a fear of losing small green pieces of paper.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 3, Insightful) by dcollins on Tuesday May 26 2015, @08:27PM
It's not just the department heads. As non-academic business people have taken over universities in the last few decades, they're the ones that keep arbitrarily tightening the screws in terms of increased number of publications in a shortened time frame for permanent employment. That stuff has basically been taken out of the hands of the academics at this point.
Immediate Example 1: This morning I received this month's "AFT On Campus" magazine with a cover story on "The pernicious effects of corporate influence" which is basically the same issue.
Immediate Example 2: Five minutes ago I got an email from faculty setting up a "workload concerns" committee to pushback on administration where I teach on the issue (for which I would not have high hopes).
http://www.aft.org/periodical/aft-campus [aft.org]
(Score: 1) by Placenta on Tuesday May 26 2015, @10:19PM
How have "non-academic business people" taken over colleges when college administration tends to be made of people with 30+ years in academia? These are people who entered college at 18, and never left. They have never worked in industry or business. This is even usually the case for those who studied and taught business!
(Score: 2, Interesting) by gamechanger on Tuesday May 26 2015, @11:43PM
The appointment of people out of industry is increasingly happening in senior executive roles in Australian universities though not in large numbers yet.
When academics are appointed to these roles, they tend to be people who have shifted to the "administrator" track from the researcher track much
earlier in their career than they did 30 or more years ago. Vice Chancellors used to be people who had had distinguished research careers and who took on the role
towards the end of their career. Now they are younger ambitious go-getters dazzled by the seven figure salary packages.
The pressure from department heads to publish results from pressure in the entire system to be high (and yet higher) in the university and discipline rankings
from Shanghai Jiao Tong, or Times, or QS or ... These in turn lead to more paying customers - international students. "Branding" is the mot du jour in
universities in Australia. This is what is driving science in Australia and I suspect elsewhere, but because of the enormous pressure for international students (and income),
I suspect the effect here is greater! And then you wonder why we are seeing bad science published?
(Score: 2) by Kell on Wednesday May 27 2015, @01:53AM
Hello fellow Aussie academic! I agree with all the points you've made - I've seen it too, myself. I seriously think about leaving academia every 6 months or so. If my field wasn't so small domestically in Australia, I'd probably have left long ago... As it is, I'm 9 months away from submitting for continuing. I can't even think of any other jobs that have a 5 year probationary period.
Scientists ask questions. Engineers solve problems.
(Score: 2) by dcollins on Wednesday May 27 2015, @02:42AM
Example: CUNY Board of Trustees -- 15 people who actually wield power over ~20 colleges in CUNY, the largest urban university in the nation. The first guy, Chairman Schmidt, you could call a lifelong academic -- albeit chairman of a global private school network (formerly Yale). From what I can see the rest are all non-academic industry people.
http://www.cuny.edu/about/trustees/board.html [cuny.edu]
Case 1: In the last few years the trustees took control of the CUNY curriculum and demanded a reduction in credit-hours for all degrees (so as to show increased graduation rates). After much shenanigans, a no-confidence vote was taken by faculty, which came in 92% "no confidence". The Board of Trustees basically said "we don't care" and forced its implementation anyway.
Case 2: Last year Chairman Schmidt, who is on Lynn Cheney's American Council of Trustees and Alumni (ACTA), headed a report recommending that even more functions be taken away from faculty and given to trustees selected from the business community. From the report: "faculty cannot be the last and determining voice regarding academic value, academic quality, and academic strategy... it is lay trustees – with considerable life and community experience – who can bring the big picture to bear in determining what graduates will need...".
http://www.psc-cuny.org/clarion/december-2014/benno-schmidt-backs-report-calling-trustees-reduce-faculty-authority [psc-cuny.org]
(Score: 0) by Anonymous Coward on Wednesday May 27 2015, @04:15AM
It's not just that, but the people who attend college and university are increasingly doing so to get goods jobs or make money. They don't care much about actual education or having an academic understanding of the universe. So, the business people start trying to get more people into colleges who have these motivations so they can take their money, and universities end up becoming half-assed trade schools to suit these fools.
(Score: 3, Insightful) by looorg on Tuesday May 26 2015, @08:31PM
Indeed. As long as my entire career, future and paycheck is tied to some kind of Publish or Perish-metric then that is what I and others are going to do. The only viable method is the shotgun-method; you get your name in on as many things as possible. Some will be boring duds that go straight to the archive and some will be okay. Some might even turn out to be really good. You could spend 10-20-30 years on your "THE ONE"-project but the personal risk vs reward for that one is really low, and possibly not even allowed anymore. I guess if my position was tenured and I was in my sixties and just didn't give a fuck anymore then sure I could do that. But until that time I'm doing what is going to possibly take me there. Is that sad? Yes. Do I wish I could do other things? Yes. Have I cancelled or reworked projects (sexed 'em up)? Sure. Why? Because when was the last time someone read a report where your entire model, data and project failed? Not going to happen. If you get grant money you succeed. If you can't succeed you massage the fuck out of that data until it looks like you did, or at least vaguely so.
(Score: 0) by Anonymous Coward on Tuesday May 26 2015, @08:51PM
Well I sure as hell don't massage my data.
But I agree that the incentives are all wrong. Nothing worse than feeling like there is pressure for a certain result, or that the story should be simple.
These things are changing though.
(Score: 2) by looorg on Tuesday May 26 2015, @09:00PM
Well I sure as hell don't massage my data.
Right. I wasn't as clear as I should have been. I'll try and rectify that now. I don't change the actual data (cause that would be dishonest and fraud). You change the parameters and other things such as the perspective and/or put limits on your question(s) and query. You don't make up data.
(Score: 0) by Anonymous Coward on Tuesday May 26 2015, @10:16PM
Sorry. Didn't mean to impugn of your integrity. I definitely feel pressure for things to work out certain ways sometimes. I've been pretty luck so far in that my field is relatively understudied, so even null results are somewhat novel, or have examined population differences with actually large effect sizes consistent with suspected brain damage.
(Score: 0) by Anonymous Coward on Tuesday May 26 2015, @10:01PM
Only publishing positive results has the same effect on the field as massaging the data. The journals are part of the problem.