NIH moves to punish researchers who violate confidentiality in proposal reviews
When a scientist sends a grant application to the U.S. National Institutes of Health (NIH) in Bethesda, Maryland, and it goes through peer review, the entire process is supposed to be shrouded in secrecy. But late last year, NIH officials disclosed that they had discovered that someone involved in the proposal review process had violated confidentiality rules designed to protect its integrity. As a result, the agency announced in December 2017 that it would rereview dozens of applications that might have been compromised.
Now, NIH says it has completed re-evaluating 60 applications and has also begun taking disciplinary action against researchers who broke its rules. "We are beginning a process of really coming down on reviewers and applicants who do anything to break confidentiality of review," Richard Nakamura, director of NIH's Center for Scientific Review (CSR), said at a meeting of the center's advisory council earlier this week. (CSR manages most of NIH's peer reviews.) Targets could include "applicants who try to influence reviewers ... [or] try to get favors from reviewers."
[...] The agency provided few details about the transgressions after Michael Lauer, NIH's deputy director for extramural research, published a blog post on the matter on 22 December 2017.
Related: Should Scientific Journals Publish Text of Peer Reviews?
Related Stories
Attendees of a Howard Hughes Medical Institute meeting debated whether or not science journals should publish the text of peer reviews, or even require peer reviewers to publicly sign their paper critiques:
Scientific journals should start routinely publishing the text of peer reviews for each paper they accept, said attendees at a meeting last week of scientists, academic publishers, and funding organizations. But there was little consensus on whether reviewers should have to publicly sign their critiques, which traditionally are accessible only to editors and authors.
The meeting—hosted by the Howard Hughes Medical Institute (HHMI) here, and sponsored by HHMI; ASAPbio, a group that promotes the use of life sciences preprints; and the London-based Wellcome Trust—drew more than 100 participants interested in catalyzing efforts to improve the vetting of manuscripts and exploring ways to open up what many called an excessively opaque and slow system of peer review. The crowd heard presentations and held small group discussions on an array of issues. One hot topic: whether journals should publish the analyses of submitted papers written by peer reviewers.
Publishing the reviews would advance training and understanding about how the peer-review system works, many speakers argued. Some noted that the evaluations sometimes contain insights that can prompt scientists to think about their field in new ways. And the reviews can serve as models for early career researchers, demonstrating how to write thorough evaluations. "We saw huge benefits to [publishing reviews] that outweigh the risks," said Sue Biggins, a genetics researcher at the Fred Hutchinson Cancer Research Center in Seattle, Washington, summarizing one discussion.
But attendees also highlighted potential problems. For example, someone could cherry pick critical comments on clinical research studies that are involved in litigation or public controversy, potentially skewing perceptions of the studies. A possible solution? Scientists should work to "make the public understand that [peer review] is a fault-finding process and that criticism is part of and expected in that process," said Veronique Kiermer, executive editor of the PLOS suite of journals, based in San Francisco, California.
Related: Peer Review is Fraught with Problems, and We Need a Fix
Odd Requirement for Journal Author: Name Other Domain Experts
Gambling Can Save Science!
Wellcome Trust Recommends Free Scientific Journals
Medical Research Discovered to Have Been Peer Reviewed by a Dog
Should Scientists Be Posting Their Work Online Before Peer Review?
Judge Orders Unmasking of Anonymous Peer Reviewers in CrossFit Lawsuit
(Score: 2) by FatPhil on Saturday March 31 2018, @10:03AM (12 children)
Also, this isn't what we mean by "peer review".
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 3, Insightful) by fadrian on Saturday March 31 2018, @12:38PM
I love your "If it's funded by anyone it must be biased" take on things. It makes it so easy to ignore results one doesn't agree with.
Peer review can take place in almost any context where you have a peer and is not limited to scientific results - my work gets reviewed by my peers even though what we're doing is coding, not science. It was clear here that the reviews were reviews of grant applications which, due to potential favoritism, need to be anonymous and are sent out to review to other scientists who are a lot more qualified to judge the quality of the application than the NIH employees in charge of the process.
But you go ahead believing your paranoid delusions that "It must all be biased". Nothing like taking the opportunity to bash something that the government does that's actually worthwhile.
That is all.
(Score: 1, Insightful) by Anonymous Coward on Saturday March 31 2018, @12:47PM
The science doesn't care, not very much anyway. But the many millions of tax dollars do, which are being allocated according to these reviews.
This is not about scientific integrity (we have enough other problems there). It's about potential corruption at the national level.
(Score: 4, Insightful) by looorg on Saturday March 31 2018, @01:31PM (4 children)
You are correct, this isn't what is normally considered to be peer review. But it is still a form of peer review, just not the one we mostly think of or associate with the term.
It could be both a financial, a scientific and ethical dilemma. At this stage they are reviewing the proposal for the research to decide if they want to fund it or not. It might be a bit odd that a grants proposal should be secret, but there is a valid cases for it -- after all they don't have an infinite stack of grant money to hand out. If there are many different projects competing for the grants then a need for secrecy might be required. After all they want to get the most and best science for their grant money. So just like with public contracts if they can get just as good a science with as good a scientists for less bucks then from someone else they are going to go with that option. So if you have someone on the review board that can tell various project groups what the other applicants have put in their proposals and how much they are asking for they in turn can send in an application at the last moment and slide right under the others promising more and better science for less funds and so securing their own job for another couple of years. There is fierce competition for grants, so this isn't all that odd at all. Which in turn sort of makes this a big deal and it does matter.
Will it change the science? It could change science. In the sense that the other project might have another perspective or method or just by taking all the grant monies they prevent another project from doing their science.
(Score: 2, Interesting) by nishi.b on Saturday March 31 2018, @03:26PM (3 children)
True.
But as a scientist I can tell you that in a lot f research projects it is almost impossible not to know who is the reviewer as there are a very limited number of people working in your specialized subfield. Just by the type of remarks, especially when are presenting a theory that conflicts with another one, if you get a review nitpicking about everything it comes from the other group, and as you meet them in scientific conference, you know who is going to express this or that concern. It can be even easier for grant applications to know that, because it is usually more high-profile professors that will be asked to fund you than ordinary journal papers.
So we answer to the reviewer's comments when that's part of the review process, we must answer as if we did not know who the reviewer is, which is often hypocritical...
(Score: 3, Interesting) by rleigh on Saturday March 31 2018, @06:02PM (2 children)
My (software/methods) paper got rejected last year, not based upon its merits, but because the reviewers lab is a "competitor" of my supervisor (apparently, it's not like we work on exactly the same stuff). They didn't like one aspect of the design, which came down essentially to one opinion over another, neither of which is objectively correct, but just a pet preference. The journal rejected it because it was "controversial", ignoring the fact that the particular design feature was in widespread use for over 15 years. Seems very odd that a single individual can negatively impact the careers and success of other people by abusing the power they have been given. That's not "review", it's sabotage in my opinion, and it really sucks.
(Score: 3, Insightful) by rleigh on Saturday March 31 2018, @06:08PM (1 child)
Hit submit too soon. I meant to add that if you see this stuff happening at the level of paper review, it almost certainly happens at the level of grant review as well. All the same set of people all sit on each others' grant review panels as well.
(Score: 2) by FatPhil on Sunday April 01 2018, @08:29AM
(The last time I was any where near academia, the guy I was doing mathematical monkeywork for got rejected 5 times, but when his paper finally got accepted, the reviewer said something along the lines of "this rips the final pages out of all of the textbooks, and should be compulsory material when teaching the field". So I don't particularly trust peer review, even in a field as "opinion"-less as pure mathematics. OK, it worked in the end, but even then it was met with "but there's too much for one paper, can you split it".)
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2, Insightful) by khallow on Saturday March 31 2018, @06:36PM (4 children)
The science doesn't care if it exists or not. It doesn't care if we understand more or less. And come to think of it, why should we care about what science cares about? Doesn't make a difference to us or to the research we do or the benefits we gain as a result.
The thing is, such corruption as described in the article leads to more non-science than science. What would be compromised is the opportunity cost of the research that wasn't done, because someone else gamed the system for a non-scientific outcome.
While they're still rewarding a lot of p-hacking, irreproducible research, it is nice that they have some sort of enforced standard in place to keep this publicly funded research from becoming completely irrelevant to us.
(Score: 2) by FatPhil on Sunday April 01 2018, @08:32AM (3 children)
And guns lead to shootings. Therefore...
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2) by Bot on Sunday April 01 2018, @09:16AM (1 child)
> And guns lead to shootings. Therefore...
... me is happy.
Account abandoned.
(Score: 2) by FatPhil on Sunday April 01 2018, @10:38AM
Thank god Trump's got a really big button, which totally works.
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 1) by khallow on Sunday April 01 2018, @12:55PM
Broken analogy. These review boards are a system intended to evaluate the quality of proposed research and help select and fund the research that is best, according to the criteria of the NIH at the time. So the whole point of them is to yield better research in the first place. Is are guns a system which tries to reduce shootings? Of course not.
(Score: 0) by Anonymous Coward on Sunday April 01 2018, @04:15PM
Is there any evidence that this institutionalized peer review is impeding the "needless expenditure of government funds"? To me, its primary role seems to be as a wholly inadequate substitute for actually funding/running the needed replication studies.*
*Besides this, they have also replaced checking the predictions of a theory with checking the predictions of a null hypothesis.