Phys.org is running a story on some of the issues with modern peer review:
Once published, the quality of any particular piece of research is often measured by citations, that is, the number of times that a paper is formally mentioned in a later piece of published research. In theory, this aims to highlight how important, useful or interesting a previous piece of work is. More citations are usually better for the author, although that is not always the case.
Take, for instance, Andrew Wakefield's controversial paper on the association between the MMR jab and autism, published in leading medical journal The Lancet. This paper has received nearly two thousand citations – most authors would be thrilled to receive a hundred. However, the quality of Wakefield's research is not at all reflected by this large number. Many of these citations are a product of the storm of controversy surrounding the work, and are contained within papers which are critical of the methods used. Wakefield's research has now been robustly discredited, and the paper was retracted by the Lancet in 2010. Nevertheless, this extreme case highlights serious problems with judging a paper or an academic by number of citations.
Personally, I've been of the opinion that peer review is all but worthless for quite a while. It's nice to know I'm not the only one who has issues with the process.
Related Stories
Attendees of a Howard Hughes Medical Institute meeting debated whether or not science journals should publish the text of peer reviews, or even require peer reviewers to publicly sign their paper critiques:
Scientific journals should start routinely publishing the text of peer reviews for each paper they accept, said attendees at a meeting last week of scientists, academic publishers, and funding organizations. But there was little consensus on whether reviewers should have to publicly sign their critiques, which traditionally are accessible only to editors and authors.
The meeting—hosted by the Howard Hughes Medical Institute (HHMI) here, and sponsored by HHMI; ASAPbio, a group that promotes the use of life sciences preprints; and the London-based Wellcome Trust—drew more than 100 participants interested in catalyzing efforts to improve the vetting of manuscripts and exploring ways to open up what many called an excessively opaque and slow system of peer review. The crowd heard presentations and held small group discussions on an array of issues. One hot topic: whether journals should publish the analyses of submitted papers written by peer reviewers.
Publishing the reviews would advance training and understanding about how the peer-review system works, many speakers argued. Some noted that the evaluations sometimes contain insights that can prompt scientists to think about their field in new ways. And the reviews can serve as models for early career researchers, demonstrating how to write thorough evaluations. "We saw huge benefits to [publishing reviews] that outweigh the risks," said Sue Biggins, a genetics researcher at the Fred Hutchinson Cancer Research Center in Seattle, Washington, summarizing one discussion.
But attendees also highlighted potential problems. For example, someone could cherry pick critical comments on clinical research studies that are involved in litigation or public controversy, potentially skewing perceptions of the studies. A possible solution? Scientists should work to "make the public understand that [peer review] is a fault-finding process and that criticism is part of and expected in that process," said Veronique Kiermer, executive editor of the PLOS suite of journals, based in San Francisco, California.
Related: Peer Review is Fraught with Problems, and We Need a Fix
Odd Requirement for Journal Author: Name Other Domain Experts
Gambling Can Save Science!
Wellcome Trust Recommends Free Scientific Journals
Medical Research Discovered to Have Been Peer Reviewed by a Dog
Should Scientists Be Posting Their Work Online Before Peer Review?
Judge Orders Unmasking of Anonymous Peer Reviewers in CrossFit Lawsuit
(Score: 3, Informative) by dltaylor on Wednesday November 19 2014, @01:45AM
Peer review and citations have nothing in common. A paper is peer-reviewed PRIOR to publication to evaluate its fitness for publication. The reviewers of Wakefield's paper were incompetent, and should never be allowed to review another. There is a later review by, of a sort, by researchers trying to build on a previous work; if they cannot re-establish the baseline from the prior work, it may be found flawed (and then there are papers like the FTL neutrinos, where the team was hoping that someone would either find the flaw that they suspected was there, or give them a Nobel).
Citations are a post-publication figure of merit, usually associated work on which others successfully build. A simple fix for papers like Wakefield's is to have negative citations not count as simple citations.
(Score: 2) by cafebabe on Wednesday November 19 2014, @02:29AM
I wish that applied to hyperlinks too.
1702845791×2
(Score: 2) by TheRaven on Wednesday November 19 2014, @09:28AM
Citations are a post-publication figure of merit, usually associated work on which others successfully build. A simple fix for papers like Wakefield's is to have negative citations not count as simple citations.
The UK's REF (the system that evaluates research output from universities) explicitly did not include bibliometrics (for computer science, at least) because they are generally pretty poor at judging research impact. Papers with catchy titles end up cited a lot when someone wants a citation for some broad area and papers about tools and techniques are vastly over-cited. For example, the vast majority of computer architecture papers cite gem5, which is a popular simulator. That doesn't mean that gem5 is particularly interesting research (or that the numbers that it generates are in any way useful - it's very easy to abuse it and come up with completely nonsense results).
sudo mod me up
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @09:39AM
This, a thousand times...
(Score: 1) by mathinker on Wednesday November 19 2014, @08:19PM
You're totally correct that the quoted passage is unrelated to the stated problem. But there's a third elephant in the room: the fact that much research is never attempted to be replicated, because papers dealing with replication (either negative or positive) are less likely to be published and do not confer enough "academic karma" compared with the effort involved.
(Score: -1, Offtopic) by Anonymous Coward on Wednesday November 19 2014, @01:52AM
Most people here at SN will acknowledge that peer review is horribly broken when it comes to academics and research.
But then most of those same people won't acknowledge that the moderation system here at SN, which is peer review, is also horribly broken.
Before we can discuss this topic, I think we need to get our own house in order. We need to get rid of the moderation system here. While it's not as bad as reddit, and nowhere near as downright tyrannical as Hacker News, it still isn't good.
Everybody's comments should be treated as equal. If you dislike what somebody is saying, don't mod them down. Post a reply instead! If you like what somebody is saying, don't mod them up. Post a reply instead!
That's peer review done right. It's about maximizing the expression of ideas, rather than shutting down original thought, like happens when moderation is in play, regardless of whether it's downmods or journal editorial boards.
(Score: 4, Insightful) by c0lo on Wednesday November 19 2014, @02:41AM
I disagree - I think having a moderation system is part of the fun.
I disagree with your base assumption that a moderation system really equates with peer review. May have some overlapping, but again there are parts/purposes/effects specific to it. E.g.:
Except the suggested behaviour doesn't address trolling (deliberately inflammatory posting, with zero meaningful content, much less one that qualifies as "original thought").
(with a perfectly troll, this is of course true: in a total meaningless message, there's nothing to refute. But we aren't living in a perfect world)
Having a moderation system allows for decanting the mud without engaging in a flamewar.
If you love so much reading all the comments, you can of course browse them with a -1 threshold.
As you have a solution, you don't need to ask me to drop something I consider valuable
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @03:39AM
Censorship should be an option for those people, like yourself, who wish to limit their exposure to the real world.
However, it should never be the default.
Everyone should browse at -1 by default.
If you want to subject yourself to the censorship the moderation system provides, then you should have to voluntarily sign up and enable it for your account, and your account alone.
(Score: 3, Insightful) by c0lo on Wednesday November 19 2014, @03:56AM
Isn't it a bit like saying: it's wrong for the SN members to value the community above other publicity?
How about: if you do not want to subject yourself to the default setting, make yourself an account and read to whatever level you like? Because the members can do that.
Careful with that inflammation, buddy, treat it early. You have no idea what threshold I'm using when browsing SN or what are my choice of limits in regards with the "exposure to the real world". As such, I don't see what stand you have to make value judgements.
Anyway... allow me a question: if you are willing to unlimited expose yourself to the real world, why posting as AC? Make yourself an account and feel free to expose your real name and email address. Eh?
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: -1, Redundant) by Anonymous Coward on Wednesday November 19 2014, @04:08AM
Why do I post as an AC? Because I'm not going to waste my time creating yet another fucking website account, especially when I don't need one to comment here. I've got enough other accounts that I rarely use for shitty sites that require them. I'm not going to create another one here! That would be fucking idiotic.
Besides, it doesn't matter who posts a comment. The name is totally irrelevant. It's the message that's what matters. I don't care whose name is on a comment. I usually don't even bother to look. It's the comment I'm reading and replying to, not the goddamn name above it.
(Score: 2) by c0lo on Wednesday November 19 2014, @04:41AM
Never asked that, never been interested.
I asked what standing do you have to make judgement values about my willingness (or the lack thereof) to "exposed myself to real world".
Used the matter of the "why not create an account?" to point you have limits in this regards, so you should accept others having them as well (take it as a hint: don't preach what you can't follow. It hurts the message).
Others will tend to disagree. Even letting aside the matter of authorship, take for instance those who value some tunes more for the melody and less for the lyrics; is a bad thing?
...
(speaking of "form over meaning": I don't care how strong you feel about the real world, using profanity in a gratuitous mode sounds bad to my ear. In other words, your form impacts on your message.
...
Eh...eh... stop there. Don't even think to say "I don't care" in whatever form).
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @06:25AM
You don't need an account to set the threshold. I change it all the time.
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @07:05AM
Are you some kinda moron?
(Score: 2) by monster on Thursday November 20 2014, @09:15AM
You say it as if there hadn't ever been AC posts getting +5 moderation (hint: there are).
(Score: 2, Insightful) by NotSanguine on Wednesday November 19 2014, @07:08AM
If you want to subject yourself to the censorship the moderation system provides, then you should have to voluntarily sign up and enable it for your account, and your account alone.
Funny that. That's exactly what I do. I tend to browse at '-1' because I'm too lazy to change my settings every time I have moderator points. You can do that too.
As someone who doesn't like censorship, why are you trying to foist your opinions and beliefs on everyone else and make them have the same experience that you do?
Here comes one of my favorite quotes again:
Make of that what you will.
No, no, you're not thinking; you're just being logical. --Niels Bohr
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @04:41PM
Censorship should be an option for those people, like yourself, who wish to limit their exposure to the real world.
"Exposure to the real world." Awesome. Sounds like it's supposed to mean that all opinions are valid while completely dismissing the possibility that some statements fail to be either opinions or coherent. Clearly, anyone who disagrees with this statement is a sheltered fascist.
It's the message that's what matters.
My message is m+ON2dhhqgsFa6R/N3HkbJHFukkJvtgj1VAclJA+4nyBkeF2XYOeGQuCx0VPomw7SynEkLBG1nYtScZ2XHdfrmk+Rv4Xmpjn+QPgdjtKZgI34L1PflPij9VEhEKFCbshK+rI0AbTFnc0Kb0RwgC1OOFMsmtaunI2Wk/5Qq8MV9m+bSOlrLyiAR5abffAcXrqK+7coR0HxZrvK2
/AhRw3/PAbZkavnrKPhQNKQwNbO9kQoUxZivkX
That's some deep matter, there, my friend. I hope you can appreciate it, and I'll be happy explain any parts that seem confusing to you.
(Score: 0) by Anonymous Coward on Thursday November 20 2014, @05:59PM
Shouldn't the second "k" have been a "c"?
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @03:06PM
I think the ideal moderation system is hypothetical, it would be done by the site owners who have a business interest, not just a professional interest, in the site. Craigslist comes close to that although they don't have a system of upmodding and downmodding - Craig Newmark personally tends to the site and culls troll posts and posters.
The best moderation system would welcome a broader spectrum of viewpoints than we typically see at /. at the level of 2 or above, especially on issues where the regs tend to congregate on one side or the other. Trolls, however, would still be banished.
Historically, small businessmen who operate retail stores on Main Street have been among the most welcoming for minorities in their communities, because they knew that customers can easily cross the street to the competition and they can't afford to write off a large chunk of their customer base. Shoplifters and other troublemakers are different - they need to be dealt with in a curt fashion. Of course, the businessman can't greet and check out all the shoppers himself, he has to delegate some of that to the help. And some of the help, even after training, aren't going to see the need to be quite so welcoming to folks that are of a different race, age, religion, sexual orientation, political party, etc.
(Score: 2) by c0lo on Wednesday November 19 2014, @11:07PM
Have a look on the percentage already covered from funds needed by SN and think how much "business interest" is at play within SN.
Then, think that SN got their membership (especially the subscribing one) around the idea of "self-sufficient and self-regulated community that likes to discuss news interesting for people". This is the main attraction point.
Now, are you still insisting in replacing "moderation by community"? Then, read again the above until you get it.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 4, Insightful) by Sir Garlon on Wednesday November 19 2014, @02:43AM
There are plenty of places on the internet where every troll and idiot's comments are treated equally. If you actually enjoy discussing things there, then I don't think you need Soylent News.
Bringing us back toward the topic, moderation should be used more like citation count and less like peer review. When you're first exploring a topic, highly-cited papers are a good place to start because they will define the major principles of the discussion. Likewise, the highly-moderated posts in a discussion are the ones that best capture the fundamental points. Moderation should indicate that a post is worth the time to read, that it is relevant and illuminating. Confirmation bias being what it is, it is a lot easier to see the merit in a post one happens to agree with. In principle, I think we all know that a coherent, well-argued post we disagree with is worthy of positive moderation.
I agree that moderation seems too often to be used as a substitute for the facebook "like" button, but that is what the "overrated" mod is for.
Now, if you think a post is especially good or bad, flawed, stimulating, ignorant, whatever, that's what the reply button is for. That is what peer review should be in a scientific paper -- a response to the research.
[Sir Garlon] is the marvellest knight that is now living, for he destroyeth many good knights, for he goeth invisible.
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @03:50AM
A "citation count"? That sounds a fuck of a lot like "karma". You know, the goddamn numbers that turn places like Hacker News and Stack Overflow into absolute shitholes. It's bad enough seeing people fawn over tptacek and Jon Skeet, and otherwise engage in the community-wide circle-jerks that those places have become. It's totally disgusting to suggest bringing such a flawed concept here. The /. moderation system is total shit, but it's nowhere near as utterly broken and futile as the moderation systems over at HN and SO are.
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @03:11AM
FWIW, I think the real problems in the academic community lie elsewhere.
Scientists are being rewarded more for publishing stuff often. Not for making actual progress in science.
They need to publish or they don't get $$$$$, so some seem to split stuff up to obviously incomplete studies that come to wrong (preferably headline baiting) conclusions because they are incomplete - and the final correct result that they are "working towards" ends up being many years away.
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @03:23AM
So let me get this straight: SN moderation involves a small number of anointed moderators reviewing the comments of their peers, but somehow it's not peer review?
(Score: 2) by c0lo on Wednesday November 19 2014, @04:43AM
Indeed is not. The same way even if a both a duck and a penguin have feathers, they aren't the same. Surprised?
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 3, Funny) by richtopia on Wednesday November 19 2014, @05:36AM
Well, are we talking Python or C++? Because I think Python really does not care about the duck/penguin... they both have feathers after all.
(Score: 2) by c0lo on Wednesday November 19 2014, @06:05AM
They are different from inheritance which was what the GGP used when asking: "you say that moderation isn't peer review even if it uses peers and techniques of review?" (note the is-a relationship as the subject of her/his question).
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @10:39AM
You may not know how peer review works. Here's the main differences between peer review and SN moderation:
(Score: 1, Insightful) by Anonymous Coward on Wednesday November 19 2014, @03:16AM
> Everybody's comments should be treated as equal.
There is nothing stopping you from experiencing SN in exactly that way - just log in and set your moderation threshold to -1 and viola! all comments are treated equal FOR YOU.
But I know who you are and I know how insane you are and I know your actual complaint is that you can't force everyone that threshold on everyone else. Tough shit.
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @03:20AM
Hi Tork!
(Score: 2) by tibman on Wednesday November 19 2014, @05:25AM
Can i get in on this epic game of Guess The AC? Who's winning?
SN won't survive on lurkers alone. Write comments.
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @12:39PM
Tork is winning.
(Score: 2) by tibman on Wednesday November 19 2014, @02:30PM
Hi TheMightyBuzzard!
SN won't survive on lurkers alone. Write comments.
(Score: 2) by novak on Wednesday November 19 2014, @07:25AM
This mythical place where all comments are treated equally exists! Head on over to http://youtube.com/ [youtube.com] and prepare yourself for a deep, scintillating discussion of the issues!
novak
(Score: 1) by N3Roaster on Wednesday November 19 2014, @04:48PM
Not quite. YouTube will hold some comments for review if they look particularly spammy and channel operators can eliminate those or any other comment on their videos/channel for any reason they want. It's also possible to configure highlights such that comments that get a lot of likes, comments from people with popular channels, and the like are moved to the top, giving those more visibility. A lot of YouTube is a cess pit only because channel operators choose to allow it. I've been rather fortunate as comments on my videos (see sig) tend to be pretty good and even the lower quality comments are comparably innocuous.
Typica - Free software for coffee roasting professionals. [typica.us]
(Score: -1, Flamebait) by Anonymous Coward on Wednesday November 19 2014, @03:26AM
Science is hopelessly broken because scientists are just gossipy promiscuous twitter jerks with advanced degrees. They sleep with their students and suck cock for citations. I've seen it happen.
Human nature is the problem. Period.
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @03:43AM
Can somebody please fix that incorrect moderation? The parent is obviously not "flamebait".
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @04:04AM
Was this figurative cock sucking that you witnessed, or was it literal cock sucking?
(Score: 1) by art guerrilla on Wednesday November 19 2014, @06:49PM
*sigh*
in case you had not gotten the meme-o, they are both the same now...
'literally' literally means 'figuratively' these days, the dictionary mavens have cried uncle and given in to the illiterati...
wait a minute, so 'literally' is not literally 'literally' anymore, but since it has been abused so much, it now figuratively means literally, or literally means figuratively ? ? ?
fuck, this world is confusing me (literally?) more and more... crap, i'm screwed... (figuratively ?)
for me, *that* factoid alone is sufficient to mark the end of western civilization...
words no longer have meaning, they have intent...
(Score: 2) by c0lo on Wednesday November 19 2014, @04:04AM
[Citation needed]
(ducks)
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @04:10AM
Here's the citation [soylentnews.org] that you requested.
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @04:24AM
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @04:37AM
I wasn't sure if you were correct or not, so I did some research on my own. At least one of the sources I referred to [soylentnews.org] confirms what you're saying.
(Score: 2) by c0lo on Wednesday November 19 2014, @05:29AM
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 1) by splodus on Wednesday November 19 2014, @03:37AM
I think there's a problem with 'cronyism' in peer review, in that the same groups of researchers get published time and again, and newcomers struggle to get their work in the reputable journals. But peer review itself is valuable - it's a filter that tries to ensure a minimum standard for research papers so that badly designed studies don't pollute the pool of science.
The stuff about citations is something else. It's a blunt instrument and yes, it really is flawed, and it really does need to change.
Don't mix the two issues together. Peer review needs tweaking, sure. What we're seeing (increasingly) with scoring the value of research based on citations needs to stop right now.
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @04:17AM
What are your thoughts about the allegations posted here that some academics "sleep with their students and suck cock for citations" [soylentnews.org]?
(Score: 0) by Anonymous Coward on Wednesday November 19 2014, @10:53AM
"Some academics" is a sufficiently loose term that you can expect it to be true (as in, from purely statistical considerations, there are probably indeed at least two academics in the world who indeed have slept with their students and sucked cocks for citations). By the same account, there are also some internet trolls who sleep with their mother, some programmers that kill people, some sysadmins who fuck children, …
(Score: 2) by FatPhil on Wednesday November 19 2014, @09:20AM
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2) by aristarchus on Wednesday November 19 2014, @04:59AM
Personally, I've been of the opinion that peer review is all but worthless for quite a while. It's nice to know I'm not the only one who has issues with the process.
You are aware, are you not, that if this is your personal opinion, it is by definition not peer reviewed. And no matter how many similarly cynical persons cite your skepticism, it does not rise to the level of a peer review where the peers are actually experts in the field rather than a gaggle of denizens of the intertubes. I am glad we are having this discussion, and Dave, I have the greatest enthusiasm for the mission, but this comes across as good old fashioned American anti-intellectualism where we cannot establish a statistically significant result because we are afraid of going out in public without a concealed weapon. (I prefer a Thermal Detonator: Good enough for Princess Leia, good enough for me.)
(Score: 1) by khallow on Wednesday November 19 2014, @06:11PM
You are aware, are you not, that if this is your personal opinion, it is by definition not peer reviewed.
The writer made no pretense to having a peer reviewed opinion. At worst, their words could be construed to extrapolate a consensus from one or two other peoples' agreement.
but this comes across as good old fashioned American anti-intellectualism where we cannot establish a statistically significant result because we are afraid of going out in public without a concealed weapon.
Because discussions about concealed weapons in the US come naturally out of any discussion of peer review. And "American anti-intellectualism"? Ouch. He won't be able to show his face in public for weeks after that burn!
You seem to have a rather large number of fallacies (straw man, non sequitur, ad hominem to name the common ones I see right away in the above quoted bits) in your post for someone complaining about "anti-intellectualism". Now, I'm just another rube on the intarwebs, but it strikes me that using multiple fallacies in a short post is not the mark of an intellectual. Just saying.
(Score: 1) by JohnnyComputer on Thursday November 20 2014, @04:44AM
..why would something so obviously misconceived as an article conflating the process of peer-review with citation metrics be published here at SoylentNews?
FAIL.
(Score: 0) by Anonymous Coward on Thursday November 20 2014, @06:01PM
Because SN submissions are not peer reviewed? ;-)
(Score: 2) by buswolley on Saturday December 06 2014, @11:07AM
I admit that my submission proposing a potential solution to the problems wbich American democracy faces isn't news, but this article's conflation of the peer review process with the accumulation of citations is frustrating as his hell.
subicular junctures