Peer review has failed, and that's great news – for diamond open access, science and society:
[...] Mastroianni explains that although ubiquitous today, peer review is a relatively new phenomenon. After World War II governments poured huge amounts of money into research; peer review was supposed to make sure the money was well spent. But as Mastroianni documents, peer review has failed on just about every metric.
Research productivity has been flat or declining for decades; reviewers consistently miss major flaws in submitted papers; fraudulent work is published all the time. Peer review often encourages bad research because of unhelpful comments; and scientists themselves don't care about peer review: they actively seek to circumvent it, and ignore it in their own reading.
[...] last month I published a paper, by which I mean I uploaded a PDF to the internet. I wrote it in normal language so anyone could understand it. I held nothing back—I even admitted that I forgot why I ran one of the studies. I put jokes in it because nobody could tell me not to. I uploaded all the materials, data, and code where everybody could see them. I figured I'd look like a total dummy and nobody would pay any attention, but at least I was having fun and doing what I thought was right.
Then, before I even told anyone about the paper, thousands of people found it, commented on it, and retweeted it.
What Mastroianni describes is essentially the diamond open access approach, something Walled Culture has discussed several times. It is designed to provide an extremely simple and lightweight publishing platform to help researchers get their papers quickly and easily before as many people as possible. It is costless, for both the person uploading their paper, and those who download it.
(Score: 5, Insightful) by garfiejas on Thursday January 19 2023, @01:05PM (10 children)
If I had to read every random theory from every random crackpot on the internet where both are trying to score points from some random viewpoint, my research would absolutely. stop. dead.
The current system isn't right, it needs fixing - the reviewers, researchers and audience are being exploited by vested media interests for $$ - but removing that gateway entirely - that way madness lies...
(Score: 5, Interesting) by Ox0000 on Thursday January 19 2023, @02:06PM (2 children)
Wholeheartedly agree.
I also somehow take issue with their claim:
This is not accurate. Since the days of Gauss, heck even further back, there was 'peer review'. It wasn't formalized nor was it the same 'kind' of peer review we use today (which has mostly been hijacked by the big journals) but peer review existed. You could argue it was more 'collaboration' than peer review but it did serve as a check/balance on 'publications'.
People would correspond about their findings with others in the field to touchstone their findings, to get new ideas, to hone in on a more accurate truth. In essence, they were having peers review their findings.
I think Mastroianni should do some more literature study before publishing his utterances. Maybe even get their ideas peer reviewed by some close collaborators first, who may have persuaded them to alter their viewpoint a bit...
(Score: 4, Interesting) by garfiejas on Thursday January 19 2023, @02:29PM (1 child)
Agree with you "peer review" has been around since before Ancient Greece; i.e. correspondence between Ada Lovelace and Charles Babbage; its how science (religion, philosophy, politics even) is done - living truth; do we want some random critic (or ChatGPT) from the other side of the planet to have an "influence" on it, "cancel" it, shout it down because it doesn't agree with their tiny bubble or dead dogma.
Science "influencers" on rating test tube racks... I think I'd take up phone sanitisation...
(Score: 3, Insightful) by jb on Friday January 20 2023, @02:20AM
There's some very strange timeline distortion going on here.
Countess Lovelace & Charles Babbage lived in the 19th century; Ancient Greece was BCE (and therefore Lovelace & Babbage were definitely not "before" it).
Maybe my memory is fading, but I could have sworn that it was generally accepted that formal peer review in academic publishing originated during Oldenburg's tenure at the Royal Society. That was in the 17th century (lifetimes before Babbage, but nowhere near as far back as Ancient Greece).
(Score: 4, Interesting) by ElizabethGreene on Thursday January 19 2023, @03:46PM (4 children)
Do professional journals verify the bona fides of their authors, or is the (massive detrimental parasitic) publishing fees sufficient filter to remove the crackpots?
(Score: 5, Informative) by garfiejas on Thursday January 19 2023, @04:44PM
Hmm, thats a very good point; define "professional journals", there is a list of "poor quality journals" https://guides.library.yale.edu/c.php?g=296124&p=1973764 [yale.edu] - and issues around open access peer reviews in general https://link.springer.com/article/10.1007/s00264-020-04504-1 [springer.com] but as the last paper points out "peer review" "is an essential step in protecting the quality and integrity of scientific publications".
For "who" - the "identity" thing, organisation, tenure and some titles helps me significantly; perhaps some audit trail on the peer review process (in the paper 76% of reviewers agreed to have their name presented to authors) as a way forward, held in escrow...
The $$ issues definitely needs fixing, these folk https://www.nature.com/articles/s41586-022-05543-x [nature.com] - report a decline since 1945 of the disruptive nature of papers (and patents) - and link some of this to the closed nature of the "commercial" journals including peer review; I'm not convinced but thats my opinion.
However, sites like https://www.researchgate.net/, [researchgate.net] arxiv and (to a lesser extent for me https://academia.edu) [academia.edu)] have enabled a more curated but open discussion with researchers who use it....
(Score: 3, Insightful) by maxwell demon on Thursday January 19 2023, @07:04PM
At least in physics, many traditional journals don't have publication fees, but the readers pay them. This has downsides for the readers (you don't get to read the papers unless you pay), but it means that the journal is not interested in publishing any random nonsense. They are interested in publishing what their audience is interested to read. Now that does mean that if you're researching an unpopular topic, you'll have more trouble to publish, so the incentive structure isn't perfect, but it works pretty well to keep the biggest rubbish out.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 4, Informative) by Kell on Friday January 20 2023, @01:20AM (1 child)
Hi - I'm a senior editor for an IEEE journal. We do not assess the bona fides of authors: we assess them only on the quality of the manuscript itself, and we trust that the stated affiliation in their author information is correct. We don't have the means to check those things out, anyway. That said, it shouldn't matter where an author is from or who they are if the work itself is true and correct. We put more effort into using our own professional networks into recruiting good reviewers, and substantial work into choosing quality associate editors. Being a senior editor means you're someone known and valued in the community. In worthwhile journals, EiCs are lofty laureates without exception.
The problem we have is that journals require reviewers and editors to do a lot of free labour, and authors are charged money to publish and readers are charged money to read - all so the organisation can make a profit. For something that's a non-profit like IEEE, that money goes back into the community, so I feel it's not so terrible ethically, but ultimately I believe it's unsustainable. If I charged IEEE what my time is worth nobody would be able to afford to publish in our journal, which means it wouldn't be serving its purpose.
Scientists ask questions. Engineers solve problems.
(Score: 5, Insightful) by ElizabethGreene on Friday January 20 2023, @02:43AM
You've hit on the big problem.
Joe Scientist works to produce a paper. They submit that paper to a Journal and pay a small fee to get it looked at. If it's accepted for peer review, then they pay a large publicatoin fee. The machine grinds through the process of the unpaid/volunteer editor and peer reviewers doing their thing, and Joe gets a publication date. Then, if Joe wants the paper to be open access, they get to pay for that privilege too.
I struggle to make a strong argument why Joe should be paying Elsevier $5K to "publish" a three-page pdf on a website. This is even more ludicrous when you consider that nine of the ten people that actually read the paper are also paying Elsevier for the privilege through a facility subscription.
This is a burn-it-down-and-start-over broken system.
(Score: 0) by Anonymous Coward on Thursday January 19 2023, @07:14PM
What are you talking about? The answer is right there in TFA: just use as your metric the number of retweets it gets! Plus, if your account doesn't get banned, then you can probably assume Musk has approved of it too.
(Score: 3, Interesting) by aafcac on Thursday January 19 2023, @09:25PM
Same goes for replication studies. There should be a lot more funding for both peer review and replicating studies if we care at all about having research that's reliable. And while we're at it, negative findings are legitimate findings, as long as s researcher isn't primarily getting negative result it shouldn't be an issue. (In which case, they're terrible at founding hypotheses)
(Score: 3, Interesting) by PiMuNu on Thursday January 19 2023, @03:16PM (5 children)
As a funding agency, or university employer, I need some way to discriminate between person (A) and person (B). Last century, this was done by a qualitative assessment by the committee (still the way it is in many countries). i.e. does the hiring committee/grant committee think the guy is clever and influential?
Latterly in the west, especially following the rise of HR departments and equivalent, a more quantitative and apparently objective approach was sought. The aim is noble, to promote diversity and block the "old boys network" dominating things. This is done by looking at publication records, H factors and other such statistical representations of "importance" or "profound research".
There is an argument that such an approach is a sticky plaster (bandaid, to use the brand name) that doesn't really fix the problem. But, if one does away with the publications process altogether, it doesn't really fix the problem; we merely go back to the old way of smoking rooms and discussion over whisky. The old boys grow in strength, rather than diminishing as is the apparent desire of the author.
(Score: 1, Insightful) by Anonymous Coward on Thursday January 19 2023, @03:29PM (1 child)
As a funding agency, or university employer, you (not "you" you, the "you" one that is referred to) maybe should actually do due diligence as opposed to looking for shortcuts that try to distill a person or team to a single number that is used as the main input in your evaluation of whether or not you want to give them money.
Maybe as a funding agency or university employed, you should practice what you preach? Oh, too much work you say?
(Score: 3, Touché) by PiMuNu on Friday January 20 2023, @10:19AM
> do due diligence
What due diligence are you referring to?
(Score: 1, Troll) by crafoo on Thursday January 19 2023, @06:13PM (2 children)
So the story is we saw corruption in the sciences so we instituted objective metrics to overcome this corruption (nepotism).
What actually happened is that resentful losers found a way to corrupt and destroy an institution from within, make their friends money, and push their own personal agenda forward. Yes I'm talking about HR and other administration bodies that sit on all productive endeavors like a fat tick.
Now they put a happy face on their own corruption and blame those that came before.
(Score: 0) by Anonymous Coward on Thursday January 19 2023, @11:18PM (1 child)
> Yes I'm talking about HR and other administration bodies that sit on all productive endeavors like a fat tick.
Harsh way to state this, but for once I agree with crafoo.
I've yet to come across an HR department that contributes something useful to an organization. As the owner of a tiny engineering company with a long customer list (going back c. 1980), I've seen the negative effects of quite a few HR departments. Best I can tell they are eye candy for the C-suite bosses they report to.
(Score: 2) by PiMuNu on Friday January 20 2023, @10:17AM
Cynically I disagree. HR is not eye-candy, they are there for the purpose of evil i.e. to help senior management sack people legally.
(Score: 0) by Anonymous Coward on Thursday January 19 2023, @07:23PM
Oh yeah, this is exactly what I want to keep up on my research field, lots of imprecise folksy text and jokes and irrelevant crap to wade through ("Gee, I don't know why I ran that test -- must have been a Monday morning LOL!"). Just go all the way and put in "funny" meme gifs and emojis. Sort of a Reader's Digest scientific journal. Can't wait to see the rise of all the new Dr. Oz and "Dr." Phils to come up in other fields.
(Score: 3, Interesting) by ShovelOperator1 on Thursday January 19 2023, @07:31PM
The problem with peer review is that publishers and ministries greed is killing it.
Publisher's greed, because the review is usable for scientists only for introducing a point like "Reviewing papers for ..." in personal Resume. So this is OK for one or two papers, in a few journals, but nothing too much. Mostly reviewers work for free.
The second thing is the various "verification instruments", with ministries responsible for academic institutions being in the first place.
A long, long time ago, it was "Develop and publish". These are papers describing complete discoveries, equations, models and systems.
Later, it was "Publish or Perish".
Now, it is "Publish in high-ranking journals, which are moving target, frequently, so maybe you won't perish.
The problem is that these papers made only to be OK with lists are not too valuable. The "Application of [already known technology] to [already known phenomenon]" was the previous generation, now it is like "Considerations of application of [totally unusable technology] to [totally unproven phenomenon]". Someone has to review this, and these papers are... well, not too usable. So reviewers, working for free, do the simple thing: Known author - "Accept with notes", new author - "Reject". This makes the system even more hermetic and makes more feedback loops.
And while such situation previously led to development of highly technical subcultures, in current economic conditions it leads to more failed startups.
(Score: 5, Interesting) by dalek on Thursday January 19 2023, @08:15PM
This seems to be written by someone who doesn't really understand what peer review is. They say peer review has failed because it's unsuccessful in things it was never intended to do.
There is already a mechanism to quickly disseminate results and theories without peer review. That's exactly what preprints are. You can submit your work somewhere like arXiv.org. Some conferences allow presenters to upload an extended abstract, which is effectively the same thing as a preprint.
As a reviewer, I'm usually given somewhere in the range of two weeks to 30 days to complete my review. With other obligations like research and teaching, I might only be able to dedicate one full day of work to reviewing the manuscript. That means reading it over a few times, looking for flaws in the methodology, trying to identify obvious errors, and verifying that the data presented (e.g., tables and figures) support the conclusions. A single reviewer might miss flaws in the manuscript despite being as careful as possible during the review. That is why multiple reviewers give their feedback to an editor, who also reviews the manuscript. Even with three or four people reviewing the manuscript, it's possible they might all miss something despite their best efforts. We do our best. Most editors and reviewers try to be constructive. As an author, I've had editors be very willing to discuss reviewer concerns to me and offer ideas to improve my manuscript.
Peer review does not exist to reproduce research before it is published. Reviewers have neither the time nor the resources to reproduce the research, particularly given the short amount of time in which reviews need to be completed. I would agree that there needs to be more transparency in the sharing of data and software so reproducing research is easier. However, data sets can be quite large, and someone has to pay for the storage of the data once the grant funding the research has expired, so sharing data is easier said than done. Limited funding for research often means that new research is prioritized for funding instead of reproducing existing research. These are valid concerns, but they are not due to failures of peer review. It is not the job of reviewers to attempt to reproduce research.
There are other mechanisms for other researchers to criticize published papers. For example, at least in some journals, after the paper has been published, other researchers can submit a reply to the journal explaining why they believe the paper is flawed. The original authors have the opportunity to issue a rebuttal to the criticism, and that is also published.
Some journals tweak the peer review process, such as publishing the names of the reviewers and all correspondence between the reviewers and the author. The transparency might prevent some abuses and fraud. But eliminating peer review will make things worse without actually solving problems with reproducibility.
THIS ACCOUNT IS PERMANENTLY CLOSED
(Score: 3, Insightful) by darkfeline on Thursday January 19 2023, @08:34PM
It's always the incentives, stupid.
I do agree with making research freely available (the scientific method demands it), but there are problems. Like it or not, funding is an issue. Paying reviewers and/or editors may or may not be an issue, depending on how/if you want to subsidize unpopular areas (smell research went from 0 interest to 200% after covid; if we invested in it earlier we might be better off now).
Join the SDF Public Access UNIX System today!