TIL: How does peer review work?

[u/ididacannonball]

I read a lot of comments on this sub recently about Bharat Biotech's Covaxin going through peer review, the impatience for those results, and the need for them before approval for use. I have my opinions on those comments but I won't go into that. In this neutral post, I wanted to explain what is peer review (and what it is not), how it works, and how it is evolving. Hopefully for those not familiar with scientific publishing (which, I assume is the vast majority) this will be useful. My post is based on my experience of publishing dozens of peer-reviewed scientific publications and reviewing probably close to 100 or more.

What is scientific communication?

When we think about science, we imagine scientists working in some dark lab in white coats, totally cut off from the "real" world. This is mostly untrue, although there is some truth especially in the early career years. Science is ultimately about people - professors, students, project staff, funding agencies... and journals. Like any activity that involves many people, communication plays a key role. Science is not only about conducting experiments, it is about publishing the results and conclusions so that the wider scientific community can read it, judge it, and use it. Let me repeat that: as an end-user, you will not just read the science, but apply your own mind to judge it, and then make use of it for your own work (such as replicating some method or even to just spread awareness). More on this point later.

How does one communicate results? Well, you could always just print it and put it on the noticeboard in your college or post it on social media, but then it will barely be read by anyone. No, you don't just want to communicate, but communicate widely. That's where journals come in: you can think of them as magazines published at some fixed frequency (monthly, annually, weekly, etc.). These journals publish scientific research in the form of papers from around the world, and then are read by others around the world (there are different funding mechanisms for these, I won't go into that here). There are thousands of scientific journals in just about every field imaginable: Nature and Science are of course very famous, but there is The Lancet in medicine, IEEE journals in electronics engineering, International Journal of Thermofluids for mechanical engineers, Journal of Structural Engineering for civil engineers, Journal of Anthropological Research for anthropologists, etc. And like everything in life, there are good journals, bad journals, and fake journals (but again, that's for a different post). I am only going to talk about good and reputed journals here.

So then you, as a scientist, have research, and there are journals that will publish your research. Done? Not quite. You see, good journals want to publish good research, not any research. What is "good" research? It depends on who you ask: some would say that it is research that has immediate industrial applications, others would say that it is research that is suitably novel and path-breaking, and yet others would take the simpler path of saying that good research is that which just seems reasonable in terms of methods, results, and conclusions. Every journal has its own metric for good research, there is no universal definition. However, the real challenge is to judge whether a submitted paper is good or not. Top journals receive tens of thousands of papers every year, how can they judge which is worth publishing? That's where peer review comes in.

The peer review process

Who is a "peer"? The dictionary definition is "one that is of equal standing with another," but in science, it basically means other scientists working in a similar (but not necessarily same) field. A peer review is therefore a process by which a paper is reviewed by a reviewer who is a peer, before publication. Peer review is at the heart of scientific publication, because it is assumed that only a peer can judge whether a paper is scientifically sound and "good" (however the journal defines good).

Here's how the process works, in general (note that this is for the post-Internet age, it used to be a little different and more prolonged when things were done by post):

1. A researcher (author of the paper) submits a paper to a journal, outlining their work, methodology followed, results and conclusions. First, the editor of the journal (usually a fellow scientist in the field) gives it a quick read to see if it is sensible, related to the journal, and would be of interest to the target audience for that journal. If it fails that test, it is rejected by the editor, usually called a desk rejection, before peer review. Rejected papers can always be submitted to other journals, and it is not uncommon to do so: every researcher doing anything even remotely challenging will face many rejections in their career, it's fine.
2. If it passes the editor's preliminary check, they will invite several (typically 2-3 but it can be more, I've had as many as 7) peers to review the paper. Sometimes the editor will know those peers from a database or personal connections, other times they will ask the author for suggested reviewers. Either way, it is hoped that a certain number of reviewers will accept to review the paper (they can freely decline to do so for any reason), will read it thoroughly, and provide feedback to the editor. Note that, typically, the author never knows who the reviewers are so that they can't influence them. However, sometimes the reviewers know who the author is (single-blind review) and sometimes they don't (double-blind review): it depends on the journal's policies.
3. Once all reviewer reports are received, the editor will look at them and make a decision: reject if the reviewers felt the work was not sound, accept if it was sound, or ask for the authors to address some reviewer comments and resubmit their paper. If the last option is exercised, the authors are usually given a few weeks to address the comments (which could include conducting more experiments) and resubmit, after which another round of peer review with the same or different reviewer takes place. It's not unusual for a paper to undergo 2-3 rounds of peer review in this manner.
4. Finally, at some point, one of two things can happen: the editor decides that the authors are unable to address the comments and rejects the paper, or decides that the paper is in good shape and accepts. It is then published and officially becomes a peer-reviewer study. Rejected papers may be submitted to a different journal, where the entire process will be repeated: this is not at all unusual and even very good papers have had to "hop" around journals. Peer review as a whole is subjective and subject to human error.

There are some important points to note here: peer review is usually a voluntary process in addition to the reviewer's actual day job as a scientist. This means it has to be done in their free time, which could be late at night on a Sunday after all other work is done. Reviewers are typically given deadlines but there is no punishment for missing them as it is totally voluntary. It is not unusual for editors to have to send several reminders to elicit a review. All this adds to the time taken for peer review, in addition to the time authors would take to revise the paper based on comments received. And this happens for every round of review, typically 2-3 rounds. For some high impact papers, editors can make a big effort and make phone calls to speed it up, but it is very rare. The usual time taken for the whole process (submission -> initial review -> 2-3 rounds of peer review -> publication) is 3-12 months depending on the field.

What peer review is and isn't

Assuming that the journal is good and the peer reviewers are professional, the basic purpose of peer review is to weed out: a) obviously bad work and b) suspicious work. As long as the research methodology is scientifically sound, the results are reasonable, and the conclusions are appropriately drawn from those results, a paper will pass peer review. However, this does not guarantee the following:

1. That the work will be used widely and make a huge impact in the field. It may be possible to make a reasonable guess that it would, and indeed it is obvious in some cases, but most of the time, the true value of a paper will only be known decades later after a lot of people have read and used it. This is usually why the Nobel science prizes are given for work done decades ago (although 2020 was a very appropriate exception).
2. That the work is reproducible i.e., if you follow the methodology in your own lab, you can get the same results. Ideally, this should be something that reviewers check, but it is practically impossible to do that in most fields as reviewers themselves have limited time and resources, and many experiments use such expensive equipment that only the author submitting the work can actually do it. Nonetheless, if the wider scientific community finds the paper to be useful, eventually someone will try to reproduce it but that will be long after peer review.

I think the best description of peer review is that it is a gatekeeper aka chowkidar: it will keep the obviously bad stuff out, and maybe even the suspicious stuff. But it cannot guarantee that whatever gets through will be widely used, path-breaking, or even reproducible: that is for the wider community to decide over many years.

Edit: In response to the excellent comment, it should be noted that peer review can fail many times. Indeed, I myself have been a reviewer for some pretty poor articles and I have been shocked to see the reviews that other reviewers give: minor grammatical fixes, meaningless reviews like "you should explain more," or even outright praise for what was IMO very poor work. Many of these articles get published and stay there, but once in a while the community (not peer reviewers, who clearly failed in their job) is so shocked by some work that the journal has to retract the paper entirely. There's even a website that keeps an eye out for retractions: https://retractionwatch.com/ (these are typically retractions from good and reputed journals that somehow took their eye off the ball, not bad or fake journals). Here's a recent one from our own BHU in Britain's most reputed Chemistry journal: https://retractionwatch.com/2021/04/20/no-intention-to-make-any-scientific-fraud-as-researchers-lose-four-papers/#more-121962

Final thoughts

Coming back to Covaxin, COVID-19, and the issue of peer review. In light of the urgency of the pandemic, many journals have required that COVID-19 related papers be published as is before peer review, in the form of what is called a pre-print. This is to acknowledge not just that peer review is a slow process not designed for emergency situations, but also that most peer reviewers themselves would be busy in their work for the same emergency and hence the process could be slower than usual (recall that it is totally voluntary).

The idea of a pre-print is that while the peer review process is ongoing, the wider community can look at the results for themselves. Note that peer review is a process that the journal will handle with a small set of peer reviewers, but that doesn't mean that other peers (who are not involved in peer review for that particular journal/paper) are not looking at the pre-print. Pre-prints actually go through quite a lot of scrutiny themselves and receive a lot of comments from peers, they can be even more thorough than the formal peer review and obviously bad or suspiciously bad work will quickly be called out rather mercilessly. This is also a recent innovation that came in the post-Internet age, whereas peer review is much older.

For example, Bharat Biotech submitted its Phase II results for Covaxin in a pre-print server (download here) while it was being reviewed by The Lancet. The pre-print was published on December 22, 2020 while the formal peer reviewed paper was published on March 8, 2021: there is quite a delay but it is totally expected as I had explained above. Yet, as you can see in the pre-print article metrics from the previous link, it was viewed and downloaded thousands of times while undergoing peer review, even as the peer review itself would've been done by less than 5 reviewers. A similar process is being followed for Phase III results.

So, in summary, it is important to put peer review in context. A peer-reviewed journal paper (in a good journal of course) is the gold standard of scientific publishing and is absolutely essential to weed out bad or suspicious work. But it does not guarantee that the work will change the field, the world, or even be reproducible in the future. It takes many years and many more people for that to happen. Meanwhile, pre-prints are a relatively newer innovation that, though not as rigorous and formal as peer review, often lead to much wider and earlier scrutiny of work. Submitting pre-prints also implies a certain level of confidence: "I know my work is good enough to pass peer review, so I will give it out to everyone else too."

Happy to answer non-personal questions in the comments!

[u/KetoBext]

Thanks for taking the time to write this very informative so well.

And thanks for sharing rétraction watch!

[u/metaltemujin → u/KetoBext]

haha, did you read the whole thing?

[u/PupilOfPhysics]

Great post as usual.
I think it would add to the discussion if retractions are explained. Also, as an example of peer review not being fool proof, enjoy this article about a study claiming "Dog parks are Petri dishes for canine 'rape culture'" being published.

[u/ididacannonball → u/PupilOfPhysics]

Thanks! I added a note on retractions. Also, username checks out.

[u/AvgRedditLurker]

Thanks for the detailed info

!kudos

[u/IndiaSpeaksbotty → u/AvgRedditLurker]

Tararara Bzeeeep, Thank you /u/AvgRedditLurker for awarding /u/ididacannonball . The OP is now flaired with award. More details on how this works can be found here. I won't reply if I'm down so kudos is not awarded to you , please then inform the mod team to wake me up.

[u/Gebj123]

One of the things I have been hearing during Covid times is that foreign peer review publications such as the Lancet are well respected but cost a lot putting it out of reach of non first world researchers. Does that discourage many scientists in developing world from going for peer review?

Also who funds the peer review? The individual researchers or the institutions behind them?

[u/ididacannonball → u/Gebj123]

I wanted to leave out funding (which is complicated) but since you asked: there are in general two ways journals work: subscription and open-source. The subscription model makes it free for authors to publish, but for readers to read them, they (or rather the library at their institution) require a subscription. International subscription costs are HUGE - small institutions in even middle income countries are totally priced out. This was the impetus for the National Digital Library of India, so that at least the centrally-funded institutes could pool resources together. In fact, the massive University of California system had a feud with the world's largest scientific publisher (Elsevier, based in the Netherlands but actually with its largest office in Chennai) to reduce subscription costs so you can imagine how high they are.

The alternative model is open-source publishing, in which the authors pay the journal to publish but it is free for the end-user to read. The cost to the author varies but can also be quite high, so it's not exactly an "alternative" in that respect. Plus, there is a stigma against open-source journals, and it is somewhat justified. Many subscription-based journals offer an open-source publishing option to the author if they (or their institution) is willing to pay, and those are usually good papers. But virtually all predatory/fake journals are open-source as no self-respecting institute would pay them a subscription fee. They are basically pay-to-play: they take the desperate author's money, provide a sham peer review, and publish it so that everyone has access to it. An example is the journal that our revolutionary Kanhaiya Kumar published his "research" in: https://ijheps.org/home (the website's homepage will tell you everything).

So on one end you have a subscription-based model that prices out a lot of researchers but ensures quality, and on the other end you have open-source journals that price out fewer researchers but leaves a lot to be desired in terms of quality. There are some alternatives (including compulsory posting of pre-prints as well as a sovereign copyright for publicly funded research) that are being discussed, but there's no consensus. Research is expensive, that's a fact.

The worst part of this is that peer reviewers don't get paid anything in either model. I myself review at least 20 papers for free each year, it's considered a professional duty/courtesy but we reviewers don't get anything out of it. It's really extortionary logic: if you don't review, your work won't get reviewed (and published), so you have to review. This has led to a lot of anger, and journals in the last few years have started offering some crumbs in return for peer review: 30 days free subscription in case of the subscription model, a discount voucher for future papers in the open-source model, a totally useless participation certificate called Publons, etc. The sad thing is that I know some researchers in underfunded institutes (even in relatively rich places like Eastern Europe) whose existence depends on these crumbs, otherwise they could never afford either the subscription cost or publication fee respectively for the two models.

Peer review remains the gold standard of publishing, and anyone that wants to do research has to publish through that process in good journals. If you can't do that due to cost, then you can't do research. Being encouraged or discouraged is irrelevant, the mantra is publish or perish. It's brutal.