Reviewing Peer Review

By Katherine Wright and Matteo Rini

Peer review—the evaluation of scientific work by experts in the field—is the main method by which papers are published, grants assigned, and scientific laurels won. As the old joke goes, peer review is like democracy: a flawed system that’s the least worst of all options. Celebrating the importance of peer review and working on its failings are the goals of Peer Review Week, a yearly event inaugurated in 2010 that brings together learned societies, publishers, researchers, funding agents, and other actors involved in scholarly publishing.

This year’s Peer Review Week addressed the role of identity in peer review: how personal and social identities affect the process. The question of identity is related to an often-voiced concern about peer review—that the presence of a bias might disfavor certain groups of people, such as women, people of color, researchers from lesser-known institutions, or those from less developed countries. Here is a sample of what Physics Magazine heard at the sessions that took place over the week.

Confidential or Transparent?

Presently, the dominant peer review model for the physical sciences is “single blind,” meaning that the referees are kept anonymous but the authors are not—their names are visible to the referees. Many scientists, however, say that this system is susceptible to unfair bias—papers may be judged, consciously or subconsciously, based on the pedigree of the authors, on their geographical information, and even on their ethnicity. “Researchers move from Oxford to Kenya and suddenly struggle to publish,” says Rebecca Lawrence, the Managing Director of F1000—an open-access publisher of life sciences.

Lawrence spoke at a session organized by APS, which discussed the pros and cons of blind, or confidential, peer review and of alternative approaches based on transparency.

Lawrence said that F1000 follows a fully transparent approach, disclosing referee names, referee reports, and other details of the editorial process. “A lot of evidence suggests that transparency makes peer review more constructive,” she said. With everything laid out in the open, “people feel they have to behave a little better,” so they keep their biases at bay and deliver higher-quality reviews. Lawrence said that many researchers in Africa chose transparency exactly because “they want to be treated on equal footing as everyone else.”

Beverley McKeon, a professor of aeronautics at the California Institute of Technology and co-Lead Editor of Physical Review Fluids, said that the main concern about transparency is that it may favor established researchers. “In an ideal world everything is transparent and fair, but will junior researchers feel confident in a transparent peer review?” Fear of retaliation, for instance, may prevent a junior researcher from rejecting the work of more senior scientists.

Dan Kulp, Editorial Director of the American Chemical Society, said that his organization is initiating various pilot projects to test new forms of peer review. He finds that transparency has many advantages. “The change, however, cannot happen overnight,” he said. A first step could be a compromise, such as making the referee reports public, while keeping the referee anonymous.

All panelists agreed that—transparent or confidential—the peer review system must be urgently upgraded by making the reviewer pool broader and more diverse. One way to do that is to provide editors with AI-based search tools for recommending referees in a way that could be less prone to personal biases and to “homophily”—the tendency of people to seek out those who are similar to themselves.

Session chair Daniel Ucko, an editor with Physical Review Letters, took the pulse of the audience with polls asking attendants whether they felt transparency or confidentiality was the more important value. The discussion appeared to change the audience’s sentiment, with an initial, clear majority of votes going to confidentiality turning into a tie by the time the session ended.

What’s in a Name?

Biomedical engineer Elsie Effah Kaufmann of the University of Ghana goes by many names. Some days it’s Elsie, others it’s E. E. Kaufmann, and on yet others it’s E. A. B. E. Kaufmann. In Ghana, where Kaufmann was born, “we generally are named after the day [of the week] that we are born,” said Kaufmann, who spoke at the session Identity in Peer Review organized by IOP Publishing. “That leaves a lot of people with the same name.” Ghanaians add and drop names when they get married, distinguishing themselves from others. But, Kaufmann noted, this cultural norm can cause problems for her and her academic colleagues.

In academia, a person’s identity is typically tied to the name that they use when they start their PhD or publish their first research paper, something, she said, that many students fail to recognize until it’s too late. “Our training in becoming researchers doesn’t orient us to understand that our identity is tied to this name.”

Having multiple names is also true for other cultures, such as Western countries where women often change names after marriage. These practices can cause issues for peer review, as it can make it hard for editors to track the person’s publications and determine their expertise. If they can’t track that work, that person may not get invited to be a referee or sit on a journal’s board. There are ways around this problem, such as creating a profile on a platform such as ORCID, which Kaufmann said she now advises all her colleagues to do.

Others in the same IOP session also extolled the importance of having an ORCID profile for participation in peer review. “If Elsie wants to use 12 versions of her name, [ORCID gives her] one identifier for all her science, keeping it together and making sure [journals] know who she is as an individual,” said Jasmine Wallace, who manages peer review for journals published by the American Society for Microbiology.

Naming traditions aren’t the only obstacles facing Kaufmann and researchers like her who focus on local issues, such as public health and engineering problems. Editors and reviewers have told Kaufmann that these studies “aren’t of interest to our readers,” and that she should submit her work to “local journals,” despite there being no Ghanaian-owned academic publishers. Researchers need to “have some empathy,” she said. “I am all for improving the quality of research, but the variety of research should also count.”

Involving Early Career Researchers

Early career researchers (ECR) are the future of science, but this role isn’t appropriately reflected in their participation in peer review, according to Smita Jain of Cactus—a company that offers editing services for researchers. Jain chaired the session Should Early Career Scientists Become Referees? which discussed why ECRs should be part of peer review and what gaps hinder their participation.

First and foremost, “ECRs are good for science,” said Thomas Agbaedeng, a biomedical researcher at the University of Adelaide in Australia. These researchers come in with “fresh ideas and perspectives,” which could help support innovative research directions, said Agbaedeng. Compared to more experienced researchers, ECRs are generally less attached to the “status quo” and better able to judge papers independent of the authors’ pedigrees, said Asli Telli, an outreach specialist at the University of Siegen in Germany. Participating in peer review will also be very beneficial for the researchers themselves, as judging others’ work helps them “develop an external eye for their own research,” Agbaedeng said.

Several barriers, however, prevent ECRs from becoming referees. Some may not be sufficiently confident in their scientific or English-speaking skills. They may also be unfamiliar with the review process. Finally, they are typically less visible to editors and editorial board members than more senior researchers.

Addressing these gaps requires a “change of culture around ECRs,” Jain said. She, Agbaedeng, and Telli offered several ideas for increasing ECR input. Journals, for instance, should offer peer-review training resources and seek to include more ECRs in their editorial boards. Senior researchers should involve ECRs by offering them the possibility to write joint reports. Agbaedeng also urged less experienced researchers to be proactive and become more visible through social media or through services like Publons, which tracks a scientist’s review and editorial activity. “You’ve got to get out there, spread your wings and fly,” he said.

A longer version of this article was originally published by Physics Magazine on October 1, 2021.

Katherine Wright is the Deputy Editor of Physics Magazine. Matteo Rini is the Editor of Physics Magazine.