APS News

Is Double-Blind Review Better?

By Shannon Palus

Referees reviewing a paper are always anonymous. But have you ever wished you, an author of a paper, could remain anonymous to a peer reviewer?

Maybe you have one or a few people who can’t stand you for personal reasons. Maybe you’re an up-and-comer in your research area — and want to be on a level playing field, with just as much opportunity as someone with a famous name. Maybe you are in the minority — i.e., not a white male — and want to buffer yourself against implicit bias.

With those benefits in mind, Nature Geoscience and Nature Climate Change decided to offer a double-blind option to authors: Take out all identifying materials from your paper, and the editors will not tell your paper’s reviewer your name. After 21 months, fewer people had used the option than expected, and there was no notable difference in the quality of reviews. But the response from authors was positive, so as of March of this year, Nature started offering authors a double-blind peer review process across all its journals.

But double-blind peer review isn’t a one-size-fits-all-fields proposition. Many factors affect the viability of the option.

Nature Physics is offering the option, as part of the Nature decision. To our knowledge, it’s the only physics journal to do so currently. The journal’s chief editor, Andrea Taroni, isn’t sure that it will be popular. “Physicists work in a relatively open and collegial way and rely extensively on arXiv,” Taroni wrote in an email.

We can also turn to history for one scenario of how it will work out. The APS Physical Review journals ran their own two-decade experiment in double-blind review. It did not go well.

Double-Blind Peer Review in Physics

The impetus came in 1980 from the APS Committee on the Status of Women in Physics (CSWP), recalls Physical Review Letters (PRL) Editor Emeritus Stanley Brown. “They thought that there was a possibility that papers written by women were subject to more critical review than papers written by men.”

Indeed, recent studies confirm what the committee then suspected. A 2012 study in the Proceedings of the National Academy of Sciences showed that identical application materials for a lab position were more favorably read when the name at the top of the resume was “John,” rather than “Jennifer.”

So the CSWP suggested that double-blind review should be the standard. But, according to Brown, the editors felt that trying to hide every single author identity would probably be unsuccessful, and authors too recognizable, to be effective. Even in the days before researchers routinely uploaded their papers to the arXiv, a culture of openness and connections made authors easily identifiable, says Brown. “Communities were frequently small enough that you could infer.”

Still, the editors thought offering an option “would be an interesting experiment.” So, in 1980, the Physical Review editors instituted the policy for their (then) five journals. The upshot — only an update to the submissions guidelines and a sentence in an editorial in PRL.

And publication at the Physical Review journals hummed along for 22 more years with no additional note that Brown can recall, positive or negative, for the double-blind option. In 2002, he thought he should check up on the results of the trial.

Between 1993 and 2001, authors on 121 papers requested the double-blind option. (Brown doesn’t have data from before then.)

That’s 0.06% of the total manuscripts submitted. Of those, only 7 were published. Compare that to the acceptance rate for Physical Review journals in general — which is 60%. “It amounted to less than one paper per year,” Brown says.

No editor APS News spoke to could spin that as a success.

“In a nutshell it did not add to the efficacy of the process and had several other drawbacks,” recalls Jack Sandweiss, a former lead editor of Physics Review Letters.

“It was difficult to manage. It was difficult to maintain,” says Reinhardt B. Schumann, the current managing editor of PRL.

Brown sums up having the option as “basically cosmetic”: a way to answer anyone who asked about the journals’ efforts to prevent discrimination in the peer review process, which no one ever really asked about anyway. Ultimately, “It wasn’t practical in physics,” says Brown. “I know that in other fields other journals routinely use it.”

Peer Review in Other Fields

Some of the major journals in philosophy use triple-blind review, in which even the editor is anonymous. In art history, double-blind is used frequently. The American Economic Association used to have a double-blind system for its journals, but in 2011 switched to single-blind. The reason? Search engines made it too easy to uncover an author’s identity.

But in science, single-blind is the norm. The journal of the same name, Science, only offers single-blind.

It goes against the grain, then, that the field of computer science is largely split between the two systems. That’s according to Roch Guerin, Chair of the Association for Computing Machinery’s Publications Board Conference Committee, which was tasked with a review of the respective roles of ACM’s conferences and journals. (Yes, conference abstracts are peer reviewed in computer science).

Computer science, like physics, enjoys a fair amount of sharing and openness (though there is no arXiv equivalent). So that throws out the argument of it being easy to infer the author.

The double-blind system might flourish simply because a different model is not as deeply ingrained in the culture: The field is young, with its first journals born in the 1950s and 1960s [1]. “Computer science was a science, but also an experimental discipline,” says Guerin.

Whether double-blind is the way to go over single-blind is an experiment with no clear conclusion.

“People on the double side believe it leads to a more objective outcome,” says Guerin, who has sat in on a lot of discussions debating the merits of each. “Other people actually feel that knowing who the authors are provides additional qualifications.”

Important to the functioning of the double-blind method, when it is used, is that a computer science conference or journal that operates under double-blind peer review does so across the board. It’s not an option, it’s a modus operandi.

And that might be crucial to its success.

Take the example of the Nature Geoscience trial, the one that spurred Nature to provide a double-blind choice. A reader survey a couple years ago showed an “overwhelmingly positive” response to the idea of offering double-blind, says Heike Langenberg, chief editor of Nature Geoscience.

But when the trial went into effect, only around 15% of the submissions opted for double-blind review, says Langenberg. The problems, she speculates, is that people don’t know about the option before they start the submissions process and just want to get it done quickly (anonymizing a paper isn’t hard, but it does take a few minutes).

Plus, many people think that their identity can be guessed anyway.

“People overestimate how well they can guess identities,” says Langenberg. Right now, her evidence is just anecdotal, but she’s collecting data to find out for sure. The journal has started asking referees if they can guess authors, though results might not be available for a few years.

As for the benefits for geoscience researchers whose data are less identifiable? According to Langenberg, the double-blind process gets to the point of what a good peer review should be, plain and simple. “It just takes away anything but the science in the paper.”

Is Double-Blind Worth Fighting For?

Today, double-blind peer review isn’t a topic of discussion among current APS Committee on the Status of Women in Physics (CSWP) members, or those of the APS Committee on Minorities in Physics, say the chairs of those committees.

“I wouldn’t be surprised to hear there’s bias in the peer review process,” says Ashley DaSilva, a member of CSWP and postdoc at the University of Texas at Austin. “[G]ender differences in publication record start well before the peer review.”

Double-blind peer review may strip a paper down to “just the science” — and may well be an effective and fairer option for some fields. It may even prove to work out nicely for Nature Physics. As Taroni wrote in an email to me, “obviously only time will tell!”

But, there are so many interactions between physicists — those every day in labs, classrooms, job talks, scientific meetings, and at the water fountain — that determine who gets to publish science, and therefore which science makes it through the review process. In those interactions within the community, as is the case in peer review within a field that relies on sharing, collaboration, and working with other human beings to move forward, it is impossible to strip our identities out.

Reference

1. www.oxfordjournals.org/our_journals/computer_journal/historypaper_mills1.html


©1995 - 2017, AMERICAN PHYSICAL SOCIETY
APS encourages the redistribution of the materials included in this newspaper provided that attribution to the source is noted and the materials are not truncated or changed.

Editor: David Voss
Staff Science Writer: Emily Conover
Contributing Correspondent: Alaina G. Levine
Art Director and Special Publications Manager: Kerry G. Johnson
Publication Designer and Production: Nancy Bennett-Karasik