Evaluating Science or Evaluating Gender?

By Anne E. Lincoln, Stephanie Pincus, and Vanessa Schick

Impartiality is the cornerstone of scientific inquiry. As scientists, we base our entire system of knowledge collection and evaluation on the standard of universalism, the expectation that scientific claims and contributions are evaluated independently from the personal attributes of scientists. Yet it has also long been recognized that science is stratified, with research facilities and rewards unequally distributed among scientists. What sociologist Robert Merton termed the “Matthew effect” in science, the notion that prominent scientists often receive more credit than less well known scientists doing similar work, finds that advantages among scientists accumulate to bring resources necessary for future achievement to those persons who have already achieved.

Awards and prizes are critical in shaping the trajectories of scientific careers. Yet, men win more science awards than their representation in science careers would otherwise predict (see The RAISE Project, raiseproject.org). Recognizing the importance of the acknowledgment of one’s peers to scientists’ careers, in 2006 the National Academies called for more women scientists to be nominated for awards and leadership positions. While it is plausible that part of the explanation for the gender disparity in award receipt lies in the nomination process, can it be said that men do research that is inherently more important than women’s research? Or might some sort of Matthew effect for gender intervene with the dispassionate evaluation of scientific achievement?

In fact, a large body of evidence suggests that social factors do influence the review and valuation of women’s efforts differently from those of men. Indeed, social scientists find that when men and women of the same abilities and characteristics are compared, evaluators of both sexes tend to evaluate men more favorably than women or hold women to a higher standard.  

The notion that gender influences the evaluation of scholarly work was highlighted in a recent study of manuscript acceptance rates at the journal Behavioral Ecology. In 2001, this journal switched from a single-blind manuscript review process, in which reviewers knew the names of manuscript authors, to a double-blind process, providing a natural experiment to test the veracity of the universalism standard. Researchers Budden and colleagues found that the manuscript acceptance rates of articles first-authored by women increased by 7.9 percent after the editorial procedural changes, which they attributed to the change in review policies. What explains these findings? Social psychologists put forward the similarity-attraction thesis–that people are most comfortable with others who are similar to them, particularly in terms of race and gender. Birds of a feather may indeed flock together.

The American Physical Society provides an ideal circumstance to examine the evaluation of scientific work because APS keeps a record of the selection committee members for each prize. APS first began recording the selection committee members consistently for its awards in 1997. Between 1997 and 2009, APS reported selection committees for 42 different prizes given to 464 individuals, excluding student awards and awards that only women can receive (e.g. the Maria Goeppert Mayer Award). Selection committees averaged 5 members and 88 percent male. Exactly half of the committees (232) were comprised solely of men; the next most common arrangement was for one woman to be part of a five-person committee, which occurred about 36 percent of the time. All but 12 of the selection committees listed a chairperson; men chaired 86 percent of the committees.

During this period, men won 96 percent of the APS awards given to practicing physicists. That is, women won 20 of the 464 prizes bestowed during this period. On its face, this is not a particularly surprising finding, given that physics is a male-dominated discipline, but the figure still seems a bit low. (For comparison, the 2000 Census reported that men comprised 86 percent of astronomers and physicists; as of 2006, women comprised 13 percent of physics faculty in the United States and 6 percent of full professors). Given what we know about the relevance of gender in groups, does the gender composition of the committee have any relationship to the gender of the winner?

In short, yes. Logistic regression analysis, which calculates the probability of an event occurring, finds that the likelihood of a woman winning a physics award nearly doubles with the presence of each woman on the committee. So, should committees simply add women to ensure that women receive consideration for their achievements?  Surprisingly, the answer is no.


Gray arrow   Larger Graph  

Taking the analysis a step further, it turns out that committee chairs are highly important to the evaluation process. Women are 65 percent less likely than men to win an award if the selection committee chair is a man, regardless of the number of women on the committee. Indeed, women won an award 3.6% of the time from a male-chaired committee, but nearly three times as frequently under a female-chaired committee (9.5% of the time), a statistically significant difference (Figure 1).  

Put another way, men are nearly three times more likely to win a prize from a male-chaired committee than from a committee with a female chair. The opposite is also true–a woman’s chance of winning nearly triples under a woman chair. We should note that women winners and committee chairs are not concentrated in a few awards, so the disparity is not a matter of women and men specializing in different types of physics.

When we consider these findings, we must note that of the 63 committees they chaired, women chairs conferred awards to 57 men. Thus, while women chairs are more likely to reward women with a physics prize than are men chairs, men still dominate 9 to 1 as prizewinners from women-headed committees. In other words, although committees chaired by women tend to be “more impartial,” all committees tend to disproportionately confer awards to men.

With all research, there are limitations. While members of the selection committee are prohibited from making nominations, we do not know whether more women are nominated for awards when committees are headed by a woman (and vice versa for men). Since we do not know the relative proportion of men and women who were actually nominated, how certain can we be of the finding that committee chair gender predicts the gender of the prizewinner? One strength of this conclusion is that nominations remain active for three consecutive award cycles (typically annual or bi-annual) and selection committees regularly change members, so nominees who do not win one year will continue to be considered for the same award by a different committee in two subsequent award cycles. By implication, this suggests that gender of the committee chair does indeed influence the award review process.

APS is one of the few societies with such a transparent award process, so we are precluded from making firm comparisons with the award committee process of similar organizations. However, data collected through the RAISE Project permits comparison of the gender distribution of award recipients across societies. Comparing the percentage of female award recipients to the current National Science Foundation data on the percentage of women employed in physics, we found that the percentage of women receiving awards is half of the percentage of women within the field, a figure that is consistent with similar societies, such as the American Physiological Society, the American Association for Cancer Research, and the American Association for the Advancement of Science. This suggests that their award process may function similarly to APS. Thus, it is possible that the same phenomena may be present across various organizations.

We do not have a decisive answer for why the gender of a committee chair is so strongly predictive of the gender of the winners. Given that physics is a male-dominated discipline, we could conjecture that when a woman is a committee chair, her gender is made more salient to the other committee members, who become more amenable to voting for a woman. But if this were the case, why then does the gender of the committee chair–male or female–nullify the positive effect of the presence of other women on the committee? If committee chairs make the final decision, it may be that committee chairs simply do prefer persons of their own gender, but if the vote is based on consensus, committee chairs serve as a symbolic stand-in for the appropriate type of winner.

How can professional societies increase impartiality? There is no “one size fits all” answer. While some research finds that increasing the size of review committees leads to a more diverse slate of winners, our study finds this not to be true for APS awards. Ultimately, blind review remains the gold standard for reducing biases.

Far from being gender-blind, the present nomination process for APS prizes has 20/20 vision. Selection committees rely upon at least two nominating letters describing the nominee’s qualifications relevant to the award, the specific scientific work to be evaluated, and a list of the most important publications. In a tight-knit community like physics, knowledge of this body of work may not make it possible to completely blind the review process. However, a first step that selection committees might take would be to require a separate statement summarizing the specific research to be evaluated, devoid of reference to the author. This statement could be read first and then ranked by the committee before any supportive materials were considered. Both nomination and committee evaluation processes could benefit from scrutiny for ways to increase impartiality.

Fig 2 back page web

Gray arrow   Larger Graph  

A closing thought: The proliferation of women-only awards in many disciplines, like the well-intentioned Maria Goeppert Mayer award in physics, may have the unintended effect of camouflaging women’s otherwise low receipt of awards (Figure 2). By only recognizing women, such awards may not only contribute to the impression that nomination and review processes are unbiased, but also create a sort of “ghetto” for female scientists, whose work is not seen as equivalent to men’s achievements. Gender-blind (and indeed, race-blind) review will ensure that birds of any feather truly have the opportunity to flock together. Science will be the better for it.

Anne E. Lincoln is Assistant Professor of Sociology at Southern Methodist University where she studies organizations and gender differences in labor market outcomes, with particular interest in science careers. A presentation based on this summary was presented at the annual meetings of the Southern Sociological Society, New Orleans, LA, April 4, 2009. Email: Anne E. Lincoln.

Stephanie Pincus, M.D., M.B.A, Founding Director of The RAISE Project is Professor and Chair Emeritus of the Department of Dermatology at the University at Buffalo.  She has held numerous leadership positions in professional and governmental organizations where she has been a longstanding advocate for professional women and an expert in gender and workforce issues.

Vanessa Schick, M.Phil. is the RAISE Project Coordinator at the Society for Women’s Health Research. She is currently a doctoral candidate in Applied Social Psychology at the George Washington University where she studies women’s health and gender issues. Email: Vanessa Schick

The authors thank Florence Haseltine, Ph.D., M.D. for technical assistance and Alyssa Fornara for data collection.


APS encourages the redistribution of the materials included in this newspaper provided that attribution to the source is noted and the materials are not truncated or changed.

Editor: Alan Chodos

June 2009 (Volume 18, Number 6)

APS News Home

Issue Table of Contents

APS News Archives

Contact APS News Editor

Articles in this Issue
Kirby Succeeds Franz as APS Executive Officer
Brinkman Nominated to Head DOE Office of Science
Physicists Bring Their Moxie to National Intelligence
APS Launches Free News Service for its Members
Scientists Issue Call for More Public Engagement
Session Explores the Future of Global Physics Projects
Speakers Reflect on Telescopes Old and New
Pulsars Proposed as Direct Gravity Wave Detectors
Letters to the Editor
The Back Page
Members in the Media
This Month in Physics History
Zero Gravity: The Lighter Side of Science
Profiles in Versatility
International News
Washington Dispatch