NSF Task Force Fields Comments On Broader Impacts Criterion

By Michael Lucibella

A recently announced review of the National Science Foundation’s grant process has reignited discussion about whether it should award grants based solely on scientific merit or whether it should weigh societal issues as well.

NSF receives about 45,000 funding requests each year, of which about 11,500 are successful. Each proposal is evaluated based on two main criteria. The first is Intellectual Merit, which looks at the proposal’s potential to advance knowledge in a given field. The second and more controversial criterion is Broader Impacts, which weighs such issues as whether the proposal would promote education, broaden the participation of underrepresented groups in science, enhance scientific infrastructure, improve scientific understanding or otherwise benefit society in some way.

The Broader Impacts criterion, introduced in 1997 and renewed in 2007, has been controversial within the scientific community. In early spring of 2010 the National Science Board, the oversight body of NSF, announced that a task force would be reviewing the criteria for awarding grants. It released a draft of the proposed updates that broadened existing criteria to include advancing national security interests, increasing partnerships between academia and industry, improving US economic competitiveness and more explicitly calling for efforts to improve education and public outreach. The proposals can address all or some of these criteria, and can do so either through the research itself or through an ancillary program also funded by the grant.

In July, the task force asked for comments, and received 5,100 responses. The National Science Board is currently mulling over the suggestions it received before announcing final updates or changes to the policy.

“The scientific community has expressed a range of opinions about the Broader Impacts criterion. Most understand that this criterion helps to ensure that there is a connection between scientific research and society. However, many PIs and reviewers have asked NSF to provide clearer guidance about how they should address this criterion,” Joanne Tornow of NSF, who is on the NSF’s Merit Review taskforce, said in an email, “The Broader Impacts criterion helps to connect science and society, and leverages research investments more broadly.”

Objections to the criteria range from concerns about defining science in the national interest, and that requiring scientists to engage in outreach can distract from research.

In early July, Science and Nature both ran items critical of the Broader Impacts criterion. In Science, a letter by Robert Frodeman and J. Britt Holbrook, both from the University of North Texas, said that requiring research to articulate the national importance of a project would impair “scientific creativity and autonomy.” Similarly, Nature’s columnist Daniel Sarewitz said that neither “project leaders nor peer-review panels are likely to have sufficient expertise to really understand a single project’s capacity to connect to a persistent challenge such as increasing the nation’s science literacy or economic competitiveness.”

Also in early July, the four members of the APS presidential line sent Raymond Bowen, chairman of the National Science Board, an open letter outlining their concerns.

“We draw our inference from the directive contained in the revised criteria that NSF should align its research programs with a set of national goals that emphasize the worth of the envisioned applications of research rather than the worth of the scientific knowledge emanating from research,” the letter read. It adds that all other federal agencies that support research support it in line with their own agency’s mission, and the NSF should sponsor “scientific excellence wherever it leads.”

Other scientists have criticized the Broader Impacts criterion for its requirement that grant recipients engage in outreach and education. Even some physicists who actively engage in outreach and education have said that it’s a distraction from the core goal of conducting research.

“We should not discourage people from doing outreach, but requiring people to do outreach is just plain silly,” said Lawrence Krauss of Arizona State University. “I think we should encourage it, but we should give carrots and not sticks.”

Others have been more supportive of the policy, saying that outreach and education are important to encourage.

“I don’t think it’s a bad policy. I don’t think it does any harm,” said Chad Orzel a professor at Union College who has helped to review a number of NSF proposals, “A lot of the things it leads to are very uninspired, very standard outreach things. But I don’t think that really hurts anything…There are a few that are doing something interesting or doing bigger outreach things than [they] would be otherwise. In that sense it’s probably a good thing on balance.”

The Task Force on Merit Review acknowledged the controversy in its charge to reevaluate the program.

“Anecdotal evidence suggests that this requirement can be very confusing to the research community, which continues to express frustration in interpreting and thus responding effectively to the Broader Impacts criterion when creating a proposal,” the task force’s founding charge read. “[T]here appears to be substantial confusion about how best to meet the requirements of this criterion, whether on an individual project level or at the proposing institution level.”

Despite the objections of some researchers, it seems unlikely that the NSF will strike the criterion outright. In all likelihood the NSF will issue clarifying guidelines similar to the draft issued in June of 2011.

John T. Bruer, President of the James S McDonnell Foundation and co-chair of the merit review taskforce, said at July’s National Science Board meeting that “a significant proportion of the comments suggested eliminating the B.I. entirely. That’s not possible.”