|
In response to falling success
rates and rising numbers of grant applications, the National Institutes of
Health (NIH) has undertaken several initiatives aimed at overhauling and
improving the peer review system.
The problem of declining success rates for NIH grant applications has been
recognized for some time. Contributing factors include a rapid increase in
the number of applications submitted to NIH, an increased number of
researchers seeking grants, and a budget that has been declining in
inflation-adjusted dollars since it peaked in 2004. Those factors coupled
with the reality that many grant applications are not funded until after the
first or second amendment have caused significant challenges for the review
system. More applications have meant a larger workload for reviewers,
causing them to devote more time away from their own research programs in
order to participate in grant review. The increasingly complex and
interdisciplinary nature of biomedical research poses yet another challenge
in obtaining appropriate expertise on review panels.
Integrated Review Group and Study Section Alignment
The most recent examination of the peer review system took place when the
NIH Panel on Scientific Boundaries for Review restructured the integrated
review groups (IRGs) within the Center for Scientific Review (CSR).
Completed in 2004, the Panel’s goal was to revise the IRG structure to
achieve a systems and disease based focus that would enable more
translational research.
This year, CSR has again undertaken a review of the IRG alignment, holding a
series of Open House workshops where members of the extramural community are
invited to participate in a discussion with NIH officials focusing on two
main questions:
Is the science of your discipline, in its present state, appropriately
evaluated within the current study section alignment?
What will be the most important questions and/or enabling technologies you
see forthcoming within the science of your discipline in the next 10 years?
As the Open House workshops are held and reports are generated, information
will be posted to the NIH website (http://cms.csr.nih.gov/AboutCSR/Openhouses.htm).
The APS has participated in these workshops and has provided the NIH with
information collected through online member surveys. These surveys collected
information on the two questions posed above, as well as a number of other
aspects of the peer review system including streamlined review and service
on study sections. The results indicate that 55-60% of APS members in the
sections surveyed feel that the science of their discipline is appropriately
evaluated within the current study section alignment. Interdisciplinary
research and integrative physiology were the areas most often identified as
underserved in the current system, but also as areas that will be vitally
important to moving science forward in the next 10 years.
In addition to the Open Houses, CSR plans to carry out ongoing internal
reviews by examining one IRG per month. With 24 total IRGs, each one is
reviewed within NIH every two years.
Other reforms under consideration
In addition to examining the alignment of IRGs and study sections, NIH has
also been considering a number of other initiatives aimed at speeding and
improving grant review. These include changing the requirements for grant
appendix materials, shortening the review cycle for new investigators to
allow them to submit revised applications in consecutive cycles, moving to
electronic grant submission, and considering shortening the R01 grant
application from the current 25 page limit. In addition to electronic grant
submission, CSR is developing a system that will automatically refer grants
to IRGs based on a sophisticated computer search that extracts referral
requests from cover letters and analyzes text.
CSR is also experimenting with alternative formats for reviews, including
asynchronous electronic discussions (secure online chatrooms where reviewers
can post comments) and video enhanced discussions, where reviewers meet via
a computer based videoconferencing system. The goal of the alternative
review formats is to facilitate recruitment of reviewers with appropriate
expertise by eliminating barriers to participation in face to face meetings,
such as travel and time away from lab and clinical responsibilities.
Overhauling peer review
Given the large number of factors involved, officials at NIH have decided to
undertake a much more comprehensive look at the peer review system. Two
working groups of the Advisory Committee to the NIH Director were
established to examine the peer review system, and those working groups
issued a request for information (RFI) in July 2007. The RFI asked for
“creative, concrete suggestions” in six major areas: challenges of the NIH
system of research support and the peer review process, solutions to those
challenges, core values of the peer review process that should be enhanced
or maintained, whether the peer review process uses appropriate criteria and
scoring procedures, and whether there is adequate support for investigators
at different stages in their career paths.
The APS response to the RFI was prepared by the Public Affairs Committee and
members of the APS Council. The response focused on key areas that were of
greatest concern, including: modifying the practice of streamlined
review—the APS recommends that grants receiving disparate reviews always be
discussed, every grant should get a score, and that applications from new or
first-time investigators be exempt from streamlining.
The absence of unmentored entry level grant opportunities for new
investigators – the APS recommends that NIH consider reviewing applications
from new investigators in separate study sections that will focus their
reviews on the quality of the PI’s training, research environment and
support, merit of the research questions being posed in the application,
feasibility of the experiments proposed and probability of success, past
productivity and potential for future productivity.
Shortening the review cycle – the APS commends the NIH for shortening the
review cycle for new investigators, and recommends that NIH work toward the
goal of reducing the review time for all grant applications.
The APS also recommends that any major reforms to the peer review process be
focused on clearly defined problems, and that implementation of new
initiatives include specific plans for evaluation and measurement of success
or failure. The Society’s response can be viewed in its entirety on the APS
Science Policy website (http://www.the-aps.org/pa).
In conjunction with the RFI, NIH officials are meeting with the scientific
community to gather input directly. Working group representatives met with
scientific societies on July 30, 2007 in Washington, DC. The APS was
represented at that meeting by John Chatham, a member of the Public Affairs
Committee, who was selected to give a statement on behalf of the Society.
APS will also have representatives at a series of regional meetings with NIH
officials in San Francisco, Chicago and New York.
The stated goal of the working groups is to gather input until the end of
2007, at which time pilot projects will be implemented and evaluated. To
keep the scientific community up to date and informed on peer review
initiatives, NIH has created this website:
http://enhancing-peer-review.nih.gov/. |
