APS Comments on RFI on Recommendations for Improving NRSA Fellowship Review

The following statements were submitted in response to the NIH Request for Information on proposed changes to the peer review of Ruth L. Kirschstein National Research Service Award (NRSA) fellowship applications (NOT-OD-23-110).

Please provide your comments on the proposed changes to NRSA fellowship review criteria.

The American Physiological Society (APS) supports the efforts of the Center for Scientific Review (CSR) to restructure the review of NRSA fellowships to address concerns about bias and fairness and to refocus the review on the qualities of the candidate that indicate scientific potential. In particular, APS appreciates the evidence-driven process through which the Working Group generated the recommendations that the proposed changes are based on. The proposed changes to the review criteria are likely to remove some sources of implicit bias which disadvantage otherwise highly qualified candidates. In addition, the three proposed scored criteria have the potential to improve the clarity and focus of the review, if steps are taken to ensure that both reviewers and applicants fully understand how applications will be evaluated using the new criteria. While APS supports the proposed changes, further steps may be needed to fully achieve CSR’s goals.

The proposal to remove scored consideration of an applicant’s sponsor and their institution will minimize one factor contributing to bias in the review process. However, it is not clear how reviewers will account for significant differences in the volume of preliminary data resulting from the sponsor’s seniority. A more established lab will be able to provide more preliminary data, allowing the applicant to cite a more extensive publication record. The data reviewed by the Working Group indicate that sponsor seniority is a significant determinant of an application’s impact score. Specifying how the review process will address differences in the volume of preliminary data and publications will clarify the process for both applicants and reviewers. One possible strategy to address this disparity would be implementation of recommendation 5 from the Working Group’s report, which suggests clustering applications from early career sponsors or those from less-resourced institutions during the discussion stage of the review process. The Working Group also recommended withholding institutional identification until the end of the application review. If these strategies are not implemented as recommended it would be helpful for the community to hear how CSR plans to address the issues they were intended to remedy.

The success of the proposed update to NRSA fellowship review depends on applicants and reviewers having a clear and aligned understanding of which qualities are most important for an application to be considered high impact. To achieve this, CSR should implement Recommendation 3 from the Working Group report and provide additional guidance for both applicants and reviewers. For reviewers, this should include peer review bias awareness training, as well as possibly guidelines or rubrics for the evaluation of applications using the updated criteria. For applicants, CSR can work with program officials at NIH Institutes and Centers to create guides for writing a competitive fellowship application.

Finally, CSR should take additional steps to support the diversity of high-quality candidates who apply for NRSA fellowships. This should include acknowledging that many applicants have career goals that fall outside of the academic research sector, such as careers in government or industry where scientific skills and knowledge provide significant value. CSR should ensure that review panels represent researchers from diverse backgrounds, at various career stages, and from a diversity of institution types. The applicant’s training plan and training resources can differ significantly based on these and other factors, and applications should not receive lower scores simply for having different training goals.

Please provide your comments on the proposed changes to the NRSA application instructions and materials.

The American Physiological Society (APS) supports many of the proposed changes to the application instructions, particularly as they align the application to the updated review criteria and improve the clarity and accessibility of the application process. The current application is lengthy, which can present a barrier for applicants who are already conducting full-time research, so APS supports a reduction in the overall length of the application. Further, APS agrees with revising the letters of support, but cautions that it may be difficult for some candidates to find three appropriate references.

While the five Applicant Statements help focus the application on the proposed review criteria and reduce the overall length of the Fellowship Applicant section, there may be redundancy among the questions resulting in a longer application than necessary. It is unclear how the Fellowship Qualifications and Self-assessment statements would provide different information about the candidate’s scientific potential and preparedness. A Self-assessment statement can also introduce unwanted bias, as applicants who are from historically underrepresented groups may face additional challenges writing a statement that resonates with their review panel. Since the scientific potential and preparedness of an applicant can likely be evaluated based on the Fellowship Qualifications and Scientific Perspective statements, the Self-assessment statement is redundant and should be removed.

The proposed update to the Sponsor and Co-sponsor Section will bring this section of the application into alignment with the proposed review criteria. However, APS recommends adjusting the third statement, “Applicant’s Qualifications and Potential for a Research Career,” to clarify that not all candidates are seeking a traditional academic research career. An NRSA fellowship can support training for many science-related careers either outside academia or in roles not directly involved in conducting research.

Providing prompts for reference letters can be helpful to guide referees, particularly for those who are early-career researchers who have not written reference letters before. APS supports the inclusion of these guidelines. However, applicants may struggle to find three appropriate references for a variety of reasons. For example, applicants who have had research experience outside the U.S. often encounter difficulty getting a letter from a reference outside the country, and applicants from smaller institutions may have had fewer opportunities to work with different advisors. CSR should consider reducing the number of letters required to two. Additionally, the third prompt, “What scientific and intellectual contributions has the applicant made during their training/research experience with you?” should be broadened, as it implicitly limits referees to current or former academic mentors of the applicant. Applicants who are graduate students or who have just started a postdoc may not have more than one referee with significant research mentoring experience.

Overall, APS strongly agrees with most of the proposed changes to the NRSA fellowship application and review criteria and appreciates CSR’s focus on reducing bias and addressing concerns about the current application review process. APS thanks CSR for the opportunity to comment on the proposed update.