About the Project: Promoting Effective Program Evaluation

In 1997, recognizing the importance of program evaluation, the Minority Opportunities in Research (MORE) division of the National Institutes of Health (NIH)/National Institute of General Medical Sciences (NIGMS), decided that resources and training for directors of MORE funded programs would enhance the quality of their program evaluation. With NIGMS support, the American Physiological Society (APS) began its four-year effort, Promoting Effective Program Evaluation (RU13 GM56019). 

Project Goal

Goal of this project was to promote the development and continuation of effective MORE programs by assisting program staff to develop and implement effective formative and summative evaluation plans.

Project Objectives

As a result of their participation in the program, program staffs for MORE-sponsored projects were expected to:

  • Significantly improve their understanding of the benefits of program evaluation;
  • Significantly improve their skills in developing both formative and summative evaluation plans;
  • Significantly improve their understanding of the pro’s and con’s of internal and external evaluations;
  • Develop an initial evaluation plan; and
  • Begin implementation of their evaluation plan.

Target Audience

The project sought to provide technical assistance to Minority Access to Research Career (MARC) and Minority Biomedical Research Support (MBRS) program staff in developing and implementing effective evaluation plans. In 2000, the project’s technical assistance was expanded to include Bridges program directors and their staff.

Project Activities

The project began with a collaborative needs assessment with input gathered from surveys of current MARC and MBRS program directors and via a brainstorming session with a subgroup of program directors, NIH-MORE staff, evaluation consultants, and APS staff. The results of the collaborative needs assessment, as well as the ongoing formative evaluation results from each project activity and from periodic meetings with NIH-MORE staff, were used to guide the overall project development, activities, and materials development.

Based on the needs assessment, the APS staff developed and implemented a number of 4-day Evaluation Exploration Short Courses. These courses were designed to develop participants’ understanding of evaluation basics and planning strategies and to provide time to begin development of an evaluation plan for individual programs. Sessions included plenary sessions with individual speakers and panels and interactive small group sessions where participants could apply the evaluation principles and strategies presented to their own programs. Speakers included program evaluation experts and MARC/MBRS project directors. Evaluation experts led the small group sessions. Participants completed entry and exit evaluation forms to provide data for the formative and summative evaluation of the project. A sample agenda and list of evaluation short courses can be viewed here as a PDF file.

The APS staff also presented numerous workshops at meetings of MORE principal investigators and at the annual minority research symposium sponsored by the MORE program. These workshops provided an opportunity to both promote the four-day short courses and to expand further on topics raised during the short courses (e.g., non-response to survey questionnaires and data handling and security).

Throughout the project, the APS staff provided additional resources and information to MORE project directors and staff via both print and online communications. The project developed and disseminated 6 issues of The PEN: Program Evaluation News, a print newsletter devoted to resources and information on both the project and on program evaluation, in general. Individual issues of The PEN can be viewed here. Issues of The PEN, along with other information on the project and on evaluation resources, were available at a project website. Resources from both the newsletter and the website were incorporated into the online short course and the CD short course (below).

In order to provide ongoing information and training on the development of program evaluation plans, the APS staff, in conjunction with MediaWorks, LLC, developed online and CD-ROM versions of an evaluation exploration short course that incorporates many of the features, materials, and resources of the four-day short courses. Both the CD and web course allow the user to create an evaluation planning document that includes the project’s goals, measurable objectives, activities, and description. They also include resources such as links to evaluation websites, guides for finding and interviewing an external evaluator, and budget planning spreadsheets. To enter the short course, click here. For ordering information on the CD, click here.

Finally, the program held a closing brainstorming session on future evaluation needs for MORE program staff. A number of suggestions were brought forth and NIH-MORE staff has established a continuing dialog with project directors and evaluation experts on the subject of evaluation and technical support.

Program Impacts

More than 350 MARC, MBRS, and Bridges directors and staff participated in the projects’ short courses and workshops. Short-term workshop impacts were evaluated by pre and post surveys of short course participants. Long-term impacts were assessed by success rates of workshop participants in obtaining funding for MORE proposals developed subsequent to the workshop. A summary of program impact evaluation results is available here as a PDF file.

For additional information on the project, see the home page or contact the APS Education Office (education@the-aps.org).

(Top of page)