Changing the USMLE

Step 1 assesses whether medical students understand, and are able to apply, important concepts of the sciences basic to the practice of medicine. The emphasis is on the principles and mechanisms underlying health, disease, and therapy. Step 2 assesses whether medical students or graduates can apply medical knowledge, skills and understanding of clinical science essential for the provision of medical care under supervision. The emphasis is on the principles of clinical sciences and basic patient-centered skills, including health promotion and disease prevention. This Step is given in two parts, which tests the examinees clinical knowledge (Step 2 CK) and clinical skills (Step 2 CS), the latter using standardized patients. Step 3 tests whether medical graduates can apply medical knowledge and understanding of biomedical science and clinical science essential for the unsupervised practice of medicine.

In addition to its primary mission, the certification of an individual’s knowledge and skills necessary for the practice of medicine, the USMLE also serves several secondary purposes. Many medical schools, for example, use student performance at the different Steps of the USMLE in promotion and graduation decisions and curriculum evaluations. Student performance on the USMLE also is used as a factor in the selection process by post-MD clinical training programs.

The current USMLE was designed about 20 years ago. Given the changes that have taken place in medical education and medical practice, is the current structure suitable for the 21st century? Basic science knowledge is tested largely in Step 1. Though some test items have clinical vignettes, these vignettes are often “window dressing” to assess the student’s knowledge of particular aspects of basic sciences that underlie the practice of medicine. Clinical knowledge and skills are tested in Step 2, with an ensuing separation between the basic sciences and the principles of medical practice. It also has been argued that Step 3 lacks relevance because few physicians currently enter practice after one or two years of post-graduate training, which some have suggested has led Step 3 to become “a license to moonlight.” Indeed, an underlying assumption of Step 3, that there is a body of knowledge (which the examination is designed to assess) that is essential for the unsupervised medical practice as a generalist, no longer seems to fit the professional activities of a substantial number of medical graduates.

A particular concern relating to the present USMLE structure was the observation that student performance in the basic sciences deteriorated between Steps 1 and 2 (Figure 2).

This deterioration could be ascertained because the National Board of Medical examiners (NBME) over the years have inserted Step 1 questions into the Step 2 exam. These questions are not used for grading purposes but instead, for evaluating how well the students retain their basic science knowledge. Apart from Biochemistry, where the retention historically has been low and the Behavioral Sciences and Pathology, where the retention generally has increased, the trend is a decrease in the retention of basic science knowledge. Though this could reflect “binge and purge” study habits, as has been argued by some, the seemingly systematic decrease in the performance in Microbiology, Pharmacology and Physiology should be cause for concern. Basic science departments need to examine how the teaching material has been selected and presented—and whether the curricular reforms, with the move toward more integrated curricula, have had unintended negative consequences, in particular with respect to the learning and retention of basic science information. For example, do students purchase and use the textbooks that historically defined the syllabus in the traditional curricula? If not, what has replaced the textbooks? Does the presentation of material in the classroom and in syllabi provide preclinical medical students basic science coverage of sufficient depth and integration so as to foster optimal retention? Is the material presented in a manner that is accessible for students later on? (Black & white copies of color PowerPoints should not be considered “accessible information.”) Another reason for the decreased retention may be that the material is not reinforced in the students’ clinical training because the clinical faculty is under increasing pressure to generate income and may not have sufficient time (interest or, perhaps, knowledge) to teach and reinforce basic science information, such as pathophysiological mechanisms, molecular basis of clinical features and therapeutic rationale.

Another set of concerns related to the content of the USMLE, is that it currently may not reflect the evolving ideas on evidence-based practice, the gathering and interpretation of information, the application of biostatistics and epidemiology, public health and cost-effective practice. Though not necessarily of concern to the basic science faculty in our roles as educators, this is of concern to all of us in our roles as consumers of health care.

So, though it is possible that the current USMLE remains the most effective and efficient method to meet the needs of all the stakeholders—ranging from the Federation of State Medical Boards (FSMB), representing the public (the consumers of health care), to medical educators and basic and clinical scientists—it seemed prudent to conduct a comprehensive review of the USMLE. This was not a simple undertaking, as may be deduced from Figure 3,which shows the organizational structure of the USMLE and the flowchart for the review process by the Committee to Evaluate the USMLE Program (CEUP), with representation from medical scientists and educators, educational deans, residents, students, state medical boards and the public. The appropriate metaphor for changing the USMLE is not “turning the battleship” but “maneuvering a battle fleet at high speed” (Figure 4)—where each interested party has its own set of priorities.

To guide the review, CEUP sought input from many different groups including: FSMB, the state medical boards and the public; the Association of American Medical Colleges (AAMC), in particular the Groups on Student Affairs (GSA) and Education Affairs (GEA) and the Organization of Student Representatives (OSR); the American Medical Association (AMA); residency program directors; and the American Medical Student Association (AMSA). Members of the basic and clinical science community, and the societies representing their interests, were consulted relatively late in the review.

Not surprisingly, these different groups had quite different perspectives on how the USMLE should be changed.

The state medical boards and the public felt strongly that the USMLE structure should recognize the need for licensure decision at two points (or gateways):

These gateways are licensure decision points, not examination events, meaning that the licensure decision in each
gateway could be based on multiple examinations. The examination components in these licensure decisions should: measure all competencies related to patient-centered care that can be tested in a valid, reliable manner; be able to assure at least minimum competency in these areas; and provide scores to the state medical boards that will assist them in making licensure decisions when the performance is marginal.

Figure 4: The British Grand Fleet’s and the German Hochseeflotte’s maneuvers during the Battle of Jutland. (From wikimedia.org/wikipedia/commons/4/41/
Jutland_fleet_action.png). The numbers 1-7 refer to the approximate fleet positions at various times between 6 and 9 pm.

Medical school curriculum and student affairs deans expressed concern that the separation of basic science and clinical science in Steps 1 and 2 CK was artificial and that it interferes with the curricular design and delivery—to the point that some education deans stated that the current USMLE structure made meaningful curricular reform impossible. There also was concern that the Step 1 score may disproportionately affect career choices and decisions.

Residency program directors stated that scores on USMLE examinations are essential as they are important for the screening of candidates for interview and the final ranking in the residency match. The structure and content of the USMLE was deemed to be less important than the timing.

Students preferred to “get basic science over with” and were generally not in favor of integrated exams (that test both basic and clinical science). Students in years 3 and 4, who were concerned about their post-graduate careers and the residency match, preferred scored exams over pass/fail grading. Students in years 1 and 2 were ambivalent about the types of grading that would be optimal.
Basic science educators stated that although basic science is the foundation of medicine, some important basic science concepts have no clinical “wrapping.” They also felt strongly that Step 1 reinforces the value of basic science in the medical school curriculum. Some in the basic science community also felt that Step 1 is necessary for promotion decisions—and that it is important to have meaningful, normed grades and a means to evaluate the success of courses and teaching efforts by comparing their students performance on Step 1 to that of students from all US medical schools.

CEUP Recommendations: The CEUP report and recommendations were released in May 2008: http://www.usmle.org/general_information/CEUP-Summary-Report-June2008.PDF. The recommendations have been endorsed by the Composite Committee that governs the USMLE program, and are scheduled for final governance review at the spring 2009 annual meetings of the NBME and the Federation of State Medical Boards.

At present, three recommendations have been approved, http://www.usmle.org/general_information/review.html. They are, in abbreviated form:

Concerns About the Proposed Changes: As noted above, members of the basic science community generally became involved in the discussions relatively late. It seems that, despite major efforts by the USMLE to inform the medical schools (primarily the education deans) about the likely changes, there was little discussion of the proposed changes within most US medical schools. The first group to express concern was the National Association of MD-PhD Programs, and that only happened because I, by chance, met Dr. Peter V. Scoles, Senior Vice President for Assessment Programs of the NBME, at the 2007 meeting of the Western Group of Student Affairs. At that time (May 2007), the structure of the two licensure decision points (gateway A and gateway B) had not been finalized; one proposal was that the current Steps 1 and 2 (CK and CS) exams be combined into one integrated exam, which would be taken in late year 3 or early year 4. Though such a structure indeed would serve to integrate the testing of the basic sciences in a clinical context, it also would have major (most likely negative) implications for the teaching of the basic sciences in medical schools—and for the structure of MD-PhD training programs.

The Association of MD-PhD Programs expressed its concerns about the possible changes in the fall of 2007, which led to a number of organizations representing the interests of the basic biomedical sciences—including the ACDP and the American Physiological Society—becoming involved. To discuss the concerns that had been expressed, the NBME convened a meeting in early January 2008 with a number of MD-PhD program directors, members of CEUP, and representatives from the Council of Academic Societies, the American Society for Clinical Investigation, ACDP, FSMB and other stakeholders.

In addition to stressing that the role of science in 21st century medicine is likely to increase (not decrease) and that future medical students (and physicians) would need to learn and master material that is barely being taught today, the group discussed a number of other issues:

As one participant summed up this part of the discussion: what distinguishes the physician from the physician assistant is the science!
A major topic at the meeting was the importance of the USMLE—in addition to its role as a medical licensure examination. Indeed, the USMLE fulfills two other important functions, albeit functions that are not always acknowledged:

First, the USMLE has become an important pedagogical tool because, whether students are taught in a traditional or an integrated curriculum, both the basic and clinical sciences are taught (and usually also tested) in modular courses or clerkships. Studying for the USMLE, in particular Step 1, thus becomes the first, perhaps the only, time medical students take control of their curriculum to integrate the knowledge they have acquired in the different modular courses, and fill the gaps, into a coherent/organismic understanding of how the human body functions. This taking control and integration is critical for the students’ ability to go beyond algorithmic decision-making to think critically/mechanistically about diagnostic and therapeutic problems that patients present to their physicians. Though seemingly self-evident, this was apparently the first time this crucial point was made in relation to the on-going evaluation of the USMLE. (It was reiterated in a student-run survey conducted by the American Physician Scientists Association. The students in the Tri-Institutional MD-PhD Program requested that the underlined text be added.)

Second, just as the MCAT serves as the “great equalizer,” enabling strong students from less well-known colleges to be evaluated based on their merits when they apply to medical school, the USMLE Step I serves a similar role in the applications of MD students for post-graduate training.

In addition, though it has been stated that USMLE Step 1 prevents curricular reform, the evidence for this statement could not be identified. Indeed, the idea seemed to be based on the opinions of education deans from certain medical schools. Some participants asked: if a curricular reform causes students to do less well on USMLE Step I, does that reflect poorly on USMLE Step I or on the nature and implementation of the curricular reform?

Subsequently, the NBME established a task force to undertake a Comprehensive Review of the USMLE (CRU). The task force included representation from basic science and clinical scientists/educators and educational deans. The task force considered a number of models for a new USMLE, as well as the pedagogical and practical challenges associated with the different models. A key point in these discussions was that medical schools should prepare to teach basic science material in years 3 and 4, and to teach some clinical material in years 1 and 2.

What’s Next: Based on the CEUP recommendations, it is reasonable to assume that there will be changes to the USMLE program. Although the guiding principles for the revision of the examination process have been established, the design process is just beginning. To the extent possible, the USMLE design will map to those competencies (as defined by the ACGME, see Table 1) that can be measured in a valid and reliable manner. The current blueprints correspond to the competencies in knowledge, patient care, and communication and interpersonal skills. Systems-based practice is more difficult to define, and the competencies that usually lumped under “professionalism” may best evaluated by other measures.

The changes will be incremental and evolutionary, and unlikely to prompt sudden or radical shifts in the basic science curriculum design or delivery. There will be two gateways, but a gateway is not an exam; each gateway may be composed of several (maybe two to four) “testing events” or “exam components.” State medical boards may see Step scores aggregated into two clusters corresponding to the two decision points for licensure. Pass/fail scoring is not under consideration outside the Clinical Skills examination, and individual Step scores will continue to be reported to students and schools in the current manner.

In each gateway (and maybe each testing event) there will be substantial testing/integration of basic science, clinical science (clinical knowledge) and clinical skills. Step 1 will continue to focus on the “scientific foundations” of medicine; it also will test the students’ qualitative and quantitative reasoning ability and ability to use literature sources, with greater integration of abnormal structure and function and translational science. The clinical vignettes that inform many Step 1 questions will continue to improve in clarity and relevance, and factoid questions will disappear.

Students taking Steps 2 and 3 will soon notice that increased numbers of clinical test questions will draw on scientific materials and reasoning processes that were emphasized in the pre-clinical curriculum. To an increasing extent candidates taking Steps 2 and 3 will be tested in their ability to integrate fundamental science with medical knowledge—with increased emphasis on biostatistics, epidemiology, qualitative and quantitative reasoning ability and use of the literature plus, of course, their clinical skills. Competency in medical knowledge, clinical reasoning and judgment and the ability to integrate the advances in translational science into clinical practice is likely to become increasingly important in Step 3.

As soon as practical, test materials that require interpretation and evaluation of evidence will begin to appear in all three Steps of the examination. Later, if it proves possible, test formats that require the appropriate use of on-line data base searches to make clinical judgments will be included in the examination. Simulation testing will likely be increased, though this will be gradual. Heart sound simulations have already been embedded in the examinations. More sophisticated simulations will take time to develop and deploy. Because some of these formats will be difficult to incorporate into the current structure of the exams, changes in the layouts of the testing days may be necessary.

At some time within the next five to seven years, these incremental changes—the new test item and scoring formats and the more rigorous assessment of fundamental science in the later phases of the examination—may make the current USMLE Step framework obsolete and thus require changes in the USMLE terminology. Eventually, it may no longer be possible to combine the exam components from the present and the revised USMLE; most likely, however, such combinations will be permitted for some reasonable time, as has been the case in past revisions of the USMLE Challenges and Opportunities: Though most basic science scientists and educators are likely to claim that “we do not teach for the USMLE,” any medical school’s curriculum needs to prepare students to pass the USMLE, and the proposed changes in the USMLE are likely to impact on the teaching responsibilities of basic science departments. Though the increasing integration of the basic and clinical sciences could lead to a further erosion of the role of the basic sciences in the medical curriculum, the increasing emphasis on fundamental science throughout the USMLE, as well as the emphasis on qualitative and quantitative reasoning, is likely to increase the importance of the basic sciences in medical education. Thus, basic scientists—in particular physiologists—are likely to have an even greater role in medical school curriculum, going beyond the current “preclinical years.”

A key concern for basic scientists and educators will be who will define what constitutes the basic science that is relevant, whether directly or indirectly, for the practice of medicine—and who will ensure that this science is taught well and tested in a satisfactory manner. As noted above, part of the reason for the declining performance on the basic science questions between Steps 1 and 2 (Figure 2) may be due to a decreased emphasis on teaching the fundamental mechanisms in the clinical clerkships, a situation that is unlikely to change given the increasing demands on the clinical faculty. But, importantly, there will be increased emphasis on the fundamental principles underlying the practice of medicine in Step 2! Thus, basic science departments will need to consider how to become involved in the teaching in years 3 and 4. This represents both an opportunity and a challenge; the latter because teaching in the clerkships usually is done in small-group sessions, with the same material being taught as often as 12 times/year! Basic science departments probably also need to consider how additional pathophysiology and translational science can be incorporated into what is traditionally considered the first year curriculum—in a manner that strengthens the basic science teaching. Finally, no matter the intent of any proposed changes to the USMLE, the devil is in the details. In this case, what is key is to ensure that the different test elements adequately probe the students’ command of the basic sciences that are relevant for the practice of medicine. This responsibility cannot be delegated: The basic science faculty needs to be actively involved in the design of the new test elements.
Acknowledgments: I wish to thank Drs. Judith A. Bond (Pennsylvania State University), Lawrence F. Brass (University of Pennsylvania MD-PhD Program), David M. Engman (Northwestern University Medical Scientist Training Program), Michael J. Friedlander (Baylor College of Medicine), Paul A. Insel (UCSD Medical Scientist Training Program) and in particular Peter V. Scoles (NBME) for many helpful discussions and comments on the manuscript. I also wish to thank the students in the Tri-Institutional MD-PhD Program for our many discussions about medical education and the USMLE and their comments on this manuscript, and F.T. Nguyen and E. Schauberger (American Physician Scientists Association) for their efforts to survey medical student attitudes toward the USMLE.

The author may be contacted at:
Olaf S. Andersen
Weill Cornell/Rockefeller/Sloan-Kettering
Tri-Institutional MD-PhD Program and
Department of Physiology and Biophysics
Weill Cornell Medical College
New York, NY 10065, USA
sparre@med.cornell.edu