The Step 2 Clinical Knowledge (CK) examination of the United States Medical Licensing Examination sequence is a requirement for the certification of international medical graduates (IMGs) by the ...Educational Commission for Foreign Medical Graduates. An association between scores on the test and the quality of care later provided by those who take it is central to its use in certification and licensure. The purpose of this study was to determine whether there is a relationship between scores on Step 2 CK and patient outcomes for IMGs.
This is a retrospective observational study of the 60,958 hospitalizations from 2003 to 2009 in Pennsylvania where the principal diagnosis was acute myocardial infarction or congestive heart failure and the attending physician (N = 2,525) was an IMG who had taken the Step 2 CK. The main measures were the three-digit scores on Step 2 CK and in-hospital mortality.
After adjustment for severity of illness, physician characteristics, and hospital characteristics, performance on Step 2 CK had a statistically significant inverse relationship with mortality. Each additional point on the examination was associated with a 0.2% (95% CI: 0.1%-0.4%) decrease in mortality. The size of the effect is noteworthy, with each standard deviation (roughly 20 points) equivalent to a 4% change in mortality risk.
These findings provide evidence for the validity of Step 2 CK scores. Given the magnitude of its relationship with patient outcomes, the results support the use of the examination as an effective screening strategy for licensure.
Context
Competency‐based medical education (CBME) has emerged as a core strategy to educate and assess the next generation of physicians. Advantages of CBME include: a focus on outcomes and learner ...achievement; requirements for multifaceted assessment that embraces formative and summative approaches; support of a flexible, time‐independent trajectory through the curriculum; and increased accountability to stakeholders with a shared set of expectations and a common language for education, assessment and regulation.
Objectives
Despite the advantages of CBME, numerous concerns and challenges to the implementation of CBME frameworks have been described, including: increased administrative requirements; the need for faculty development; the lack of models for flexible curricula, and inconsistencies in terms and definitions. Additionally, there are concerns about reductionist approaches to assessment in CBME, lack of good assessments for some competencies, and whether CBME frameworks include domains of current importance. This study will outline these issues and discuss the responses of the medical education community.
Methods
The concerns and challenges expressed are primarily categorised as: (i) those related to practical, administrative and logistical challenges in implementing CBME frameworks, and (ii) those with more conceptual or theoretical bases. The responses of the education community to these issues are then summarised.
Conclusions
The education community has begun to address the challenges involved in implementing CBME. Models and guidance exist to inform implementation strategies across the continuum of education, and focus on the more efficient use of resources and technology, and the use of milestones and entrustable professional activities‐based frameworks.
Inconsistencies in CBME definitions and frameworks remain a significant obstacle. Evolution in assessment approaches from in vitro task‐based methods to in vivo integrated approaches is responsive to many of the theoretical and conceptual concerns about CBME, but much work remains to be done to bring rigour and quality to work‐based assessment.
Discuss ideas arising from the article at http://www.mededuc.com discuss.
Patients can be treated by a physician, a nurse practitioner (NP), or a physician assistant (PA) despite marked differences in the education and training for these three professions. This natural ...experiment allows examination of a critical question: What is the minimum education and training required to practice primary care? In other words, how tall is the shortest giant? State licensing requirements, not educational bodies, legislate minimum training. The current minimum is 6 years, which includes 27.5 weeks of supervised clinical experience (SCE), for NPs. In comparison, PAs train for 6 years with 45 weeks of SCE, and physicians for at least 8 years with 110 weeks of SCE. Initial, flawed studies show equivalent patient outcomes among the professions. If rigorous follow-up studies confirm equivalence, the content and length of medical education for primary care physicians should be reconsidered. Unmatched medical school graduates, with 7 years of training and 65 weeks of SCE, more than the required minimum for NPs, deserve to practice independently. So do PAs. If equivalence is not confirmed, the minimum requirements for NPs and/or PAs should be raised, including considering a required residency (currently optional). Alternatively, the scope of practice for the three professions could be defined to reflect differences in training. There is an urgent need to set aside preconceived notions and turf battles, conduct rigorous independent studies, and generate meaningful data on practice patterns and patient outcomes. This should inform optimal training, scope of practice, and workforce development for each invaluable primary care clinical practitioner.
Today, medical schools graduate doctors, not physicians. Thousands of doctors who are U.S. citizens and graduates of U.S. and international medical schools will never become physicians because they ...do not obtain a residency position. Doctors need at least one year of residency to become a licensed physician. However, 4,099 applicants in 2018 and 4,170 in 2019 failed to get a position through the National Resident Matching Program Main Match; about 1,000 students get positions after the Main Match each year. The personal and societal cost is enormous: each year, approximately 3,000 nonphysician doctors cannot use 12,000 education years and three-quarters of a billion dollars they invested in medical education and cannot mitigate the shortfall of 112,000 physicians expected in 2030.To ameliorate this problem, medical schools could guarantee one year of residency. This is affordable: despite federally funded slots being capped, residency positions have increased for 17 consecutive years (20,602 in 2002 to 32,194 in 2019) because residents are cost-effective additions to the workforce. Alternatively, a 3-year curriculum plus required fourth-year primary care residency is another option. The salary during the residency year could equal other first-year residents', or there could be a token amount for this "internship." Both models decrease the cost of medical education; the second financially unburdens the hospital.Since the Flexner Report (when there was no formal postgraduate training), the end point of medical education has moved from readiness for independent medical practice (physician) to readiness for postgraduate training (doctor). To benefit individuals and society, medical education must take steps to ensure that all graduates are physicians, not just doctors.
Karolinska Prize winner John Norcini offers an argument for how micro‐credentials offer an educational model that can support the development of competence across professions and careers.
Educational assessment for the health professions has seen a major attempt to introduce competency based frameworks. As high level policy developments, the changes were intended to improve outcomes ...by supporting learning and skills development. However, we argue that previous experiences with major innovations in assessment offer an important road map for developing and refining assessment innovations, including careful piloting and analyses of their measurement qualities and impacts. Based on the literature, numerous assessment workshops, personal interactions with potential users, and our 40 years of experience in implementing assessment change, we lament the lack of a coordinated approach to clarify and improve measurement qualities and functionality of competency based assessment (CBA). To address this worrisome situation, we offer two roadmaps to guide CBA’s further development. Initially, reframe and address CBA as a measurement development opportunity. Secondly, using a roadmap adapted from the management literature on sustainable innovation, the medical assessment community needs to initiate an integrated plan to implement CBA as a sustainable innovation within existing educational programs and self-regulatory enterprises. Further examples of down-stream opportunities to refocus CBA at the implementation level within faculties and within the regulatory framework of the profession are offered. In closing, we challenge the broader assessment community in medicine to step forward and own the challenge and opportunities to reframe CBA as an innovation to improve the quality of the clinical educational experience. The goal is to optimize assessment in health education and ultimately improve the public’s health.
Objective This instalment in the series on professional assessment provides an introduction to methods of setting standards.
Method A standard is a special score that serves as a boundary between ...those who perform well enough and those who do not. The practical steps in selecting it include: deciding on the type of standard; deciding the method for setting it; selecting the judges; holding the meeting; calculating the cutpoint, and deciding what to do afterwards. Four of the more popular methods are illustrated for both written and clinical examinations.
Results The most important criteria for selecting a method for setting standards are whether it is consistent with the purpose of the test, based on expert judgement, informed by data, supported by research, transparent, and requires due diligence. The credibility of the standard will rely largely on the nature of the standard setters and the selection of a broadly representative and knowledgeable group is essential. After the standard has been set, it is important to ensure that stakeholders view the results as credible and that the pass rates have sensible relationships with other markers of competence.
Conclusions A standard is an expression of professional values in the context of a test's purpose and content, the ability of the examinees, and the wider social or educational setting. Because standards are an expression of values, methods for setting them are systematic ways of gathering value judgements, reaching consensus and expressing that consensus as a single score on a test.
PDF
