Background
Medical educators need valid, reliable, and efficient tools to assess evidence-based medicine (EBM) knowledge and skills. Available EBM assessment tools either do not assess skills or are ...laborious to grade.
Objective
To validate a multiple-choice–based EBM test—the Resident EBM Skills Evaluation Tool (RESET).
Design
Cross-sectional study.
Participants
A total of 304 medicine residents from five training programs and 33 EBM experts comprised the validation cohort.
Main Measures
Internal reliability, item difficulty, and item discrimination were assessed. Construct validity was assessed by comparing mean total scores of trainees to experts. Experts were also asked to rate importance of each test item to assess content validity.
Key Results
Experts had higher total scores than trainees (35.6 vs. 29.4,
P
< 0.001) and also scored significantly higher than residents on 11/18 items. Cronbach’s alpha was 0.6 (acceptable), and no items had a low item-total correlation. Item difficulty ranged from 7 to 86%. All items were deemed “important” by > 50% of experts.
Conclusions
The proposed EBM assessment tool is a reliable and valid instrument to assess competence in EBM. It is easy to administer and grade and could be used to guide and assess interventions in EBM education.
Comprehensive medication reviews (CMRs) are provided by providers such as pharmacists to eligible beneficiaries. Although CMRs have been shown to provide value to patients, little is known about the ...service uniformity, quality, and content of CMRs.
This study aimed to characterize the current state of CMR services from diverse stakeholder perspectives and describe variation in responses to content and delivery of telephonic CMR services.
Semistructured interviews were conducted with 10 key informants. The interview guide contained 6 key questions with additional probing questions. Transcripts were analyzed using the inductive saturation model and phenomenological approach to code emergent themes, which were iteratively refined until saturation was achieved.
Key informants included CMR payers (n = 3), providers (n = 5), and standards-setting organizations (n = 2). Ten themes about CMRs emerged from qualitative analysis: (1) definition, (2) organizational goals, (3) content, (4) eligibility, (5) frequency, (6) acceptance and completion, (7) process and personnel, (8) quality assurance, (9) preparation, and (10) future directions. CMR content descriptions were consistent across perspectives. Key informants described scenarios appropriate for expanded CMR eligibility criteria, although none were consistently reported. Providers emphasized patient CMR acceptance rates whereas payers and standard-setting organizations emphasized completion rates. Completion rates and adherence to Centers for Medicare and Medicaid Services standards were characterized as core organizational goals (n = 8), whereas patient satisfaction was less frequently identified (n = 4). A lack of incentive for CMR providers to follow-up with patients was a barrier to expanded services. Overall, key informants were dissatisfied with the CMR completion rate measure and would prefer measures focused on service quality and outcomes.
CMR services largely met perceived guidelines, with variation in value-added services. Key informants desired adoption of an actionable measure that is focused on quality rather than completion rate. To inform a quality measure, future research should analyze completed CMRs to determine the extent of variation in content and delivery.
PDF