A Mixed Methods Evaluation of the Reliability and Validity of an Entrustable Professional Activities-Based System for Assessing the Clinical Competence of Medical Students

Abstract

Introduction: The question of how to assess the clinical competence of medical students has posed a challenge throughout the history of the field. Entrustable Professional Activities (EPAs) describe discrete activities that can be assessed in the workplace by a preceptor, who indicates the degree to which they “entrust” the student to perform the activity. Recently, the University of Minnesota Medical School (UMMS) implemented a system based on the Core Entrustable Professional Activities for Entering Residency (Core-EPAs) to assess the clinical skills of third-year medical students during their clinical clerkships. The purpose of this study was to evaluate the validity and reliability of this system, using quantitative data, qualitative data, and the integration of the two. Method: This study employed a two-phase, sequential explanatory mixed methods design to obtain evidence regarding the reliability, predictive validity, construct validity, and face validity of the Core-EPA system. Quantitative EPA assessment data from AY 22-23 were analyzed via interrater agreement analysis, linear regression modeling to predict scores on an Objective Structured Clinical Examination (OSCE), and growth curve modeling. A purposive sample of eight students was selected from the quantitative data to describe their experiences via semi-structured interviews. The qualitative data were analyzed using a Reflexive Thematic Analysis framework. Results: The interrater reliability analysis found that levels of interrater agreement were acceptable given the complexity of the clinical context (47.35% or 69.40% depending on the definition of interrater agreement), but with room for improvement. The linear regression analysis did not find convincing evidence that EPA ratings are predictive of OSCE scores. The growth curve analysis found that growth curves aligned with those expected by learning curve theory. The qualitative analysis generated five themes and two subthemes describing students’ experiences with the Core-EPA system, their perspectives on its validity and reliability including factors that contribute to those dimensions, and comparisons between the Core-EPA system and OSCEs. Discussion: The findings of this study indicate that the Core-EPA system holds promise as a tool for assessing the clinical competence of medical students. Recommendations, limitations of the study, and ideas for next steps are described.