RESUMEN

CONTEXT: The National Board of Osteopathic Medical Examiners (NBOME) administers the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA), a three-level examination designed for licensure for the practice of osteopathic medicine. The examination design for COMLEX-USA Level 3 (L3) was changed in September 2018 to a two-day computer-based examination with two components: a multiple-choice question (MCQ) component with single best answer and a clinical decision-making (CDM) case component with extended multiple-choice (EMC) and short answer (SA) questions. Continued validation of the L3 examination, especially with the new design, is essential for the appropriate interpretation and use of the test scores. OBJECTIVES: The purpose of this study is to gather evidence to support the validity of the L3 examination scores under the new design utilizing sources of evidence based on Kane's validity framework. METHODS: Kane's validity framework contains four components of evidence to support the validity argument: Scoring, Generalization, Extrapolation, and Implication/Decision. In this study, we gathered data from various sources and conducted analyses to provide evidence that the L3 examination is validly measuring what it is supposed to measure. These include reviewing content coverage of the L3 examination, documenting scoring and reporting processes, estimating the reliability and decision accuracy/consistency of the scores, quantifying associations between the scores from the MCQ and CDM components and between scores from different competency domains of the L3 examination, exploring the relationships between L3 scores and scores from a performance-based assessment that measures related constructs, performing subgroup comparisons, and describing and justifying the criterion-referenced standard setting process. The analysis data contains first-attempt test scores for 8,366 candidates who took the L3 examination between September 2018 and December 2019. The performance-based assessment utilized as a criterion measure in this study is COMLEX-USA Level 2 Performance Evaluation (L2-PE). RESULTS: All assessment forms were built through the automated test assembly (ATA) procedure to maximize parallelism in terms of content coverage and statistical properties across the forms. Scoring and reporting follows industry-standard quality-control procedures. The inter-rater reliability of SA rating, decision accuracy, and decision consistency for pass/fail classifications are all very high. There is a statistically significant positive association between the MCQ and the CDM components of the L3 examination. The patterns of associations, both within the L3 subscores and with L2-PE domain scores, fit with what is being measured. The subgroup comparisons by gender, race, and first language showed expected small differences in mean scores between the subgroups within each category and yielded findings that are consistent with those described in the literature. The L3 pass/fail standard was established through implementation of a defensible criterion-referenced procedure. CONCLUSIONS: This study provides some additional validity evidence for the L3 examination based on Kane's validity framework. The validity of any measurement must be established through ongoing evaluation of the related evidence. The NBOME will continue to collect evidence to support validity arguments for the COMLEX-USA examination series.

Asunto(s)

RESUMEN

Background Many residency programs do not accept the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) alone for osteopathic applicants. Furthermore, among those programs that do accept the COMLEX-USA, it is unknown how programs scale their minimum COMLEX-USA scores compared to their minimum United States Medical Licensing Examination (USMLE) scores. Objective Our objective was to examine the variation of relative within-program differences between minimum USMLE Step and COMLEX-USA Level scores required for consideration by United States residency programs. Methods We performed a cross-sectional analysis of the Fellowship and Residency Electronic Interactive Database Access (FREIDA) database from April 2023, including the 10 specialties with the most training spots in 2022. These specialties were internal medicine, family medicine, pediatrics, emergency medicine, psychiatry, surgery, anesthesiology, obstetrics-gynecology, orthopedic surgery, and neurology. Within-program differences were calculated by subtracting the minimum USMLE Step 1 and 2 scores from the converted minimum USMLE Step 1 and 2 scores calculated from the minimum COMLEX-USA Level 1 and 2 scores using two conversion tools. We present differences as medians with interquartile ranges (IQR). Additionally, we report the proportion of programs with greater than 10-point differences for each step (1 and 2). Results Of the 3,364 accredited programs from the examined specialties, we included 1,477 in the Step 1 analysis and 1,227 in the Step 2 analysis with complete data. The median within-program difference between the minimum Step 1 score and the predicted Step 1 score was 12.0 (IQR 2.0, 17.0) using the Barnum and colleagues' conversion tool and -1.7 (IQR -6.2, 6.3) using the Smith and colleagues' tool. The median differences for Step 2 were 2.0 (IQR -8.0, 12.0) and -6.5 (IQR -13.9, -1.5) for each tool, respectively. Using the Barnum and Smith conversion tools, 937 (63%) and 435 (29%) programs had a greater than 10-point Step 1 score difference, respectively. Similarly, for Step 2, 564 (46%) and 515 (42%) programs had a greater than 10-point difference with each conversion tool. Conclusion There is wide variation in the within-program differences between minimum USMLE and predicted minimum USMLE (from COMLEX-USA) scores. Many programs have greater than 10-point differences, which may be a source of bias in osteopathic applicant selection.

RESUMEN

The Coalition for Physician Accountability's Undergraduate Medical Education-Graduate Medical Education (UME-GME) Review Committee (UGRC): Recommendations for Comprehensive Improvement of the UME-GME Transition final report includes a total of 34 recommendations and outlines opportunities to transform the current processes of learner transition from a US-based MD- or DO-granting medical school or international medical education pathway into residency training in the United States. This review provides a reflection on the recommendations from the authors, all members of the UGRC, describing the pros and cons and the opportunities and limitations, in the hopes that they might inspire readers to dig deeper into the report and contribute to meaningful improvements to the current transition. The UGRC Recommendations highlight the many opportunities for improvement in the UME-to-GME transition. They are built on the connection to the system of education and formation of physicians to a more just healthcare system, with attention to diversity, equity, and inclusion to improve health disparities and to the quality of care that patients receive. However, there are justifiable concerns about changes that are not fully understood or that could potentially lead to unintentional consequences. This analysis, reached through author consensus, considers the pros and cons in the potential application of the UGRC Recommendations to improve the UME-to-GME transition. Further debate and discussion are warranted, without undue delay, all with the intention to continue to improve the education of tomorrow's physicians and the care for the patients who we have the privilege to serve.

Asunto(s)

RESUMEN

Background To improve their standing in residency selection, many osteopathic medical students choose to take the United States Medical Licensing Examination (USMLE). Although scores on USMLE Step 1 and Level 1 of the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) are known to be highly correlated, scarce data exist on the association between COMLEX-USA Level 2-Cognitive Evaluation (CE) and USMLE Step 2 Clinical Knowledge (CK) scores. In this study, we aimed to determine the association between COMLEX-USA Level 2-CE and USMLE Step 2 CK scores and derive an equation to predict performance on USMLE Step 2 CK for applicants who have only taken COMLEX-USA. Methodology We reviewed COMLEX-USA Level 2-CE and USMLE Step 2 CK scores for all students at the Lake Erie College of Osteopathic Medicine from May 2020 to April 2021. Linear regression was used to evaluate the relationship between COMLEX-USA Level 2-CE and USMLE Step 2 CK scores. Results A total of 340 students took both COMLEX-USA Level 2-CE and USMLE Step 2 CK. There was a linear association between COMLEX-USA Level 2-CE and USMLE Step 2 CK scores such that every one-point increase in COMLEX-USA was associated with a 0.13-point increase in USMLE Step 2 CK score (standard error = 9.1; model R2 = 0.64). Conclusions Students or programs interested in predicting performance on USMLE Step 2 CK from performance on COMLEX-USA Level 2-CE can do so using the following equation: USMLE Step 2 CK = 0.13(COMLEX-USA Level 2-CE) + 163.5.

RESUMEN

CONTEXT: Many medical schools have a distributed model for clinical clerkship education, challenging our ability to determine student gaps during clinical education. With the graduating class of 2017, A.T. Still University's School of Osteopathic Medicine in Arizona (ATSU-SOMA) began requiring additional online curricula for all clerkship courses. OBJECTIVES: To determine whether third year and fourth year students receiving ATSU-SOMA's online curricula during core clerkships performed better overall on national standardized examinations than students from previous years who had not received the curricula, and whether scores from online coursework correlated with outcomes on standardized examinations as possible early predictors of success. METHODS: This retrospective cohort study analyzed existing data (demographics and assessments) from ATSU-SOMA classes of 2017-2020 (curriculum group) and 2014-2016 (precurriculum group). The effect of the curriculum on national standardized examinations (Comprehensive Osteopathic Medical Achievement Test [COMAT] and Comprehensive Osteopathic Medical Licensing Examination of the United States [COMLEX-USA]) was estimated using augmented inverse probability weighting (AIPW). Correlations between assignment scores and national standardized examinations were estimated using linear regression models. RESULTS: The curriculum group had 405 students with a mean (standard deviation [SD]) age of 25.7 (±3.1) years. Two hundred and fifteen (53.1%) students in the curriculum group were female and 190 (46.9%) were male. The precurriculum group had 308 students (mean ± SD age, 26.4 ± 4.2 years; 157 [51.0%] male; 151 [49.0%] female). The online curriculum group had higher COMAT clinical subject exam scores in obstetrics and gynecology, osteopathic principles and practice (OPP), psychiatry, and surgery (all p≤0.04), as well as higher COMLEX-USA Level 2-Cognitive Evaluation (CE) family medicine and OPP subscores (both p≤0.03). The curriculum group had a 9.4 point increase in mean total COMLEX-USA Level 2-CE score (p=0.08). No effect was found for the curriculum overall on COMAT mean or COMLEX-USA Level 2-Performance Evaluation scores (all p≥0.11). Total coursework scores in each core clerkship, excluding pediatrics, were correlated with COMAT mean score (all adjusted p≤0.03). Mean scores for five of the seven assignment types in core clerkships, excluding evidence based medicine types, were positively correlated with COMAT mean scores (all adjusted p≤0.049). All assignment types correlated with COMLEX-USA Level 2-CE total score (all adjusted p≤0.04), except interprofessional education (IPE). CONCLUSIONS: Results from this study of 713 students from ATSU-SOMA suggested that our online curriculum supplemented clinic based learning during clerkship courses and improved student outcomes on national standardized examinations.

Asunto(s)

RESUMEN

CONTEXT: The Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) is a three level examination used as a pathway to licensure for students in osteopathic medical education programs. COMLEX-USA Level 2 includes a written assessment of Fundamental Clinical Sciences for Osteopathic Medical Practice (Level 2-Cognitive Evaluation [L2-CE]) delivered in a computer based format and separate performance evaluation (Level 2-Performance Evaluation [L2-PE]) administered through live encounters with standardized patients. L2-PE was designed to augment L2-CE. It is expected that the two examinations measure related yet distinct constructs. OBJECTIVES: To explore the concurrent validity of L2-CE with L2-PE. METHODS: First attempt test scores were obtained from the National Board of Osteopathic Medical Examiners database for 6,639 candidates who took L2-CE between June 2019 and May 2020 and matched to the students' L2-PE scores. The sample represented all colleges of osteopathic medicine and 97.5% of candidates who took L2-CE during the complete 2019-2020 test cycle. We calculated disattenuated correlations between the total score for L2-CE, the L2-CE scores for the seven competency domains (CD1 through CD7), and the L2-PE scores for the Humanistic Domain (HM) and Biomedical/Biomechanical Domain (BM). All scores were on continuous scales. RESULTS: Pearson correlations ranged from 0.10 to 0.88 and were all statically significant (p<0.01). L2-CE total score was most strongly correlated with CD2 (0.88) and CD3 (0.85). Pearson correlations between the L2-CE competency domain subscores ranged from 0.17 to 0.70, and correlations which included either HM or BM ranged from 0.10 to 0.34 with the strongest of those correlations being between BM and L2-CE total score (0.34) as well as between HM and BM (0.28).The largest increase between corresponding Pearson and disattenuated correlations was for pairs of scores with lower reliabilities such as CD5 and CD6, which had a Pearson correlation of 0.17 and a disattenuated correlation of 0.68. The smallest increase in correlations was observed in pairs of scores with larger reliabilities such as L2-CE total score and HM, which had a Pearson correlation of 0.23 and a disattenuated correlation of 0.28. The reliability of L2-CE was 0.87, 0.81 for HM, and 0.73 for BM. The reliabilities for the L2-CE competency domain scores ranged from 0.22 to 0.74. The small to moderate correlations between the L2-CE total score and the two L2-PE support the expectation that these examinations measure related but distinct constructs. The correlations between L2-PE and L2-CE competency domain subscores reflect the distribution of items defined by the L2-PE blueprint, providing evidence that the examinations are performing as designed. CONCLUSIONS: This study provides evidence supporting the validity of the blueprints for constructing COMLEX-USA Levels 2-CE and 2-PE examinations in concert with the purpose and nature of the examinations.

Asunto(s)

RESUMEN

Introduction The reliance on the United States Medical Licensing Examination (USMLE) Step 1 scores in residency selection creates problems for osteopathic medical students and the programs that review their applications. Although many osteopathic students take the USMLE to improve their standing for residency selection, students who score poorly may harm their candidacy. Simultaneously, programs unfamiliar with the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) may struggle to evaluate applicants who have not taken USMLE. Objective To determine the association between COMLEX-USA Level 1 and USMLE Step 1 scores and derive an equation that could be used to predict USMLE performance or approximate USMLE scores for applicants who have only taken COMLEX-USA. Methods We reviewed COMLEX-USA Level 1 and USMLE Step 1 scores for all students at the Lake Erie College of Osteopathic Medicine (LECOM), Bradenton campus, from January 2012 until December 2016. Linear regression was used to evaluate the relationship between COMLEX-USA Level 1 and USMLE Step 1 scores. Results Overall, 2097 students took both examinations during the study period. Every one-point increase in COMLEX-USA was associated with a 0.15 point increase in USMLE Step 1 score (standard error 11.5; model R2 0.56). On average, students scored 30 percentile points lower on USMLE Step 1 than on COMLEX-USA, and 24% of students scoring <500 on COMLEX-USA Level 1 failed USMLE Step 1. Conclusions Students or programs interested in predicting performance on USMLE Step 1 from performance on COMLEX-USA Level 1 can do so with the following equation: USMLE Step 1 = 0.15 (COMLEX-USA Level 1) + 138.7.

RESUMEN

CONTEXT: The Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) is a three level national standardized licensure examination for the practice of osteopathic medicine. The Comprehensive Medical Self Assessment Examination (COMSAE) is a three phase self assessment tool designed to gauge the base knowledge and ability of candidates preparing for COMLEX-USA. OBJECTIVES: To investigate how COMSAE Phase 1 (Phase 1) was used by candidates and how completing Phase 1 impacted their performance on the COMLEX-USA Level 1 (Level 1) examination. METHODS: Using data from the 2018-2019 administration of Level 1 and Phase 1 examinations, we counted the frequency of the unique Phase 1 forms taken by the candidates and calculated the correlation between the candidates' first attempt Phase 1 scores and the number of Phase 1 forms taken. We then calculated the correlation between the Level 1 scores and the Phase 1 scores. Next, we applied a multilevel regression model to examine the candidates' score improvement on the multiple Phase 1 forms taken. Finally, we investigated the effect of practicing through Phase 1 on the candidates' Level 1 performance using logistic regression models. RESULTS: The majority of candidates took one (2,414; 33.9%) to two (2,196; 30.8%) timed Phase 1 forms prior to the Level 1 examination. There was a significant negative correlation (r=-0.48, t(6,505)=-44.05, p<0.001) between the candidates' first attempt Phase 1 scores and the number of Phase 1 forms taken. There was a strong and positive correlation (r=0.66 to 0.74, p<0.001) between Phase 1 and Level 1 scores. With other variables controlled, on average, candidates' Phase 1 scores increased 23.2 points on one attempt from the previous attempt. Having the most recent Phase 1 score controlled, a greater number of Phase 1 forms taken was associated with an improvement on the Level 1 performance. CONCLUSIONS: The significant correlation between Phase 1 and Level 1 performance provided validity evidence for Phase 1. Moreover, our results suggested that candidates, especially those with lower performance on their initial Phase 1 attempt, might improve their Level 1 performance by taking multiple Phase 1 forms to monitor their academic improvement and gauge their readiness for Level 1.

Asunto(s)

RESUMEN

Purpose Examine the factors improving performance on national medical licensing board examinations. Rationale Accreditation Council for Graduate Medical Education (ACGME) accredited residency programs report the United States Medical Licensing Examination (USMLE) Step 1 and Comprehensive Osteopathic Licensing Examination-USA (COMLEX-USA) Level 1 scores as the most important criteria in selecting candidates to interview. Hypotheses (1) Certain resources are superior for exam preparation. (2) Certain practice tests better assess exam preparedness. (3) USMLE performance will correlate with the COMLEX-USA. Methods One-hundred and two (102) medical students were surveyed regarding preparation for and performance on COMLEX-USA Level 1 and USMLE Step 1. Results USMLE-specific question banks were positively correlated with performance on COMLEX-USA Level 1 and USMLE Step 1 while COMLEX-specific question banks showed no correlation. National Board of Medical Examiners (NBME) Comprehensive Basic Science Self Assessment (CBSSA) and National Board of Osteopathic Medical Examiners (NBOME) Comprehensive Osteopathic Medical Self-Assessment Examination (COMSAE) practice examinations were positively correlated with performance on the USMLE Step 1 and the COMLEX-USA Level 1. Scores on USMLE Step 1 and COMLEX-USA Level 1 were highly correlated. Students who took USMLE Step 1 performed better on COMLEX-USA Level 1 than those who did not. Conclusion COMLEX-specific resources may not adequately prepare students for COMLEX-USA Level 1. Students studying for COMLEX-USA Level 1 may benefit by preparing for USMLE Step 1.

RESUMEN

BACKGROUND: Responding to mandates from the Accreditation Council for Graduate Medical Education (ACGME) and American Osteopathic Association (AOA), residency programs have developed competency-based assessment tools. One such tool is the American College of Osteopathic Pediatricians (ACOP) program directors' annual report. High-stakes clinical skills licensing examinations, such as the Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation (COMLEX-USA Level 2-PE), also assess competency in several clinical domains. OBJECTIVE: The purpose of this study is to investigate the relationships between program director competency ratings of first-year osteopathic residents in pediatrics and COMLEX-USA Level 2-PE scores from 2005 to 2009. METHODS: The sample included all 94 pediatric first-year residents who took COMLEX-USA Level 2-PE and whose training was reviewed by the ACOP for approval of training between 2005 and 2009. Program director competency ratings and COMLEX-USA Level 2-PE scores (domain and component) were merged and analyzed for relationships. RESULTS: Biomedical/biomechanical domain scores were positively correlated with overall program director competency ratings. Humanistic domain scores were not significantly correlated with overall program director competency ratings, but did show moderate correlation with ratings for interpersonal and communication skills. The six ACGME or seven AOA competencies assessed empirically by the ACOP program directors' annual report could not be recovered by principal component analysis; instead, three factors were identified, accounting for 86% of the variance between competency ratings. DISCUSSION: A few significant correlations were noted between COMLEX-USA Level 2-PE scores and program director competency ratings. Exploring relationships between different clinical skills assessments is inherently difficult because of the heterogeneity of tools used and overlap of constructs within the AOA and ACGME core competencies.

Asunto(s)