RESUMEN

Background: With the increasing application of large language models like ChatGPT in various industries, its potential in the medical domain, especially in standardized examinations, has become a focal point of research. Objective: The aim of this study is to assess the clinical performance of ChatGPT, focusing on its accuracy and reliability in the Chinese National Medical Licensing Examination (CNMLE). Methods: The CNMLE 2022 question set, consisting of 500 single-answer multiple choices questions, were reclassified into 15 medical subspecialties. Each question was tested 8 to 12 times in Chinese on the OpenAI platform from April 24 to May 15, 2023. Three key factors were considered: the version of GPT-3.5 and 4.0, the prompt's designation of system roles tailored to medical subspecialties, and repetition for coherence. A passing accuracy threshold was established as 60%. The χ2 tests and κ values were employed to evaluate the model's accuracy and consistency. Results: GPT-4.0 achieved a passing accuracy of 72.7%, which was significantly higher than that of GPT-3.5 (54%; P<.001). The variability rate of repeated responses from GPT-4.0 was lower than that of GPT-3.5 (9% vs 19.5%; P<.001). However, both models showed relatively good response coherence, with κ values of 0.778 and 0.610, respectively. System roles numerically increased accuracy for both GPT-4.0 (0.3%-3.7%) and GPT-3.5 (1.3%-4.5%), and reduced variability by 1.7% and 1.8%, respectively (P>.05). In subgroup analysis, ChatGPT achieved comparable accuracy among different question types (P>.05). GPT-4.0 surpassed the accuracy threshold in 14 of 15 subspecialties, while GPT-3.5 did so in 7 of 15 on the first response. Conclusions: GPT-4.0 passed the CNMLE and outperformed GPT-3.5 in key areas such as accuracy, consistency, and medical subspecialty expertise. Adding a system role insignificantly enhanced the model's reliability and answer coherence. GPT-4.0 showed promising potential in medical education and clinical practice, meriting further study.

Asunto(s)

RESUMEN

BACKGROUND: The National Medical Licensing Examination (NMLE) is the only objective, standardized metric to evaluate whether a medical student possessing the professional knowledge and skills necessary to work as a physician. However, the overall pass rate of NMLE in our hospital in 2021 was much lower than that of Peking Union Medical College Hospital, which was required to be further improved. METHODS: To find the reasons for the unsatisfactory performance in 2021, the quality improvement team (QIT) organized regular face-to-face meetings for in-depth discussion and questionnaire, and analyzed the data by "Plato analysis" and "Brainstorming method". After finding out the reasons, the "Plan-Do-Check-Action" (PDCA) cycle was continued to identify and solve problems, which included the formulation and implementation of specific training plans by creating the "Gantt charts", the check of effects, and continuous improvements from 2021 to 2022. Detailed information about the performance of students in 2021 and 2022, and the attendance, assessment, evaluation and suggestions from our hospital were provided by the relevant departments, and the pass rate-associated data was collected online. RESULTS: After the PDCA plan, the pass rate of NMLE in our hospital increased by 10.89% from 80.15% in 2021 to 91.04% in 2022 (P = 0.0109), with the pass rate of skill examination from 95.59% in 2021 to 99.25% in 2022 (P = 0.0581) and theoretical examination from 84.5% in 2021 to 93.13% in 2022 (P = 0.027). Additionally, the mean scores of all examinees increased with the theoretical examination score increasing from 377.0 ± 98.76 in 2021 to 407.6 ± 71.94 in 2022 (P = 0.004). CONCLUSIONS: Our results showed a success application of the PDCA plan in our hospital which improved the pass rate of the NMLE in 2022, and the PDCA plan may provide a practical framework for future medical education and further improve the pass rate of NMLE in the next year.

Asunto(s)

RESUMEN

The Clinical Medicine Proficiency Test developed by the National Medical Examination Center of China in 2020 aims to assess whether clinical medical students possess the necessary medical humanities and basic medical theoretical knowledge and skills required for clinical internships. Since 2002, Japan has implemented the pre-clinical clerkship objective structured clinical examination and computer-based testing, which share similarities with China's proficiency test in terms of examination objectives, content, format, and score evaluation. Through comparing the examinations of China and Japan, this article concludes that it is necessary to learn from Japan's experience to expedite the process of promoting the Clinical Medicine Proficiency Test as an industry admission examination in China. We suggest that medical schools should keep tracking the development of the proficiency test and fully leverage its role in urging students to pay more attention to clinical internships. We also urge medical schools to establish a three-level examination system based on post competency criteria in line with the standards of the National Medical Licensing Examination to improve the quality of medical education.

RESUMEN

BACKGROUND: ChatGPT (OpenAI) has gained considerable attention because of its natural and intuitive responses. ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers, as stated by OpenAI as a limitation. However, considering that ChatGPT is an interactive AI that has been trained to reduce the output of unethical sentences, the reliability of the training data is high and the usefulness of the output content is promising. Fortunately, in March 2023, a new version of ChatGPT, GPT-4, was released, which, according to internal evaluations, was expected to increase the likelihood of producing factual responses by 40% compared with its predecessor, GPT-3.5. The usefulness of this version of ChatGPT in English is widely appreciated. It is also increasingly being evaluated as a system for obtaining medical information in languages other than English. Although it does not reach a passing score on the national medical examination in Chinese, its accuracy is expected to gradually improve. Evaluation of ChatGPT with Japanese input is limited, although there have been reports on the accuracy of ChatGPT's answers to clinical questions regarding the Japanese Society of Hypertension guidelines and on the performance of the National Nursing Examination. OBJECTIVE: The objective of this study is to evaluate whether ChatGPT can provide accurate diagnoses and medical knowledge for Japanese input. METHODS: Questions from the National Medical Licensing Examination (NMLE) in Japan, administered by the Japanese Ministry of Health, Labour and Welfare in 2022, were used. All 400 questions were included. Exclusion criteria were figures and tables that ChatGPT could not recognize; only text questions were extracted. We instructed GPT-3.5 and GPT-4 to input the Japanese questions as they were and to output the correct answers for each question. The output of ChatGPT was verified by 2 general practice physicians. In case of discrepancies, they were checked by another physician to make a final decision. The overall performance was evaluated by calculating the percentage of correct answers output by GPT-3.5 and GPT-4. RESULTS: Of the 400 questions, 292 were analyzed. Questions containing charts, which are not supported by ChatGPT, were excluded. The correct response rate for GPT-4 was 81.5% (237/292), which was significantly higher than the rate for GPT-3.5, 42.8% (125/292). Moreover, GPT-4 surpassed the passing standard (>72%) for the NMLE, indicating its potential as a diagnostic and therapeutic decision aid for physicians. CONCLUSIONS: GPT-4 reached the passing standard for the NMLE in Japan, entered in Japanese, although it is limited to written questions. As the accelerated progress in the past few months has shown, the performance of the AI will improve as the large language model continues to learn more, and it may well become a decision support system for medical professionals by providing more accurate information.

Asunto(s)

RESUMEN

ChatGPT, a language model developed by OpenAI, uses a 175 billion parameter Transformer architecture for natural language processing tasks. This study aimed to compare the knowledge and interpretation ability of ChatGPT with those of medical students in China by administering the Chinese National Medical Licensing Examination (NMLE) to both ChatGPT and medical students. We evaluated the performance of ChatGPT in three years' worth of the NMLE, which consists of four units. At the same time, the exam results were compared to those of medical students who had studied for five years at medical colleges. ChatGPT's performance was lower than that of the medical students, and ChatGPT's correct answer rate was related to the year in which the exam questions were released. ChatGPT's knowledge and interpretation ability for the NMLE were not yet comparable to those of medical students in China. It is probable that these abilities will improve through deep learning.

Asunto(s)

RESUMEN

Objective:To investigate the effects of teaching reform to strengthen the integration of basic medicine and clinical medicine by analyzing the scores of the National Medical Licensing Examination from 2014 to 2018.Methods:Through the improvement of the training program, the establishment of subject integration courses, the standardization of case teaching mode and the establishment of comprehensive medical experiment courses, the teaching reform of the combination of basic medicine and clinical medicine was carried out. The differences in the scores of the National Medical Licensing Examination between the students from Dalian University and those from the whole country were compared, including the average total scores, the passing rates, the average mastery rates of the basic medicine, and the average mastery rates of the cognitive level of the basic medicine.Results:In 2014, 2015 and 2016, the average total scores of the students from Dalian University were 384.70±53.62, 395.45±50.49, and 401.43±50.88, respectively, which were lower than those from the whole country (396.15±58.85, 400.78±56.88, and 405.15±58.06). The average mastery rates of the basic medicine of the students from Dalian University were 56.28%, 62.45%, and 64.92%, respectively, lower than those from the whole country (58.41%, 62.46%, and 65.36%). The average mastery rates of the basic medicine of the students form Dalian University from 2014 to 2016 were lower than those from the whole country, and those were higher in 2017 to 2018 than the whole country. The average total scores of the students form Dalian University in 2017 and 2018 were 397.11 and 400.26 ± 61.15, respectively, higher than those from the whole country (388.91 and 397.01±58.35). The average mastery rates of the basic medicine of the students form Dalian University were 61.53% and 59.85%, respectively, higher than those from the whole country (58.64% and 58.38%). The total passing rates of the students from Dalian University in 2014 and 2016 were 91.11% and 72.73%, respectively, lower than those from the whole country (91.18% and 73.75%), while in 2015, 2017 and 2018 those were 76.47%, 77.78%, and 77.59%, respectively, all higher than those from the whole country (73.59%, 74.29%, and 76.89%). All the data showed a trend of increase with the teaching reform process.Conclusion:Strengthening the teaching reform of the combination of basic medicine and clinical medicine during the learning stage of basic medicine could improve students' learning interest and attention to the basic medical knowledge, strengthen the understanding of the relevance of basic medicine to clinical medicine, and play a positive role in the cultivation of medical students' post competency and the achievement of the goal of training applied medical talents.

RESUMEN

Since the implementation of National Medical Licensing Examination (NMLE) system in China, millions of individuals have participated in the examination and have been licensed as physicians. Over the past few decades, NMLE has played a major role in evaluating and guiding Chinese medical education and has made great progress. This commentary discusses the main experience, challenges, and prospects of NMLE in China.

Asunto(s)

RESUMEN

Objective:To explore the objective structured clinical examination (OSCE) on the improvement of clinical competence of stomatology undergraduates, and to promote their ability and skills of taking the National Medical Licensing Examination (NMLE).Methods:The OSCE was implemented to evaluate the clinical skills of stomatology undergraduates from Batch 2009 to Batch 2012. The OSCE results were analyzed and compared with the NMLE (Stomatology) results on the correlation coefficient analysis of annual passing rate.Results:The passing rate of practical skills in the NMLE (Stomatology) of Batch 2009 to Batch 2012 stomatology undergraduates was basically consistent with the excellent rate of OSCE, and the correlation coefficient was 0.94.Conclusion:It has not been proved that the OSCE assessment organized by our school can increase the passing rate of practical skills in the NMLE, but it has certainly helped students better understand and apply theoretical knowledge, and it plays a positive role in promoting students' future development.

RESUMEN

INTRODUCTION: In China, although burn treatment develops rapidly, and ranks in the forefront of the world, there is a relative shortage of burn specialists, which limits the development of burn education. In traditional curriculum of surgery education, burn surgery education accounts for few proportions, which results in the indifference to the burn surgery among medical students. To date, few research reported the application of Clinical pathway- Problem based Learning (CP-PBL) in burn surgery education. The objective of the study is to explore the teaching effect of this novel teaching method in burn surgery education. METHODS: A pilot study was performed. One hundred and six students were randomly divided into a Lecture based Learning (LBL) only group (control group) and a LBL combined Clinical pathway- Problem based Learning group (observation group). A set of test was designed as evaluation criteria based on questions of burn surgery in National Medical Licensing Examination (NMLE) from 2011 to 2018. RESULTS: The students with Clinical pathway- Problem based Learning had better academic performances in profession theory. Type A2 and Type A3/A4 scores in the observation group were higher than those in the control group (p < 0.05). The scores of the observation group were higher than those of control group in the domains of understanding and application (p < 0.05). They also have higher favorable impressions of learning experience. CONCLUSIONS: More active approaches yield more learning and are viewed more favorable, which provides a vital message for the evolution of curriculum in Chinese medical schools.

Asunto(s)

RESUMEN

In traditional medical school curriculum of cancer education in China, there is a very limited amount of teaching about breast cancer. The current situation may result in indifference to breast cancer education among medical students. Case-based learning (CBL) is a popular teaching method based on clinical cases. To date, there are few research reports about the application and research of CBL in breast cancer education. The aim of this study is to explore the teaching effect about CBL combined with lecture-based learning (LBL) in breast cancer education. Questions of breast cancer in National Medical Licensing Examination (NMLE) from 2011 to 2018 were analyzed. The questions about breast cancer were used as the evaluation criteria for this study. In this pilot study, a total of 140 students were randomly divided into a lecture only group (control group) and a lecture plus CBL group (observation group). The students in the observation group had better academic performances and abilities of memory, understanding, and application. They also had higher favorable impressions of the learning experience. In conclusion, more active approaches yield more learning and are viewed more favorably. CBL plus lecture can significantly improve education about breast cancer among medical students, which may be an important message for the evolution of curriculum in Chinese medical schools.

Asunto(s)

RESUMEN

Objective To summarize and analyze the results of our university graduates' participation in the national medical licensing examination in order to investigate the clinical medical teaching reform.Methods There were 770 candidates who took the NMLE from 2011 to 2013.Excel 2007 was used to statistically analyze the total score,passing rate,average score and mastery rate for each subject,and average score and mastery rate at different cognitive level.Results In these three years,our candidates' total pass rate was higher than the national average total pass rate,respectively higher 17.11％,18.10％,17.82％;the pass rate of practical skills examination was also higher than the national average pass rate,respectively higher 7.30％ and 7.19％ and 5.90％;the pass rate of theory examination was higher than the national average pass rate,respectively higher 12.91％,12.97％ and 14.29％.The average score and mastery rate for each subject,the average score and mastery rate at different cognitive level were also all higher than those of the national average of medical colleges.Conclusions National medical licensure examination can be used as an important reference,through optimization of medical curriculum and teaching mode,optimization of clinical practice skills training system and optimization of clinical practical skill evaluation system etc.,to further improve the clinical medical personnel training mode.

RESUMEN

BACKGROUND: Contaminated blood cultures lead to diagnostic challenges and place a burden on healthcare services. AIM: To determine the impact of introducing a clinical skills test (CST) as part of the medical licensing examination and an institutional education programme on the contamination rates of blood cultures. METHODS: A prospective cohort study was conducted from 2009 through 2013 in all wards of a tertiary-care teaching hospital. We evaluated the effects of the CST, which was added to the National Medical Licensing Examination in Korea (KMLE) in 2010 and our institutional education programme, which began in 2013. The medical interns in charge of collection of blood for culture were divided in three groups with presence or absence of CST and the institutional education programme. The primary outcome was the percentage of blood cultures contaminated in each group, which were compared using the Poisson regression model. Participants' self-rated scores for the blood draw procedure were also analysed. FINDINGS: Although introduction of the CST in the KMLE failed to reduce blood culture contamination rate (1.36% vs 1.35%; P = 0.734), the institutional education programme significantly reduced the contamination rate (1.35% vs 1.00%; P < 0.0001). Most participants answered that they always followed each step correctly except for waiting the recommended contact time after applying the antiseptic. CONCLUSION: The educational intervention, not the introduction of CST in the KMLE, was effective in reducing overall contamination rates.

Asunto(s)

RESUMEN

Objective The empirical validity of the National Medical Licensing Examination (NMLE) has rarely been studied. This is one of the first studies comparing NMLE and annual assess-ment scores. This study provided the criterion-related validity evidence for the NMLE. The NMLE can be used to test the abilities and skills of physicians. Method We evaluated the test 163 residents in 12 medical school-affiliated hospitals who had finished their first year of tralning and had taken the NMLE and an annual skills assessment. We used Pearson correlations to estimate the relationship be-tween NMLE scores and annual assessment scores. Results A total of 163 residents met inclusion requirements. Correlations between NMLE scores and annual assessment scores ranged from 0.300 to 0.843. Conclusions The criterion-related validity of NMLE was high, so it can be use to evaluate the residents clinical competence indeed.

RESUMEN

This paper gives a brief introduction to China National Medical Licen sing Examination(NMLE)and focuses on the definition,modification and application protocol of Angoff Method in setting a standard for NMLE Comprehensive Written Test.Statistic analysis indicates the validity of modifled Angoff Method.Suggestion is provided for future improvement.