RESUMEN

BACKGROUND: Brazilian medical schools equitably divide their medical education assessments into five content areas: internal medicine, surgery, pediatrics, obstetrics and gynecology, and public health. However, this division does not follow international patterns and may threaten the examinations' reliability and validity. OBJECTIVE: To assess the reliability indices of the content areas of serial, cross-institutional progress test examinations. DESIGN AND SETTINGS: This was an analytical, observational, and cross-sectional study conducted at nine public medical schools (mainly from the state of São Paulo) with progress test examinations conducted between 2017 and 2023. METHODS: The examinations covered the areas of basic sciences, internal medicine, surgery, pediatrics, obstetrics and gynecology, and public health. We calculated reliability indices using Cronbach's α, which indicates the internal consistency of a test. We used simple linear regressions to analyze temporal trends. RESULTS: The results showed that the Cronbach's α for basic sciences and internal medicine presented lower values, whereas gynecology, obstetrics, and public health presented higher values. After changes in the number of items and the exclusion of basic sciences as a separate content area, internal medicine ranked highest in 2023. Individually, all content areas except pediatrics remained stable over time. CONCLUSIONS: Maintaining an equitable division in assessment content may lead to suboptimal results in terms of assessment reliability, especially for internal medicine. Therefore, content sampling of medical knowledge for general assessments should be reappraised.

Asunto(s)

RESUMEN

ABSTRACT BACKGROUND: Brazilian medical schools equitably divide their medical education assessments into five content areas: internal medicine, surgery, pediatrics, obstetrics and gynecology, and public health. However, this division does not follow international patterns and may threaten the examinations' reliability and validity. OBJECTIVE: To assess the reliability indices of the content areas of serial, cross-institutional progress test examinations. DESIGN AND SETTINGS: This was an analytical, observational, and cross-sectional study conducted at nine public medical schools (mainly from the state of São Paulo) with progress test examinations conducted between 2017 and 2023. METHODS: The examinations covered the areas of basic sciences, internal medicine, surgery, pediatrics, obstetrics and gynecology, and public health. We calculated reliability indices using Cronbach's α, which indicates the internal consistency of a test. We used simple linear regressions to analyze temporal trends. RESULTS: The results showed that the Cronbach's α for basic sciences and internal medicine presented lower values, whereas gynecology, obstetrics, and public health presented higher values. After changes in the number of items and the exclusion of basic sciences as a separate content area, internal medicine ranked highest in 2023. Individually, all content areas except pediatrics remained stable over time. CONCLUSIONS: Maintaining an equitable division in assessment content may lead to suboptimal results in terms of assessment reliability, especially for internal medicine. Therefore, content sampling of medical knowledge for general assessments should be reappraised.

RESUMEN

OBJECTIVES: The transition from face-to-face to remote teaching is yet to be fully understood. In clinical training, traditional teaching must prevail because it is essential for the acquisition of skills and professionalism. However, the responses of each school to the pandemic and the decision on when to resume clerkship rotations were mixed. In this study, we aimed to analyze whether the time to resume clerkship rotations was associated with the performance of the students by using a multi-institutional Progress Test. METHODS: This is a cross-sectional study conducted at nine different Brazilian medical schools that administer the same annual Progress Test for all students. We included information from 1,470 clerkship medical students and analyzed the time of clinical training interruption as the independent variable and the student's scores as the dependent variable. RESULTS: The comparisons of the students' scores between the schools showed that there are differences; however, they cannot be attributed to the time the clerkship rotations were paused. The correlation between the schools' average scores and the time to resume clerkship rotations was not significant for the fifth year (r= -0.298, p=0.436) and for the sixth year (r= -0.440, p=0.240). By using a cubic regression model, the time to resume clerkship rotations could explain 3.4% of the 5-year students' scores (p<0.001) and 0.9% of the 6-year students, without statistical difference (p=0.085). CONCLUSIONS: The differences between the students' scores cannot be attributed to the time when the schools paused the clerkship rotations.

Asunto(s)

RESUMEN

SUMMARY OBJECTIVES: The transition from face-to-face to remote teaching is yet to be fully understood. In clinical training, traditional teaching must prevail because it is essential for the acquisition of skills and professionalism. However, the responses of each school to the pandemic and the decision on when to resume clerkship rotations were mixed. In this study, we aimed to analyze whether the time to resume clerkship rotations was associated with the performance of the students by using a multi-institutional Progress Test. METHODS: This is a cross-sectional study conducted at nine different Brazilian medical schools that administer the same annual Progress Test for all students. We included information from 1,470 clerkship medical students and analyzed the time of clinical training interruption as the independent variable and the student's scores as the dependent variable. RESULTS: The comparisons of the students' scores between the schools showed that there are differences; however, they cannot be attributed to the time the clerkship rotations were paused. The correlation between the schools' average scores and the time to resume clerkship rotations was not significant for the fifth year (r= -0.298, p=0.436) and for the sixth year (r= -0.440, p=0.240). By using a cubic regression model, the time to resume clerkship rotations could explain 3.4% of the 5-year students' scores (p<0.001) and 0.9% of the 6-year students, without statistical difference (p=0.085). CONCLUSIONS: The differences between the students' scores cannot be attributed to the time when the schools paused the clerkship rotations.

RESUMEN

BACKGROUND: Progress tests are longitudinal assessments of students' knowledge based on successive tests. Calibration of the test difficulty is challenging, especially because of the tendency of item-writers to overestimate students' performance. The relationships between the levels of Bloom's taxonomy, the ability of test judges to predict the difficulty of test items and the real psychometric properties of test items have been insufficiently studied. OBJECTIVE: To investigate the psychometric properties of items according to their classification in Bloom's taxonomy and judges' estimates, through an adaptation of the Angoff method. DESIGN AND SETTING: Prospective observational study using secondary data from students' performance in a progress test applied to ten medical schools, mainly in the state of São Paulo, Brazil. METHODS: We compared the expected and real difficulty of items used in a progress test. The items were classified according to Bloom's taxonomy. Psychometric properties were assessed based on their taxonomy and fields of knowledge. RESULTS: There was a 54% match between the panel of experts' expectations and the real difficulty of items. Items that were expected to be easy had mean difficulty that was significantly lower than that of items that were expected to be medium (P < 0.05) or difficult (P < 0.01). Items with high-level taxonomy had higher discrimination indices than low-level items (P = 0.026). We did not find any significant differences between the fields in terms of difficulty and discrimination. CONCLUSIONS: Our study demonstrated that items with high-level taxonomy performed better in discrimination indices and that a panel of experts may develop coherent reasoning regarding the difficulty of items.

Asunto(s)

RESUMEN

ABSTRACT BACKGROUND: Progress tests are longitudinal assessments of students' knowledge based on successive tests. Calibration of the test difficulty is challenging, especially because of the tendency of item-writers to overestimate students' performance. The relationships between the levels of Bloom's taxonomy, the ability of test judges to predict the difficulty of test items and the real psychometric properties of test items have been insufficiently studied. OBJECTIVE: To investigate the psychometric properties of items according to their classification in Bloom's taxonomy and judges' estimates, through an adaptation of the Angoff method. DESIGN AND SETTING: Prospective observational study using secondary data from students' performance in a progress test applied to ten medical schools, mainly in the state of São Paulo, Brazil. METHODS: We compared the expected and real difficulty of items used in a progress test. The items were classified according to Bloom's taxonomy. Psychometric properties were assessed based on their taxonomy and fields of knowledge. RESULTS: There was a 54% match between the panel of experts' expectations and the real difficulty of items. Items that were expected to be easy had mean difficulty that was significantly lower than that of items that were expected to be medium (P < 0.05) or difficult (P < 0.01). Items with high-level taxonomy had higher discrimination indices than low-level items (P = 0.026). We did not find any significant differences between the fields in terms of difficulty and discrimination. CONCLUSIONS: Our study demonstrated that items with high-level taxonomy performed better in discrimination indices and that a panel of experts may develop coherent reasoning regarding the difficulty of items.

Asunto(s)

RESUMEN

Este estudo teve por objetivo averiguar as potencialidades do Teste de Progresso (TP) no auxílio à gestão acadêmica. Foram analisados os desempenhos dos estudantes do curso de Medicina no TP aplicado em 2008 e reaplicado sem modificações em 2011. Os resultados demonstraram que, no TP de 2008, a progressão ocorreu a cada dois anos de curso, e no TP 2011 esta progressão iniciou-se somente a partir da terceira série. Nas áreas de Saúde Coletiva, Clínica Cirúrgica e Áreas Básicas, não houve acúmulo de conhecimentos gradativo desde o início do curso, o que merece reflexões por parte dos gestores do currículo.

This study aimed to investigate the potential of progress testing to help academic management. To this end, the performances of medical students in the Progress Test applied in 2008 and reapplied without modifications in 2011 were analysed. The results showed that in the Progress Test of 2008 progression occurred once every two years of the course and in TP 2011 progression started only from the third year onwards. In the areas of Public Health, Surgery and Basic Sciences there was no gradual accumulation of knowledge since the beginning of the course, which merits reflection on behalf of the curriculum managers.

RESUMEN

Este estudo tem a finalidade de quantificar os diagnósticos de anemia em crianças de 0 a 3 anos de idade, realizados pela Unidade de Saúde da Família Vila Real, localizada em Marília/SP, e avaliar a necessidade de implementação de recomendações para diagnóstico da anemia. O estudo é retrospectivo e abrangeu amostra de crianças de 0 a 3 anos de idade atendidas nessa unidade de saúde, sendo os dados dos hemogramas e dos prontuários coletados, computados e analisados no programa Epi-Info. Entre as 278 crianças atendidas pelo posto, 63 (22,7 por cento) tinham hemograma no prontuário, sem uniformidade nas idades na época do exame. Dessas, 49 (77,8 por cento) tinham hemoglobina<11,0 g/dl; 41 (65,07 por cento), hematócrito abaixo do valor limite; e 38 (61,3 por cento), volume corpuscular médio reduzido. Essas são evidências de que existe alta prevalência de anemia e deficiência de ferro na área estudada, com forte indicativo da importância de mais projetos e ações básicas, visando à diminuição do número de crianças com anemia. Observaram-se crianças com diferentes idades na época do exame, demonstrando a necessidade de recomendações para seu diagnóstico.

Asunto(s)