RESUMEN

BACKGROUND: Brazilian medical schools equitably divide their medical education assessments into five content areas: internal medicine, surgery, pediatrics, obstetrics and gynecology, and public health. However, this division does not follow international patterns and may threaten the examinations' reliability and validity. OBJECTIVE: To assess the reliability indices of the content areas of serial, cross-institutional progress test examinations. DESIGN AND SETTINGS: This was an analytical, observational, and cross-sectional study conducted at nine public medical schools (mainly from the state of São Paulo) with progress test examinations conducted between 2017 and 2023. METHODS: The examinations covered the areas of basic sciences, internal medicine, surgery, pediatrics, obstetrics and gynecology, and public health. We calculated reliability indices using Cronbach's α, which indicates the internal consistency of a test. We used simple linear regressions to analyze temporal trends. RESULTS: The results showed that the Cronbach's α for basic sciences and internal medicine presented lower values, whereas gynecology, obstetrics, and public health presented higher values. After changes in the number of items and the exclusion of basic sciences as a separate content area, internal medicine ranked highest in 2023. Individually, all content areas except pediatrics remained stable over time. CONCLUSIONS: Maintaining an equitable division in assessment content may lead to suboptimal results in terms of assessment reliability, especially for internal medicine. Therefore, content sampling of medical knowledge for general assessments should be reappraised.

Asunto(s)

RESUMEN

RESUMO Introdução: A integralidade da atenção à saúde da criança constitui diretriz para o ensino de pediatria, sendo essenciais as atividades práticas em todos níveis de atenção do Sistema Único de Saúde. Objetivo: Este estudo teve como objetivo descrever carga horária, cenários de práticas e preceptoria utilizados no ensino da saúde da criança e do adolescente. Método: Trata-se de um estudo transversal, com aplicação de questionário constituído por 43 questões objetivas e duas questões abertas, encaminhado por meio de formulário eletrônico para 247 escolas médicas do país que contavam com pelo menos uma turma formada em 2019. Resultado: Responderam ao questionário 37 (14,97%) escolas médicas, sendo 16 públicas e 21 privadas de 14 estados da Federação e das cinco regiões. Verificou-se mediana de 940,5 horas direcionadas à saúde da criança e do adolescente, equivalente a 11% da carga horária total dos cursos; 13 escolas apresentaram inserção de temas de saúde da criança desde o primeiro ano, e a maioria (75,8%), a partir do terceiro ano. Atividades práticas predominaram no internato: 87,5% (quinto ano) e 88,8% (sexto ano). Os cenários incluíram unidades básicas de saúde, comunidade, ambulatórios de pediatria geral e especializados, unidades de internação, serviços de neonatologia, urgência e emergência e laboratórios de habilidades e simulação. Foi referido que o ensino de puericultura é desenvolvido em ambulatórios de pediatria geral (32 escolas) e em unidades básicas de saúde dos municípios (32 escolas), estas consideradas essenciais para formação. Docentes pediatras desenvolvem preceptoria na maioria dos cenários de práticas; pediatras da instituição de ensino ou do sistema local de saúde estão mais presentes em unidades de internação (70,3% e 54,0%, respectivamente) e ambulatórios especializados (54,0% e 35,1%, respectivamente). Conclusão: Com participação de 37 escolas médicas, este estudo apresenta limitações para generalizações sobre o ensino no país. Neste estudo observou-se que o ensino sobre saúde da criança e do adolescente desenvolve-se em todos níveis de atenção à saúde, visando à integralidade, sendo destinados em média 11% da carga horária total do curso para esse ensino. Houve predominância de docentes e médicos pediatras na preceptoria, e a puericultura permaneceu como componente relevante na atenção básica, sendo apontados desafios para manutenção desse cenário de práticas.

ABSTRACT Introduction: Comprehensive child health care is a guideline in pediatric teaching, and it is essential to develop practical activities considering all levels of health care in the Unified Health System. Objectives: To describe the hours involved, practice scenarios and preceptorship in pediatric and childcare teaching in medical schools. Method: A cross-sectional study using a questionnaire with 43 objectives and two open questions, sent by electronic forms to medical schools across the country that had already graduated at least one class in 2019. Results: 37 (14.97%) medical schools with at least one graduated class answered the questionnaire (16 public and 21 private schools) from 14 states in the five regions of the country. A median of 940.5 hours was found for the teaching of pediatrics, equivalent to 11% of the total medical course; 13 schools included child health topics since the first year and the majority (75.8%) from the third year onwards. Practical activities predominated in the internship: 87.5% (5th year) and 88.8% (6th year). The used settings include primary health care, general and specialty pediatric outpatient clinics, inpatient units, neonatology units, emergency services and simulation laboratories. It was reported that childcare teaching is carried out in general pediatric outpatient clinics (32 schools) and basic community health units (32 schools), with an emphasis on primary care as the essential setting for teaching childcare. Pediatric teachers provide preceptorship in all practice settings; non-teaching pediatricians from the medical institution or the local health system are more present in inpatient units (70.3% and 54.0%, respectively) and specialty outpatient clinics (54.0% and 35.1%, respectively). Conclusions: With the participation of 37 medical schools, this study has limitations for the generalizations about teaching in the country. Pediatric teaching is carried out in practice environments at all levels of care, demonstrating the importance of comprehensive child and adolescent health care, with an average of 11% of the course total workload allocated to this teaching. The predominant participation of pediatricians as teachers was observed. The learning of childcare has remained a relevant component of pediatric training and its development is significant in primary care, although there are challenges to preserving this practice setting.

RESUMEN

ABSTRACT BACKGROUND: Brazilian medical schools equitably divide their medical education assessments into five content areas: internal medicine, surgery, pediatrics, obstetrics and gynecology, and public health. However, this division does not follow international patterns and may threaten the examinations' reliability and validity. OBJECTIVE: To assess the reliability indices of the content areas of serial, cross-institutional progress test examinations. DESIGN AND SETTINGS: This was an analytical, observational, and cross-sectional study conducted at nine public medical schools (mainly from the state of São Paulo) with progress test examinations conducted between 2017 and 2023. METHODS: The examinations covered the areas of basic sciences, internal medicine, surgery, pediatrics, obstetrics and gynecology, and public health. We calculated reliability indices using Cronbach's α, which indicates the internal consistency of a test. We used simple linear regressions to analyze temporal trends. RESULTS: The results showed that the Cronbach's α for basic sciences and internal medicine presented lower values, whereas gynecology, obstetrics, and public health presented higher values. After changes in the number of items and the exclusion of basic sciences as a separate content area, internal medicine ranked highest in 2023. Individually, all content areas except pediatrics remained stable over time. CONCLUSIONS: Maintaining an equitable division in assessment content may lead to suboptimal results in terms of assessment reliability, especially for internal medicine. Therefore, content sampling of medical knowledge for general assessments should be reappraised.

RESUMEN

Progress Testing (PT) is an assessment tool whose use has grown throughout Brazil in the last decade. PT makes it possible to assess the students' knowledge gain throughout the undergraduate course and, for their interpretations to be valid, their items (questions) must have adequate quality from the point of view of content validity and reliability of results. In this study, we analyzed the psychometric characteristics of the items and the performance of students in the content area of surgery from 2017 to 2023. For the analyses, we used the assumptions of Classical Test Theory, Bloom's taxonomy and Cronbach's alpha reliability coefficient. The items were easy (average difficulty index between 0.3-0.4), with fair to good discrimination (discrimination index between 0.3-0.4) and with a predominance of medium to high taxonomy. Reliability remained substantial over the years (>0.6). Students' knowledge gain in surgery was found to be progressive and more important from the 3rd year of the undergraduate course, reaching approximately 70-75% in the 6th year. This measurements framework can be replicated in other contexts for a better understanding of student learning and for qualification of evaluation processes.

Asunto(s)

RESUMEN

PURPOSE: Trauma leagues (TLs) are extracurricular programs that offer medical students supervised exposure to trauma and acute care surgery, mentorship, and participation in other academic activities. TLs are fully approved medical schools, and currently, over 100 TLs exist in Brazil. We hypothesized that the performance/competence of medical students who participated in TLs was superior compared to non-participants. This study evaluated and compared the cognitive performance and technical skills of the two groups. METHODS: This retrospective cohort study evaluated the performance of TL medical students to non-TL alumni from 2005 to 2017, using the students' academic performance coefficient, Clinical Competence Assessment, and Progress Test results. SigmaPlot 12.0 software was used to perform statistical analyses, including Mann-Whitney comparison tests and the Kruskal-Wallis test to confirm the data. RESULTS: Of the 1366 medical students who graduated from a Brazilian university, 966 were included, with 17.9% having participated in TL. Compared to non-TL participants, TL students demonstrated better cognitive performance according to the performance coefficient (p = 0.017) and Progress Test result (p < 0.001), and higher achievement in the Clinical Competence Assessment (p < 0.001). CONCLUSION: The academic performance of TL students was superior to that of non-TL students at the University of Campinas (Unicamp), suggesting a positive impact of TL in the preparation of future doctors. The study findings suggest that participation in TL at Unicamp was beneficial in preparing better doctors and should be considered by medical schools worldwide. EVIDENCE LEVEL: II (Retrospective cohort).

Asunto(s)

RESUMEN

A reorientação da formação profissional por meio de políticas públicas ofereceu a oportunidade para a criação de componentes curriculares, para formar estudantes que exercitem práticas interprofissionais. Este trabalho faz uma análise do contexto e implicações sobre a implantação de uma disciplina interprofissional nos cursos da saúde em uma universidade pública. Foi realizado um estudo com perfil exploratório de abordagem qualitativa utilizando-se grupos focais constituídos por tutores, preceptores e estudantes. Definidas as categorias e seus descritores foi utilizado o modelo 3P para a análise de conteúdo e dos resultados. Vale destacar a ressignificação do papel docente, a percepção dos estudantes frente aos desafios do ensino interprofissional e a aproximação com o serviço de saúde. Correlacionamos na discussão geral deste texto linhas de discussão junto à assistência à pessoa, à sociologia e à pedagogia. A inovação proposta pela disciplina interprofissional de Atenção em Saúde seguiu o ideário de uma agenda internacional. Os relatos de tutores, preceptores e estudantes foram um estímulo decisivo para prosseguir avançando nas demais séries dos cursos, em atividades colaborativas e práticas interprofissionais.

The reorientation of professional training through public policies offered the opportunity for the creation of curricular components to train students to exercise interprofessional practices. This paper makes an analysis of the context and implications on the implementation of an interprofessional discipline in health courses at a public university. A qualitative exploratory study was carried out using focus groups made up of tutors, preceptors and students. Once the categories and descriptors were defined, the 3P model was used for the content and results analysis. It is worth highlighting the redefinition of the teaching role, the students' perception of the challenges of interprofessional teaching, and the approximation with the health service. We correlate in the general discussion of this text lines of discussion with the person care, sociology, and pedagogy. The innovation proposed by the interprofessional discipline of Health Care followed the ideology of an international agenda. The reports from tutors, preceptors and students were a decisive stimulus to continue advancing in the other series of courses, in collaborative activities and interprofessional practices.

La reorientación de la formación profesional a través de políticas públicas ofreció la oportunidad para la creación de componentes curriculares para capacitar a los estudiantes en el ejercicio de prácticas interprofesionales. Este trabajo hace un análisis del contexto y de las implicaciones en la implementación de una disciplina interprofesional en cursos de salud en una universidad pública. Se realizó un estudio cualitativo exploratorio mediante grupos focales formados por tutores, preceptores y estudiantes. Una vez definidas las categorías y los descriptores, se utilizó el modelo 3P para el análisis de contenido y de resultados. Cabe destacar la redefinición del rol docente, la percepción de los estudiantes sobre los desafíos de la enseñanza interprofesional y la aproximación con el servicio de salud. Correlacionamos en la discusión general de este texto líneas de discusión con el cuidado de la persona, la sociología y la pedagogía. A inovação proposta pela disciplina interprofesional da Saúde seguiu a ideologia de uma agenda internacional. Los informes de tutores, preceptores y estudiantes fueron un estímulo decisivo para seguir avanzando en la otra serie de cursos, en actividades colaborativas y prácticas interprofesionales.

RESUMEN

ABSTRACT Progress Testing (PT) is an assessment tool whose use has grown throughout Brazil in the last decade. PT makes it possible to assess the students' knowledge gain throughout the undergraduate course and, for their interpretations to be valid, their items (questions) must have adequate quality from the point of view of content validity and reliability of results. In this study, we analyzed the psychometric characteristics of the items and the performance of students in the content area of surgery from 2017 to 2023. For the analyses, we used the assumptions of Classical Test Theory, Bloom's taxonomy and Cronbach's alpha reliability coefficient. The items were easy (average difficulty index between 0.3-0.4), with fair to good discrimination (discrimination index between 0.3-0.4) and with a predominance of medium to high taxonomy. Reliability remained substantial over the years (>0.6). Students' knowledge gain in surgery was found to be progressive and more important from the 3rd year of the undergraduate course, reaching approximately 70-75% in the 6th year. This measurements framework can be replicated in other contexts for a better understanding of student learning and for qualification of evaluation processes.

RESUMO O Teste de Progresso (TP) é uma ferramenta de avaliação cujo uso tem crescido em todo o Brasil na última década. O TP permite avaliar o ganho de conhecimento dos estudantes ao longo do curso de graduação e, para que suas interpretações sejam válidas, é preciso que seus itens (questões) tenham qualidade adequada do ponto de vista de validade de conteúdo e confiabilidade de resultados. Neste estudo, analisamos as características psicométricas dos itens e o desempenho dos estudantes na área de cirurgia do TP de 2017 a 2023. Para as análises, usamos os pressupostos da Teoria Clássica dos Testes, a taxonomia de Bloom e o coeficiente de fidedignidade alfa de Cronbach. Os itens se mostraram fáceis (índice de dificuldade média entre 0,3-0,4), com discriminação de regular a boa (índice de discriminação entre 0,3-0,4) e com predomínio de questões de média a alta taxonomia. A confiabilidade se manteve substancial ao longo dos anos (>0,6). O ganho de conhecimento dos estudantes em cirurgia é progressivo e mais importante a partir do 3º ano do curso de graduação, chegando a aproximadamente 70-75% no 6º ano. Este arcabouço de aferições pode ser replicado em outros contextos para melhor compreensão do aprendizado dos estudantes e para qualificação dos processos avaliativos.

RESUMEN

OBJECTIVES: The transition from face-to-face to remote teaching is yet to be fully understood. In clinical training, traditional teaching must prevail because it is essential for the acquisition of skills and professionalism. However, the responses of each school to the pandemic and the decision on when to resume clerkship rotations were mixed. In this study, we aimed to analyze whether the time to resume clerkship rotations was associated with the performance of the students by using a multi-institutional Progress Test. METHODS: This is a cross-sectional study conducted at nine different Brazilian medical schools that administer the same annual Progress Test for all students. We included information from 1,470 clerkship medical students and analyzed the time of clinical training interruption as the independent variable and the student's scores as the dependent variable. RESULTS: The comparisons of the students' scores between the schools showed that there are differences; however, they cannot be attributed to the time the clerkship rotations were paused. The correlation between the schools' average scores and the time to resume clerkship rotations was not significant for the fifth year (r= -0.298, p=0.436) and for the sixth year (r= -0.440, p=0.240). By using a cubic regression model, the time to resume clerkship rotations could explain 3.4% of the 5-year students' scores (p<0.001) and 0.9% of the 6-year students, without statistical difference (p=0.085). CONCLUSIONS: The differences between the students' scores cannot be attributed to the time when the schools paused the clerkship rotations.

Asunto(s)

RESUMEN

SUMMARY OBJECTIVES: The transition from face-to-face to remote teaching is yet to be fully understood. In clinical training, traditional teaching must prevail because it is essential for the acquisition of skills and professionalism. However, the responses of each school to the pandemic and the decision on when to resume clerkship rotations were mixed. In this study, we aimed to analyze whether the time to resume clerkship rotations was associated with the performance of the students by using a multi-institutional Progress Test. METHODS: This is a cross-sectional study conducted at nine different Brazilian medical schools that administer the same annual Progress Test for all students. We included information from 1,470 clerkship medical students and analyzed the time of clinical training interruption as the independent variable and the student's scores as the dependent variable. RESULTS: The comparisons of the students' scores between the schools showed that there are differences; however, they cannot be attributed to the time the clerkship rotations were paused. The correlation between the schools' average scores and the time to resume clerkship rotations was not significant for the fifth year (r= -0.298, p=0.436) and for the sixth year (r= -0.440, p=0.240). By using a cubic regression model, the time to resume clerkship rotations could explain 3.4% of the 5-year students' scores (p<0.001) and 0.9% of the 6-year students, without statistical difference (p=0.085). CONCLUSIONS: The differences between the students' scores cannot be attributed to the time when the schools paused the clerkship rotations.

RESUMEN

Resumo: Introdução: O curso de Medicina da Universidade Estadual de Campinas (Unicamp) vem pondo à prova o aquisição cumulativa cognitiva de seus alunos por meio do Teste de Progresso (TP) há mais de uma década, de modo a possibilitar a análise da utilidade do exame como estratégia de apoio a decisões pedagógicas e apontar principais ameaças à validade dele. Objetivo: Este estudo teve como objetivos oferecer a análise de validade do TP e explicitar as oportunidades de utilização do teste especialmente para a determinação de padrões de suficiência cognitiva para a progressão no curso e ao final deste, e a identificação de estudantes em risco. Método: Trata-se de estudo observacional retrospectivo de uma série histórica de sucessivos testes escritos realizados para analisar o acúmulo cognitivo no período de 2006 a 2016, totalizando 11 anos e seis turmas consecutivas. Em cada momento de medida (aplicação do teste), o estudo utilizou um modelo misto, em que a exposição (realização do teste) e o desfecho (escore do teste) foram avaliados no mesmo ponto de tempo, o que caracteriza um estudo transversal (cross-sectional) cujos resultados sucessivos originarão as curvas de crescimento cognitivo Resultado: Observou-se um acúmulo cognitivo em torno de 6 pontos percentuais a cada nova testagem. Os estudantes ao completarem o sexto ano obtiveram um acerto de cerca de 65,7% (± 9,1). A cada testagem, determinou-se um "efeito piso" para identificar alunos com rendimento abaixo da média, que em geral se situou em cerca de 1,5 DP abaixo da média da respectiva turma. Conclusão: O TP-Unicamp oferece dados confiáveis para apoiar importantes decisões pedagógicas, tais como identificação de alunos em risco acadêmico por baixa performance, critérios para progressão e desempenho cognitivo ao final do curso. Como confiabilidade sofre influência da amostragem, e o aumento do número de itens de cada teste e o aumento da frequência de testagem podem ser estratégias a serem tomadas para superar essas limitações.

Abstract: Introduction: The Medical School of the State University of Campinas (UNICAMP) has been testing the cumulative cognitive acquisition among its students through the Progress Test (PT) for over a decade, making it possible to analyze the utility of the test as a strategy to support pedagogical decisions and to point out the main threats to its validity. Objective: To provide an analysis of the validity of the PT, explaining opportunities for its use, especially in determining standards of cognitive sufficiency for progression, standards of cognitive sufficiency at the end of the course, and identification of students at risk. Method: Retrospective observational study of historical series of successive written tests performed to analyze cognitive accumulation, covering a period from 2006 to 2016, totaling 11 years and 6 consecutive classes. At each instance of measurement (test application), the study uses a mixed model where exposure (test performance) and outcome (test score) are evaluated at the same time point, characterizing a cross-sectional study, the successive results of which will generate the cognitive growth curves. Result: A cognitive accumulation of around 6 percentage points was observed with each new test. Students, upon completing the 6th year, scored around 65.7% (± 9.1). A "floor effect" was determined for each test to identify students with below-average performance, which in general was about 1.5 SD below the average of the respective class. Conclusion: TP-Unicamp offers reliable data to support important pedagogical decisions, such as the identification of students at academic risk for low performance, criteria for progression and cognitive performance at the end of the course. As reliability is influenced by sampling, increasing the number of items in each test and increasing the frequency of tests could represent strategies to overcome potential limitations.

RESUMEN

Several methods have been proposed for analyzing differences between test scores, such as using mean scores, cumulative deviation, and mixed-effect models. Here, we explore the pooled analysis of retested Progress Test items to monitor the performance of first-year medical students who were exposed to a new curriculum design. This was a cross-sectional study of students in their first year of a medical program who participated in the annual interinstitutional Progress Tests from 2013 to 2019. We analyzed the performance of first-year students in the 2019 test and compared it with that of first-year students taking the test from 2013 to 2018 and encountering the same items. For each item, we calculated odds ratios with 95% confidence intervals; we also performed meta-analyses with fixed effects for each content area in the pooled analysis and presented the odds ratio (OR) with a 95% confidence interval (CI). In all, we used 63 items, which were divided into basic sciences, internal medicine, pediatrics, surgery, obstetrics and gynecology, and public health. Significant differences were found between groups in basic sciences (OR = 1.172 [CI95% 1.005 CI 1.366], p = 0.043) and public health (OR = 1.54 [CI95% CI 1.25-1.897], p < 0.001), which may reflect the characteristics of the new curriculum. Thus, pooled analysis of pretested items may provide indicators of different performance. This method may complement analysis of score differences on benchmark assessments.

Asunto(s)

RESUMEN

ABSTRACT: Progress test has been created with the necessity of an assessment method aligned with problem-based learning. Although it was specifically created to overcome the limitations of traditional assessment for problem-based learning, nowadays is used by different types of curricula. In this paper, we first present the basic assumptions, history, benefit and progress test challenges. Progress test overcomes many limitations of traditional assessment, such as validity and reliability. However, the implementation of a progress test is a logistical challenge. In addition, we discuss the limitation of progress tests when used as a summative assessment, which may not always be aligned with constructivist theory. When adding feedback and methods of analysis that consider multiple testing, progress test is then aligned with constructivist theory. Finally, the use of the progress test's sub scores may lack validity because of the low number of items; thus, pass/fail decision should not be based on the sub scores, but only on general scores. (AU)

RESUMO: O teste de progresso foi criado como um método de avalição alinhado a aprendizagem baseada em problemas. Apesar do teste de progresso ser criado especificamente para superar limitações da avaliação tradicional em currículos de aprendizagem baseada em problemas, hoje em dia, ele é utilizado em diferentes tipos de currículos. Nesse artigo, primeiro, apresentamos as premissas básicas, história, benefício e desafios do teste de progresso. O teste de progresso superou muitas limitações da avaliação tradicional, como os problemas de validade e confiabilidade. No entanto, a implementação do teste de progresso apresenta grandes desafios logísticos. Ademais, discutimos as limitações do teste de progresso quando utilizada de forma somativa, sendo que nem sempre é alinhada a teoria construtivista. Quando adicionado o feedback e métodos de análises que consideram múltiplos testes, o teste de progresso então se alinha a teoria construtivista. Finalmente, o uso das subcategorias do teste de progresso pode apresentar problemas de validade por causa do baixo número de item e consequentemente, decisões de aprovar ou reprovar não poderiam ser baseadas nas subcategorias, mas apenas na categoria geral (AU)

Asunto(s)

RESUMEN

Abstract: Introduction: during the COVID-19 pandemic, the universities suspended in-person activities and medical education adapted from the traditional format to virtual scenarios. Thus, the volunteering activities might improve the apprenticeship in critical situations and constitute a way to obtain competences and clinical skills. Objective: to identify the frequency of student volunteering during the COVID-19 pandemic and the associated factors. Methods: a cross-sectional study was performed targeting medical students in their last 3 years at a medical school in Brazil. Three hundred and fifty invitations were sent by email to ask students to participate in a self-administered, anonymous electronic survey during the pandemic. The primary outcome was the frequency of volunteer activities. Sociodemographic variables and characteristics of the medical students' activities were the dependent variables for the multivariate analysis that calculated the factors associated with volunteering. Results: One hundred and twenty-five respondents (35.8% response rate) were included in the analysis (no missing data). The frequency of volunteering was 52% and most of the participants were females (63.2%) and all had access to online activities. Telemedicine was the most frequent activity (56/65). After the multivariate analysis, it was found that a family income of 5.1-10 minimal wages (OR=2.32[0.94-6.42]), expressing the ability and confidence in a pandemic situation (OR=4.91[1.49-16.2]) and considering e-learning important before the pandemic (OR=16.46[1.35-200.32]) and exposure of more than 120 minutes to social media platforms were less motivating for volunteering. Conclusion: About half of the medical students volunteered during the COVID-19 pandemic. The presence of self-confidence, with previous training in a pandemic situation motivated the students to volunteer.

Resumo: Introdução: Durante a pandemia da Covid-19, as universidades suspenderam as atividades presenciais e o ensino médico adaptou-se do formato tradicional para os cenários virtuais. Assim, as atividades de voluntariado podem melhorar a aprendizagem em situações críticas e são uma forma de obter competências e habilidades clínicas. Objetivo: Este estudo teve como objetivo identificar a frequência do voluntariado nos estudantes durante a pandemia da Covid-19 e seus fatores associados. Método: Realizou-se um estudo transversal com estudantes de Medicina dos três anos finais de uma faculdade no Brasil. Trezentos e cinquenta alunos foram convidados via e-mail para uma pesquisa eletrônica anônima e autoaplicável, em meio à pandemia. O resultado primário foi a frequência das atividades voluntárias. Variáveis sociodemográficas e características da atividade dos estudantes foram as variáveis dependentes para análise multivariada dos fatores associados ao voluntariado. Resultado: Incluíram-se 125 alunos (taxa de resposta de 35,8%) na análise (sem dados ausentes). A frequência de voluntariado foi de 52%, a maioria dos participantes era do sexo feminino (63,2%), e todos tinham acesso a atividades on-line. A telemedicina foi a atividade mais frequente (56/65). Em análise multivariada, constatou-se que os seguintes fatores foram menos motivadores para o voluntariado: renda familiar entre 5,1 e dez salários mínimos (OR = 2,32 [0,94-6,42]), capacidade e confiança em atuar numa situação de pandemia (OR = 4,91 [1,49-16,2]), os alunos que consideraram importante uso do e-learning antes da pandemia (OR = 16,46 [1,35-200,32]) e exposição por mais de 120 minutos a plataformas de mídia social. Conclusão: Cerca de metade dos estudantes de Medicina foram voluntários durante a pandemia da Covid-10. A presença de autoconfiança com aconselhamento prévio em uma situação de pandemia motivou o aluno a se voluntariar.

Asunto(s)

RESUMEN

BACKGROUND: Progress tests are longitudinal assessments of students' knowledge based on successive tests. Calibration of the test difficulty is challenging, especially because of the tendency of item-writers to overestimate students' performance. The relationships between the levels of Bloom's taxonomy, the ability of test judges to predict the difficulty of test items and the real psychometric properties of test items have been insufficiently studied. OBJECTIVE: To investigate the psychometric properties of items according to their classification in Bloom's taxonomy and judges' estimates, through an adaptation of the Angoff method. DESIGN AND SETTING: Prospective observational study using secondary data from students' performance in a progress test applied to ten medical schools, mainly in the state of São Paulo, Brazil. METHODS: We compared the expected and real difficulty of items used in a progress test. The items were classified according to Bloom's taxonomy. Psychometric properties were assessed based on their taxonomy and fields of knowledge. RESULTS: There was a 54% match between the panel of experts' expectations and the real difficulty of items. Items that were expected to be easy had mean difficulty that was significantly lower than that of items that were expected to be medium (P < 0.05) or difficult (P < 0.01). Items with high-level taxonomy had higher discrimination indices than low-level items (P = 0.026). We did not find any significant differences between the fields in terms of difficulty and discrimination. CONCLUSIONS: Our study demonstrated that items with high-level taxonomy performed better in discrimination indices and that a panel of experts may develop coherent reasoning regarding the difficulty of items.

Asunto(s)

RESUMEN

ABSTRACT BACKGROUND: Progress tests are longitudinal assessments of students' knowledge based on successive tests. Calibration of the test difficulty is challenging, especially because of the tendency of item-writers to overestimate students' performance. The relationships between the levels of Bloom's taxonomy, the ability of test judges to predict the difficulty of test items and the real psychometric properties of test items have been insufficiently studied. OBJECTIVE: To investigate the psychometric properties of items according to their classification in Bloom's taxonomy and judges' estimates, through an adaptation of the Angoff method. DESIGN AND SETTING: Prospective observational study using secondary data from students' performance in a progress test applied to ten medical schools, mainly in the state of São Paulo, Brazil. METHODS: We compared the expected and real difficulty of items used in a progress test. The items were classified according to Bloom's taxonomy. Psychometric properties were assessed based on their taxonomy and fields of knowledge. RESULTS: There was a 54% match between the panel of experts' expectations and the real difficulty of items. Items that were expected to be easy had mean difficulty that was significantly lower than that of items that were expected to be medium (P < 0.05) or difficult (P < 0.01). Items with high-level taxonomy had higher discrimination indices than low-level items (P = 0.026). We did not find any significant differences between the fields in terms of difficulty and discrimination. CONCLUSIONS: Our study demonstrated that items with high-level taxonomy performed better in discrimination indices and that a panel of experts may develop coherent reasoning regarding the difficulty of items.

Asunto(s)

RESUMEN

Abstract: The evaluation of knowledge is an important step in the teachers' work, because it is fundamental that the evaluation results be valid and reliable. Several faculty development programs have been undertaken to improve teachers' skills on item writing. Few studies, however, have explored the benefits of providing feedback to multiple-choice item writers. Progress Testing is a longitudinal assessment of students' knowledge gain. Their results are useful for generating feedback to students, teachers and medical school managers. The aim of this report is to present a successful experience of faculty development on high-quality item writing for Progress Testing through feedback about the fate of the items written by faculty members. Annually, faculties from medical schools participating in a consortium for the application of the Progress Test are asked to formulate new items to compose the test version. However, the fate of the items is not known by their authors, that is, they do not know if the items are incorporated into the test, if the items have good quality, what is the students' performance in each of the items, and what is the psychometric function of the items. In 2017, one of the schools participating in the consortium offered its item authors a feedback regarding the flaws in item writing, modification of items by the review committee and students' performance, as well as psychometric function in each item. The percentage of flawed item (items without enough quality to be selected for the test) was more than 30%. There was an inverse relation between the number of flawed item and the number of items included in the final exam, when analyzing the different content areas of the test (that is, Internal Medicine, Surgery, Pediatrics, Obstetrics and Gynecology, Public Health, and Basic Sciences). In the following year, we observed a decrease in the number of flawed item (less than 10%) and an increase in the number of eligible items to be selected for the Progress Testing. Therefore, giving feedback to item writers seems to be a good strategy for developing the faculty's ability on writing good quality items.

Resumo: A avaliação de conhecimentos é uma etapa importante do trabalho docente, pois é preciso que os resultados das avaliações sejam válidos e confiáveis. Diversos programas de desenvolvimento docente têm sido realizados para melhorar a habilidade em elaborar questões objetivas pelos professores. Poucos estudos, no entanto, exploraram os benefícios de fornecer feedback aos autores de itens de múltipla escolha. O Teste de Progresso é uma avaliação longitudinal do ganho de conhecimentos dos estudantes cujos resultados são úteis para gerar feedback aos próprios estudantes, aos professores e aos gestores das escolas médicas. O objetivo deste relato é apresentar uma experiência exitosa em gerar desenvolvimento docente na habilidade de elaborar itens de qualidade para o Teste de Progresso por meio do feedback a respeito da qualidade e do destino destes. Anualmente, os professores das escolas médicas participantes de um consórcio para aplicação do Teste de Progresso são solicitados a formular novos itens para compor a versão do teste. Porém, o destino dos itens não é conhecido por seus autores, ou seja, eles não sabem se as questões são incorporadas à prova, se elas têm boa qualidade, qual o desempenho dos estudantes em cada questão e qual o funcionamento psicométrico delas. Em 2017, uma das escolas participantes do consórcio ofereceu aos seus autores de questões uma devolutiva sobre as falhas na redação, a modificação dos itens pelo comitê revisor, o desempenho dos estudantes nas questões e o funcionamento psicométrico em cada item. A porcentagem de itens falhos, ou seja, de qualidade não satisfatória para a inclusão no teste, era superior a 30%. Houve uma relação inversa entre a quantidade de itens falhos e o número de itens incluídos na prova final, numa análise por área de conhecimento da prova (ou seja, clínica, cirurgia, pediatria, ginecologia e obstetrícia, saúde pública e ciências básicas). No ano seguinte, observamos uma diminuição no número de falhas de itens (menor que 10%) e um aumento no número de questões elegíveis a serem selecionadas para o exame de Teste de Progresso. Portanto, oferecer feedback para os redatores de questões parece ser uma boa estratégia para desenvolver a habilidade docente em elaborar itens objetivos de boa qualidade.

RESUMEN

Abstract: Introduction: Assessment is a critical part of learning and validity is arguably its most important aspect. However, different views and beliefs led to a fragmented conception of the validity meaning, with an excessive focus on psychometric methods and scores, neglecting the consequences and utility of the test. The last decades witnessed the creation of a significant number of tests to assess different aspects of the medical profession formation, but researchers frequently limit their conclusions to the consistency of their measurements, without any further analysis on the educational and social impacts of the test. The objective of this work is to determine the predominant concept of validity in medical education assessment studies. Method: The authors conducted a bibliometric research of the literature about studies on the assessment of learning of medical students, to determine the prevalent concept of validity. The research covered a period from January 2001 to august 2019. The studies were classified in two categories based on their approach to validity: (1)" fragmented validity concept" and (2)" unified validity concept". To help with validity arguments, the studies were also classified based on Miller's framework for clinical assessment. Results: From an initial search resulting in 2823 studies, 716 studies were selected based on the eligibility criteria, and from the selected list, of which 693 (96,7%) were considered studies of the fragmented validity concept, which prioritized score results over an analysis of the test's utility, and only 23 studies (3,2%) were aligned with a unified view of validity, showing an explicit analysis of the consequences and utility of the test. Although the last decade witnessed a significant increase in the number of assessment studies, this increase was not followed by a significant change in the validity concept. Conclusions: This bibliometric analysis demonstrated that assessment studies in medical education still have a fragmented concept of validity, restricted to psychometric methods and scores. The vast majority of studies are not committed to the analysis about the utility and educational impact of an assessment policy. This restrictive view can lead to the waste of valuable time and resources related to assessment methods without significant educational consequences.

Resumo: Introdução: Avaliação é uma parte crítica da aprendizagem, e validade é sem dúvida seu aspecto mais importante. No entanto, diferentes visões e crenças levaram a uma concepção fragmentada do significado de validade, com um foco excessivo nos métodos psicométricos e escores, negligenciando a utilidade do teste. As últimas décadas testemunharam a criação de um número significativo de testes para avaliar diferentes aspectos da formação da profissão médica, mas os pesquisadores frequentemente limitam suas conclusões à consistência de suas medidas, sem nenhuma análise adicional sobre os impactos educacionais e sociais do teste. O objetivo deste trabalho é determinar o conceito predominante de validade nos estudos de avaliação em educação médica. Método: Foi realizada uma pesquisa bibliométrica da literatura de estudos sobre avaliação da aprendizagem de estudantes de Medicina para determinar o conceito prevalente de validade. A pesquisa abrangeu o período de janeiro de 2001 a agosto de 2019. Os estudos foram classificados em duas categorias: 1. "conceito de validade fragmentada" e 2. "conceito de validade unificada". Para ajudar nos argumentos de validade, os estudos também foram classificados com base na estrutura de Miller para avaliação clínica. Resultados: A partir de uma pesquisa inicial que resultou em 2.823 estudos, selecionaram-se 716 com base nos critérios de elegibilidade, e consideraram-se 693 (96,7%) estudos com conceito fragmentado de validade que priorizavam os resultados dos escores em detrimento de uma análise da utilidade do teste, e apenas 23 (3,2%) foram alinhados com uma visão unificada de validade, apresentando uma análise explícita das consequências e da utilidade do teste. Embora a última década tenha testemunhado um aumento expressivo de estudos sobre avaliação, esse crescimento não foi acompanhado por uma mudança significativa do conceito de validade. Conclusões: Esta análise bibliométrica demonstrou que os estudos sobre avaliação de aprendizagem em educação médica têm um conceito fragmentado de validade, limitados aos métodos psicométricos e escores. A grande maioria dos trabalhos não está comprometida com uma análise sobre a utilidade e o impacto educacional de uma política de avaliação. Essa visão restritiva pode levar à perda de tempo e recursos valiosos com métodos de avaliação sem consequências educacionais significativas.

RESUMEN

RESUMO O Teste de Progresso é uma ferramenta de avaliação longitudinal do ganho de conhecimento de estudantes que tem sido aplicada no Brasil há mais de dez anos. O teste situa o estudante em seu processo evolutivo de ensino-aprendizagem e permite à instituição realizar o diagnóstico de suas deficiências ao longo da estrutura curricular. Ele pode ser utilizado pelos colegiados competentes para avaliação de alterações curriculares e avaliações específicas de disciplinas ou módulos de ensino. Com base na experiência de um consórcio de escolas, a Associação Brasileira de Educação Médica (Abem) propôs um projeto que tinha como um de seus objetivos incentivar escolas de todo o País a adotarem o Teste de Progresso como uma de suas ferramentas de avaliação. Reportamos a estratégia adotada para constituir núcleos interinstitucionais de avaliação com Teste de Progresso, bem como os resultados da primeira prova nacional do Teste de Progresso, que contou com a participação de 58 escolas e 23.065 estudantes. A implantação de núcleos interinstitucionais de Teste do Progresso com processos colaborativos de realização da prova representou um avanço para as escolas envolvidas. As escolas iniciaram um processo de colaboração não apenas para o Teste de Progresso, mas também para o intercâmbio de informações e experiências que trocam com base no conhecimento de cada uma. O projeto funcionou como o início de um movimento para que escolas médicas de todas as regiões do País adotem o Teste de Progresso como uma ferramenta de avaliação com potencial para reorientar a formação médica, ao fornecer um diagnóstico de formação em nível individual e institucional.

ABSTRACT Progress Testing has been applied in Brazil for more than ten years. It is a tool for longitudinal evaluation of students' knowledge across levels of training. The test situates the student in his evolutionary teaching-learning process and allows the institution to diagnose its deficiencies throughout the curricular structure. It can be used to evaluate curricular changes and specific assessments of disciplines or teaching modules. From the experience of a consortium of schools, the Brazilian Association of Medical Education (ABEM) proposed a project that aimed to encourage schools throughout the country to adopt Progress Testing as one of its evaluation tools. We report on the strategy adopted to establish interinstitutional evaluation nuclei with Progress Testing, as well as the results of the first national exam, administered by 58 schools to 23,065 students. The implantation of cross-institutional Progress Testing triggered collaborative processes among the schools involved. Schools have begun a collaborative process not only for the Progress Testing, but also for the exchange of information. The project served as the beginning of a movement for medical schools in all regions of the country to adopt Progress Testing as an assessment tool with the potential to reorient medical training by providing information regarding institutional and individual performances.