RESUMEN

Objective Structured Clinical Examinations (OSCEs) and Work Based Assessments (WBAs) are the mainstays of assessing clinical competency in health professions' education. Underpinned by the extrapolation inference in Kane's Validity Framework, the purpose of this study is to determine whether OSCEs translate to real life performance by comparing students' OSCE performance to their performance in real-life (as a WBA) using the same clinical scenario, and to understand factors that affect students' performance. A sequential explanatory mixed methods approach where a grade comparison between students' performance in their OSCE and WBA was performed. Students were third year pharmacy undergraduates on placement at a community pharmacy in 2022. The WBA was conducted by a simulated patient, unbeknownst to students and indistinguishable from a genuine patient, visiting the pharmacy asking for health advice. The simulated patient was referred to as a 'mystery shopper' and the process to 'mystery shopping' in this manuscript. Community pharmacy is an ideal setting for real-time observation and mystery shopping as staff can be accessed without appointment. The students' provision of care and clinical knowledge was assessed by the mystery shopper using the same clinical checklist the student was assessed from in the OSCE. Students who had the WBA conducted were then invited to participate in semi-structured interviews to discuss their experiences in both settings. Overall, 92 mystery shopper (WBA) visits with students were conducted and 36 follow-up interviews were completed. The median WBA score was 41.7% [IQR 28.3] and significantly lower compared to the OSCE score 80.9% [IQR 19.0] in all participants (p < 0.001). Interviews revealed students knew they did not perform as well in the WBA compared to their OSCE, but reflected that they still need OSCEs to prepare them to manage real-life patients. Many students related their performance to how they perceived their role in OSCEs versus WBAs, and that OSCEs allowed them more autonomy to manage the patient as opposed to an unfamiliar workplace. As suggested by the activity theory, the performance of the student can be driven by their motivation which differed in the two contexts.

RESUMEN

In 2014, the Association of American Medical Colleges (AAMC) published 13 Core Entrustable Professional Activities (EPAs) that graduating students should be able to perform with indirect supervision when entering residency. A ten-school multi-year pilot was commissioned to test feasibility of implementing training and assessment of the AAMC's 13 Core EPAs. In 2020-21, a case study was employed to describe pilot schools' implementation experiences. Teams from nine of ten schools were interviewed to identify means and contexts of implementing EPAs and lessons learned. Audiotapes were transcribed then coded by investigators using conventional content analysis and a constant comparative method. Coded passages were organized in a database and analyzed for themes. Consensus among school teams regarding facilitators of EPA implementation included team commitment to piloting EPAs; agreement that: proximal EPA adoption with curriculum reform facilitates EPA implementation; EPAs 'naturally fit' in clerkships and provided opportunity for schools to reflect on and adjust curricula and assessments; and inter-school collaboration bolstered individual school progress. Schools did not make high-stakes decisions about student progress (e.g., promotion, graduation), yet EPA assessment results complemented other forms of assessment in providing students with robust formative feedback about their progress. Teams had varied perceptions of school capability to implement an EPA framework, influenced by various levels of dean involvement, willingness, and capability of schools to invest in data systems and provide other resources, strategic deployment of EPAs and assessments, and faculty buy-in. These factors affected varied pace of implementation. Teams agreed on the worthiness of piloting the Core EPAs, but substantial work is still needed to fully employ an EPA framework at the scale of entire classes of students with enough assessments per EPA and with required data validity/reliability. Recommendations stemming from findings may help inform further implementation efforts across other schools adopting or considering an EPA framework.

Asunto(s)

Asunto(s)

RESUMEN

BACKGROUND: Work-based assessments including procedure based assessments, case based discussions, clinical evaluation exercises and direct observation of procedural skills are used in Higher Surgical Training Program. This systematic review aims to investigate the trainer and trainee's perception of the usefulness of workplace based assessments in Higher Surgical Training Programme. MATERIALS AND METHODS: Embase, MEDLINE and PubMed databases were searched for relevant studies published up to Jan 15, 2021. The following search terms were used: procedure based assessments, case based discussions, clinical evaluation exercises, direct observation of procedural skills, Higher Surgical Training Program (with and without their abbreviations), surgical training, formative assessment. Usefulness was analysed according to van der Vleuten's utility formula, which is the product of educational impact, validity, reliability, acceptability, cost-effectiveness and feasibility. RESULTS: 23 studies were included; Six on procedure based assessments, two on case based discussions, one on mini clinical evaluation exercises, three on direct observation of procedural skills and eleven on multiple methods. Overall, procedure based assessments had positive Kirkpatrick level 1 or 2 impact were valid, reliable and acceptable. Mini clinical evaluation exercises showed positive Kirkpatrick level 1 satisfaction in trainees and trainers and were feasible. Case based discussions were shown to have both positive and negative Kirkpatrick levels 1 and 2 impact and were rated valid and reliable. Direct observation of procedural skills were valid with favourable Kirkpatrick level 1 impact with more usage and modification of the forms. Multiple methods as used in the intercollegiate surgical curriculum programme portfolio was found to be less encouraging in achieving positive higher level of educational impact. The barriers included lack of time, lack of faculty development and lack of engagement. CONCLUSION: It is important that work-based assessments are used properly, so that trainees get the maximum benefit from them. None of the studies included in our systematic review demonstrated an impact on behaviour or society (Kirkpatrick level 3 and 4), towards which future research should be directed to.

Asunto(s)

RESUMEN

The UK cardiology specialist training programme utilises the National Health Service (NHS) e-Portfolio to ensure adequate progression is being made during a trainees' career. The NHS e-portfolio has been used for 15 years, but many questions remain regarding its perceived learning value and usefulness for trainees and trainers. This qualitative study in the recent pre-COVID era explored the perceived benefits of the NHS e-Portfolio with cardiology trainees and trainers in two UK training deaneries. Questionnaires were sent to 66 trainees and to 50 trainers. 50% of trainees felt that their development had benefited from use of the ePortfolio. 61% of trainees found it an effective educational tool, and 25% of trainees and 39% of trainers found the ePortfolio useful for highlighting their strengths and weaknesses. 75% of trainees viewed workplace based assessments as a means to passing the ARCP. The results show that the NHS ePortfolio and workplace based assessments were perceived negatively by some trainees and trainers alike, with many feeling that significant improvements need to be made. In light of the progress and acceptance of digital technology and communication in the current COVID-19 era, it is likely to be the time for the development of a new optimal digital training platform for cardiology trainees and trainers. The specialist societies could help develop a more speciality specific learning and development tool.

RESUMEN

Introduction In the United Kingdom, work-based assessments (WBAs) including procedure-based assessments (PBAs), case-based discussions (CBDs), clinical evaluation exercises (CEXs), and direct observation of procedural skills (DOPS) have been used in Higher General Surgical Training Program (HGSTP) since the introduction of Modernising Medical Careers. Although the Intercollegiate Surgical Curriculum Project states that they should be used for the formative development of trainees using feedback and reflection, there is no study to look at the perception of their usefulness and barriers in using them, particularly in HGSTP. The aim of this study is to investigate trainer's and trainee's perception of their usefulness, barriers in using them, and way forward for their improvement in HGSTP. Methods This was a mixed method study. In phase I, after ethics committee approval, an online survey was sent to 83 trainers and 104 trainees, with a response rate of 33 and 37%, respectively, using Online Surveys (formerly Bristol Online Survey) from July 2018 to December 2018. After analysis of this result, in phase II, semistructured interviews were conducted with five trainees and five trainers who had volunteered to take part from phase I. Thematic analysis was performed to develop overarching themes. Results For professional formative development, 15% of the trainers and 53% of the trainees felt that WBAs had a low value. Among 4 WBAs-CEX, CBD, PBA, and DOPS-PBA was thought to be the most useful WBA by 52% trainers and 74% trainees. More trainers than trainees felt that it was being used as a formative tool (33 vs. 16%). The total number of WBAs thought to be required was between 20 and 40 per year, with 46% of the trainers and 53% of the trainees preferring these numbers. The thematic analysis generated four themes with subthemes in each: theme 1, "factors affecting usefulness," including the mode of validation, trainer/trainee engagement, and time spent in validating; theme 2, "doubt on utility" due to doubt on validity and being used as a tick-box exercise; theme 3, "pitfalls/difficulties" due to lack of time to validate, late validation, e-mail rather than face-to-face validation, trainer and trainee behavior, variability in feedback given, and emphasis on number than quality; and theme 4, "improvement strategies." Conclusions The WBAs are not being used in a way they are supposed to be used. The perception of educational impact (Kirkpatrick levels 1 and 2) by trainers was more optimistic than by trainees. Improvements can be made by giving/finding more time, trainer training, more face-to-face validation, and better trainer trainee interactions.

RESUMEN

Introduction Pan-speciality consensus guidance advocates mandatory emergency general surgery (EGS) training modules for specialist registrars (StRs). This pilot study evaluated the impact of EGS modules aimed at StRs over 1 year. Methods Eleven StRs were allocated a focused 4-week EGS module, in addition to the standard 1:12 on-call duty rota, in a tertiary surgical centre. Primary outcome measures included the number of indicative emergency operations and validated Procedure Based Assessments (PBAs) performed, both during the EGS module and over the training year. Results StRs performed a median of 11 (range 5-15) laparotomies during the EGS module versus 31 (range 9-49) over the whole training year. StRs attended 43.7% of available laparotomies during the module (range 24.1-63.7%). EGS modules provided more than one-third of the total emergency laparotomy experience, and a quarter of the emergency colectomy, appendicectomy and Hartmann's procedure experience. There were no differences in EGS module-related outcomes between junior and senior StRs. Significantly more PBAs related to laparotomy and segmental colectomy were completed during EGS modules than the on-call duty rota, at 32% versus 14% (p<0.001) and 48% versus 22% (p=0.019), respectively. Performance levels were maintained following module completion. Conclusions These findings provide an important baseline when considering future modular EGS training.

Asunto(s)

RESUMEN

Work-based assessments (WBAs) were only recently introduced into medicine. However, since their introduction, they have rapidly grown in popularity. WBAs are now a routine part of medical training. As WBAs are being implemented, the practical difficulties with their use have come to light. A major problem is failure of the trainees, trainers and training programmes to adequately engage with them. In this review, the reasons for this and how these can be overcome are discussed.

RESUMEN

The Specialty Certification Examination is an important part of the assessment of medical education. However, the step 2 skill examinations of 26 medical specialties in Korea are insufficient with respect to achieving the objective of practical examinations that evaluate clinical skill and competence. Among the current step 2 skill examination methods, picture testing using slides or reading of pathology slides/radiologic images is more suitable for testing cognition and knowledge than for testing performance. The oral examination has low reliability because of its relatively short testing period and absence of scoring criteria. In addition, the Specialty Certification Examination is a high-stakes test and the performance during the training course is not reflected in the skill examination. We have reviewed the various skill examinations including clinical practice examinations, objective structured clinical examinations of the United States and Canada, and work-based assessments of the United Kingdom. Based on the review, we suggest some plans for improving the Korean Specialty Certification Examination.

Asunto(s)

RESUMEN

BACKGROUND: The increasing need for doctors to be accountable and an emphasis on competency have led to the evolution of medical curricula. The Intercollegiate Surgical Curriculum Project succeeded the Calman curriculum for surgical training in 2007 in the UK. It sought to provide an integrated curriculum based upon a website platform. The aim of this review is to examine the changes to the curriculum and effect on surgical training. METHODS: A comparison was made of the Calman Curriculum and the ISCP and how they met training needs. RESULTS: The new curriculum is multifaceted, providing a more prescriptive detail on what trainees should achieve and when, as well as allowing portfolio, learning agreements, and work-based assessments to be maintained on an easily accessed website. The increasing emphasis on work-based assessments has been one of the major components, with an aim of providing evidence of competence. However, there is dissatisfaction amongst trainees with this component which lacks convincing validity. CONCLUSION: This new curriculum significantly differs from its predecessor which was essentially just a syllabus. It needs to continuously evolve to meet the needs of trainees whose training environment is ever changing.

Asunto(s)