RESUMEN

We previously developed a computer-assisted image analysis algorithm to detect and quantify the microscopic features of rodent progressive cardiomyopathy (PCM) in rat heart histologic sections and validated the results with a panel of five veterinary toxicologic pathologists using a multinomial logistic model. In this study, we assessed both the inter-rater and intra-rater agreement of the pathologists and compared pathologists' ratings to the artificial intelligence (AI)-predicted scores. Pathologists and the AI algorithm were presented with 500 slides of rodent heart. They quantified the amount of cardiomyopathy in each slide. A total of 200 of these slides were novel to this study, whereas 100 slides were intentionally selected for repetition from the previous study. After a washout period of more than six months, the repeated slides were examined to assess intra-rater agreement among pathologists. We found the intra-rater agreement to be substantial, with weighted Cohen's kappa values ranging from k = 0.64 to 0.80. Intra-rater variability is not a concern for the deterministic AI. The inter-rater agreement across pathologists was moderate (Cohen's kappa k = 0.56). These results demonstrate the utility of AI algorithms as a tool for pathologists to increase sensitivity and specificity for the histopathologic assessment of the heart in toxicology studies.

Asunto(s)

RESUMEN

PURPOSE: To compare the patient-derived modified Japanese Orthopaedic Association (P-mJOA) scale with the European myelopathy score (EMS) for the assessment of patients with degenerative cervical myelopathy (DCM). METHODS: In this register-based cohort study with prospectively collected data, included patients were surgically treated for DCM and had reported both P-mJOA and EMS scores at baseline, 1-year follow-up, and/or 2-year follow-up to the Swedish Spine Register. P-mJOA and EMS scores were defined as severe (P-mJOA 0-11 and EMS 5-8), moderate (P-mJOA 12-14 and EMS 9-12), or mild (P-mJOA 15-18 and EMS 13-18). P-mJOA and EMS mean scores were compared, and agreement was evaluated with Spearman's rank correlation coefficient (ρ), the intraclass correlation coefficient (ICC), and kappa (κ) statistics. RESULTS: Included patients (n = 714, mean age 63.2 years, 42.2% female) completed 937 pairs of the P-mJOA and the EMS. The mean P-mJOA and EMS scores were 13.9 ± 3.0 and 14.5 ± 2.7, respectively (mean difference -0.61 [95% CI -0.72 to -0.51; p < 0.001]). Spearman's ρ was 0.84 (p < 0.001), and intra-rater agreement measured with ICC was 0.83 (p < 0.001). Agreement of severity level measured with unweighted and weighted κ was fair (κ = 0.22 [p < 0.001]; κ = 0.34 [p < 0.001], respectively). Severity levels were significantly higher using the P-mJOA (p < 0.001). CONCLUSION: The P-mJOA and the EMS had similar mean scores, and intra-rater agreement was high, whereas severity levels only demonstrated fair agreement. The EMS has a lower sensitivity for detecting severe myelopathy but shows an increasing agreement with the P-mJOA for milder disease severity. A larger interval to define severe myelopathy with the EMS is recommended.

Asunto(s)

RESUMEN

BACKGROUND: The treatment approach for proximal humeral fractures is determined by various factors, including patient age, sex, dominant arm, fracture pattern, presence of osteoporosis, preexisting arthritis, rotator cuff status, and medical comorbidities. However, there is a lack of consensus in the literature regarding the optimal treatment for displaced proximal humeral fractures. This study aimed to assess and quantify the decision-making process for either conservative or surgical treatment and the choice of surgical method among shoulder surgeons when treating proximal humeral fractures. METHODS: Forty sets of true anteroposterior view, scapular Y projection view, and three-dimensional computed tomography of proximal humeral fractures were provided to 12 shoulder surgeons along with clinical information. Surveys regarding Neer classification, decisions between conservative and surgical treatments, and chosen methods were conducted twice with an interval of 2 months. The factors affecting the treatment plans were also assessed. RESULTS: The inter-rater agreement was fair for Neer classification (kappa=0.395), moderate for the decision between conservative and surgical treatments (kappa=0.528), and substantial for the chosen method of surgical treatment (kappa=0.740). The percentage of agreement was 71.1% for Neer classification, 84.6% for the decision between conservative and surgical treatment, and 96.4% for the chosen method of surgical treatment. The fracture pattern was the most crucial factor in deciding between conservative and surgical treatments, followed by age and physical activity. CONCLUSIONS: The decision between conservative and surgical treatment for proximal humeral fractures showed good agreement, while the chosen method between osteosynthesis and arthroplasty showed substantial agreement among shoulder surgeons.

RESUMEN

Advances in breast imaging and other screening tests have prompted studies to evaluate and compare the consistency between experts' ratings of existing with new screening tests. In clinical settings, medical experts make subjective assessments of screening test results such as mammograms. Consistency between experts' ratings is evaluated by measures of inter-rater agreement or association. However, conventional measures, such as Cohen's and Fleiss' kappas, are unable to be applied or may perform poorly when studies consist of many experts, unbalanced data, or dependencies between experts' ratings exist. Here we assess the performance of existing approaches including recently developed summary measures for assessing the agreement between experts' binary and ordinal ratings when patients undergo two screening procedures. Methods to assess consistency between repeated measurements by the same experts are also described. We present applications to three large-scale clinical screening studies. Properties of these agreement measures are illustrated via simulation studies. Generally, a model-based approach provides several advantages over alternative methods including the ability to flexibly incorporate various measurement scales (i.e. binary or ordinal), large numbers of experts and patients, sparse data, and robustness to prevalence of underlying disease.

RESUMEN

Clinical forensic assessments of injuries' life-threatening danger may have an impact on the legal aftermath following a violent assault. The pursuit of evidence-based guidelines should ensure a user-independent and reproducible forensic practice. However, does it? The aim of this study was to evaluate the forensic life-threatening danger assessments after a protocol implementation in 2016. The evaluation concerned usability and reproducibility of the protocol, and its influence on assessment severity. We analyzed the level of inter- and intra-rater agreement using 169 blinded, prior-protocol cases that were reassessed by two forensic specialists. We compared assessment made the year before and after protocol implementation (n = 262), and the forensic specialists' reassessments with the prior-protocol cases' original assessments (n = 169). Whether to make an assessment, the levels of agreement varied between weak agreement (inter-rater, Κ = 0.43; assessor 1, Κ = 0.57) and strong agreement (assessor 2, Κ = 0.90). Regarding severity, the levels of agreement varied between strong agreement (inter-rater, Κ = 0.87; assessor 1: Κ = 0.90) and almost perfect agreement (assessor 2: Κ = 0.94). The assessments were statistically significant redistributed after the implementation (chi-square test: p < 0.0001). The proportion of cases assessed as having not been in life-threatening danger increased from 9 to 43%, and moderate severity assessments decreased from 55 to 23%. Of the moderate severity assessments, 55% were reassessed as having not been in life-threatening danger. The protocol ensured independent and reproducible assessments when the forensic specialists agreed on making one. The protocol resulted in less severe assessments. Future studies should examine the reliability of the protocol and its consequences for legal aftermaths.

Asunto(s)

RESUMEN

Purpose: Diabetic retinopathy (DR) is a leading cause of blindness in low- and middle-income countries. This study aims to assess the agreement and diagnostic test accuracy of DR grading using fundus photographs by the allied medical personnel (AMP) as compared to ophthalmologist at DR screening program.Methods: Two 45 degree disc-centred and macula centred fundus photographs captured using digital fundus camera from 1344 eyes were graded by six AMP twice at three months and six months of training. DR grading was done using Early Treatment Diabetic Retinopathy Study criteria. Retina specialist grading was taken as the reference standard. Percentage agreement using kappa coefficient (k) and diagnostic test accuracy were assessed.Results: Intra-rater agreement of most AMP was almost perfect for detection of any DR and vision-threatening diabetic retinopathy (VTDR) at three and six months. Inter-rater agreement at 3 months at second grading was substantial for any DR (k = 0.62; 95% confidence interval (CI): 0.67-0.68) and VTDR (k = 0.63; 95% CI: 0.47-0.75). Inter-rater agreement in 6 months at second grading was also substantial for any DR (k = 0.66; 95% CI: 0.61-0.71) and VTDR (k = 0.64; 95% CI: 0.52-0.75). The sensitivity and specificity for any DR at second grading in 3 months was 94.99% and 63.97% and at 6 months was 95.55% and 66.91%, respectively.Conclusion: The inter-rater agreement of AMP as compared to ophthalmologists for any DR or VTDR was substantial at both 3 and 6 months. AMP can support as first-level DR graders for timely referral and treatment of VTDR.

Asunto(s)

RESUMEN

The Progression of Autonomies Scale (PAS) is a behavioral scale useful to assess the autonomy levels in acquired brain-injured patients. It provides a broad profile, assessing different domains of human activities ranging from personal, domestic, and extradomestic autonomies. This cross-sectional study is aimed at evaluating the reliability of this scale on a large cohort of acquired brain injury (ABI) patients. Fifty-one ABI patients (49% traumatic, 33.3% hemorrhagic, 17.7% other etiologies), hospitalized in the S. Anna Institute of Crotone, Italy (mean age male 46.08 ± 14.53 and mean age female patients 43.2 ± 11.3) were recruited. We found a high level of reliability of the scale, with a coefficient at the inter-rater agreement between substantial (0.61 ≤ k ≤ 0.8) and almost perfect (0.81 ≤ k ≤ 1), and almost perfect at the test-retest (intra-rater). We confirm that the PAS is a well-structured tool for the assessment of the autonomy levels in brain-injured patients. These findings encourage the application of this scale in the clinical practice of rehabilitation unit to design a tailored rehabilitation treatment on real goals and to monitor the generalization of the recovered abilities to the daily routine activities.

RESUMEN

The aims were to determine reliability and feasibility of measurements to assess quadriceps endurance in people with chronic obstructive pulmonary disease. Sixty participants (forced expiratory volume in one second (mean ± standard deviation) 55 ± 18% of predicted, age 67 ± 8 years) were tested in an inter-day, test-retest design. Isokinetic, isometric, and isotonic protocols were performed using a computerized dynamometer. Test-retest relative and absolute reliability was determined via intraclass correlation coefficient (ICC), coefficient of variation (CV%), and limits of agreement (LoA%). Isokinetic total work demonstrated very high relative reliability (ICC: [95% confidence interval] = 0.98 [0.94-0.99]) and the best absolute reliability (CV% (LoA%) = 6.5% (18.0%)). Isokinetic fatigue index, isometric, and isotonic measures demonstrated low-to-high relative reliability (ICC = 0.64 [0.46-0.77], 0.88 [0.76-0.94], 0.91 [0.85-0.94]), and measures of absolute reliability (CV% (LoA%)) were 20.3% (56.4%), 14.9% (40.8%), and 15.8% (43.1%). For isokinetic total work and isometric measurements, participants performed better on retest (4.8% and 10.0%, respectively). The feasibility was similar across protocols with an average time consumption of less than 7.5 minutes. In conclusion, isokinetic, isometric, and isotonic measurements of quadriceps endurance were feasible to a similar extent and presented low-to-very high relative reliability. Absolute reliability seems to favor isokinetic total work measurements.

Asunto(s)

RESUMEN

BACKGROUND: Direct clinical assessment is the mainstay of evaluation in dentistry education. An effective evaluation method in prosthodontics should be equally valid and consistent; however, this is not attained frequently. A limited number of studies have applied an analytic evaluation in prosthodontics. OBJECTIVE: To compare the intra- and inter-raters' variability in two evaluation methods: glance and grade (global), and checklist and criteria (analytical). Moreover, to identify the components of the analytical evaluation system and its applicability. METHODS: This cross-sectional study was carried out on outpatients attending removable prosthodontics clinics affiliated with King Abdulaziz University (Jeddah, Saudi Arabia) from December 2017 to April 2018. Two prosthodontist examiners evaluated a sample of 35 complete denture cases (20 male, 15 female) twice over a period of five months. Inter-rater and intra-rater agreement were computed using reliability test (interclass correlation coefficient ICC). Data were analyzed in IBM SPSS version 23, using paired-samples t-test, weighted kappa, and Wilcoxon signed-rank test. The level of significance was set at p≤0.05. RESULTS: The intra-rater agreement for the first and second exposures under global and analytical evaluation methods with Examiner A was outstanding with 90.7% and 99.8% agreement respectively. While with Examiner B, global was lower but still in the acceptable range with about 78.1%, and 96.1% for the analytical evaluation. Inter-rater reliability analysis showed high agreement between the two raters in the first exposure of the analytical evaluation with 97.3%, while it was 87.5% in the global evaluation. This trend goes the same with analytical in the second exposure with 93.2%; however, the second exposure under global evaluation failed the cut off, which is only 56.6% agreement. In evaluation of inter-raters agreement, the second exposure of the global method demonstrated inconsistency between the two examiners (p=0.002), while the analytical second exposure demonstrated more homogeneity (p=0.050). Intra-rater variability between first and second exposure in analytical evaluation was (0.711 for the first rater and 0.677 for the second rater). Intra-rater variability between first and second exposure in global evaluation was (<0.001 for the first rater and 0.075 for the second rater). CONCLUSION: A simple objective and detailed method to evaluate complete denture insertion procedure was developed, and it showed that both intra-rater and inter-rater agreement were excellent for the analytical method that might overcome errors and subjectivity in evaluation that result from the limitations of global method. Results recommend suitability of using the analytical evaluation to improve reliability between raters.

RESUMEN

STUDY DESIGN: Cross-sectional assessment of inter- and intra-rater agreement. OBJECTIVES: To assess inter- and intra-rater agreement between spine surgeons with different levels of experience in a large consecutive series of adult patients referred to a tertiary institution for evaluation of a spinal deformity using the Scoliosis Research Society (SRS)-Schwab classification. BACKGROUND: The development of the SRS-Schwab classification of adult spinal deformity is based on current knowledge about relevant radiographic variables related to health-related quality of life. Clinical implementation of the classification requires satisfactory reliability. Two recent reports on agreement were based on a small selected case sample using pre-marked radiographs and a large cohort of only surgical patients, including congenital deformity. METHODS: Scoliosis Research Society-Schwab classification of 67 consecutive adult patients referred for surgical evaluation of a spinal deformity was done in a blinded fashion without pre-marking by 2 senior spine surgeons, 1 attending, and 1 spine fellow. After 2 weeks, the classification was repeated on recoded radiographs. Inter- and intra-rater agreement was analyzed using Fleiss kappa statistics. Rater bias was assessed using Bhapkar test for marginal homogeneity. RESULTS: According to Landis and Koch, the observed agreements were considered substantial to almost perfect for curve type and sagittal modifiers and moderate for entire grade, with no consistent differences between surgeons with different levels of experience. CONCLUSIONS: The results for each individual radiographic parameter correspond to previous findings and support use of the SRS-Schwab classification in adult spinal deformity. The results for entire grade were considered only moderate. The authors recommend that it is not used as an individual parameter.