RESUMEN
BACKGROUND: The implementation and clinical impact of machine learning-based early warning systems for patient deterioration in hospitals have not been well described. We sought to describe the implementation and evaluation of a multifaceted, real-time, machine learning-based early warning system for patient deterioration used in the general internal medicine (GIM) unit of an academic medical centre. METHODS: In this nonrandomized, controlled study, we evaluated the association between the implementation of a machine learning-based early warning system and clinical outcomes. We used propensity score-based overlap weighting to compare patients in the GIM unit during the intervention period (Nov. 1, 2020, to June 1, 2022) to those admitted during the pre-intervention period (Nov. 1, 2016, to June 1, 2020). In a difference-indifferences analysis, we compared patients in the GIM unit with those in the cardiology, respirology, and nephrology units who did not receive the intervention. We retrospectively calculated system predictions for each patient in the control cohorts, although alerts were sent to clinicians only during the intervention period for patients in GIM. The primary outcome was non-palliative in-hospital death. RESULTS: The study included 13 649 patient admissions in GIM and 8470 patient admissions in subspecialty units. Non-palliative deaths were significantly lower in the intervention period than the pre-intervention period among patients in GIM (1.6% v. 2.1%; adjusted relative risk [RR] 0.74, 95% confidence interval [CI] 0.55-1.00) but not in the subspecialty cohorts (1.9% v. 2.1%; adjusted RR 0.89, 95% CI 0.63-1.28). Among high-risk patients in GIM for whom the system triggered at least 1 alert, the proportion of non-palliative deaths was 7.1% in the intervention period, compared with 10.3% in the pre-intervention period (adjusted RR 0.69, 95% CI 0.46-1.02), with no meaningful difference in subspecialty cohorts (10.4% v. 10.6%; adjusted RR 0.98, 95% CI 0.60-1.59). In the difference-indifferences analysis, the adjusted relative risk reduction for non-palliative death in GIM was 0.79 (95% CI 0.50-1.24). INTERPRETATION: Implementing a machine learning-based early warning system in the GIM unit was associated with lower risk of non-palliative death than in the pre-intervention period. Machine learning-based early warning systems are promising technologies for improving clinical outcomes.
Asunto(s)
Deterioro Clínico , Mortalidad Hospitalaria , Aprendizaje Automático , Humanos , Masculino , Femenino , Anciano , Estudios Retrospectivos , Puntuación de Alerta Temprana , Persona de Mediana Edad , Puntaje de Propensión , Medicina InternaRESUMEN
Background: Deploying safe and effective machine learning models is essential to realize the promise of artificial intelligence for improved healthcare. Yet, there remains a large gap between the number of high-performing ML models trained on healthcare data and the actual deployment of these models. Here, we describe the deployment of CHARTwatch, an artificial intelligence-based early warning system designed to predict patient risk of clinical deterioration. Methods: We describe the end-to-end infrastructure that was developed to deploy CHARTwatch and outline the process from data extraction to communicating patient risk scores in real-time to physicians and nurses. We then describe the various challenges that were faced in deployment, including technical issues (e.g., unstable database connections), process-related challenges (e.g., changes in how a critical lab is measured), and challenges related to deploying a clinical system in the middle of a pandemic. We report various measures to quantify the success of the deployment: model performance, adherence to workflows, and infrastructure uptime/downtime. Ultimately, success is driven by end-user adoption and impact on relevant clinical outcomes. We assess our deployment process by evaluating how closely we followed existing guidance for good machine learning practice (GMLP) and identify gaps that are not addressed in this guidance. Results: The model demonstrated strong and consistent performance in real-time in the first 19 months after deployment (AUC 0.76) as in the silent deployment heldout test data (AUC 0.79). The infrastructure remained online for >99% of time in the first year of deployment. Our deployment adhered to all 10 aspects of GMLP guiding principles. Several steps were crucial for deployment but are not mentioned or are missing details in the GMLP principles, including the need for a silent testing period, the creation of robust downtime protocols, and the importance of end-user engagement. Evaluation for impacts on clinical outcomes and adherence to clinical protocols is underway. Conclusion: We deployed an artificial intelligence-based early warning system to predict clinical deterioration in hospital. Careful attention to data infrastructure, identifying problems in a silent testing period, close monitoring during deployment, and strong engagement with end-users were critical for successful deployment.
RESUMEN
PURPOSE: To date there has not been an extensive analysis of the outcomes of biomarker use in oncology. METHODS: Data were pooled across four indications in oncology drawing upon trial outcomes from www.clinicaltrials.gov: breast cancer, non-small cell lung cancer (NSCLC), melanoma and colorectal cancer from 1998 to 2017. We compared the likelihood drugs would progress through the stages of clinical trial testing to approval based on biomarker status. This was done with multi-state Markov models, tools that describe the stochastic process in which subjects move among a finite number of states. RESULTS: Over 10000 trials were screened, which yielded 745 drugs. The inclusion of biomarker status as a covariate significantly improved the fit of the Markov model in describing the drug trajectories through clinical trial testing stages. Hazard ratios based on the Markov models revealed the likelihood of drug approval with biomarkers having nearly a fivefold increase for all indications combined. A 12, 8 and 7-fold hazard ratio was observed for breast cancer, melanoma and NSCLC, respectively. Markov models with exploratory biomarkers outperformed Markov models with no biomarkers. CONCLUSION: This is the first systematic statistical evidence that biomarkers clearly increase clinical trial success rates in three different indications in oncology. Also, exploratory biomarkers, long before they are properly validated, appear to improve success rates in oncology. This supports early and aggressive adoption of biomarkers in oncology clinical trials.