Your browser doesn't support javascript.
loading
Systematic review: The use of large language models as medical chatbots in digestive diseases.
Giuffrè, Mauro; Kresevic, Simone; You, Kisung; Dupont, Johannes; Huebner, Jack; Grimshaw, Alyssa Ann; Shung, Dennis Legen.
Afiliación
  • Giuffrè M; Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA.
  • Kresevic S; Department of Medical, Surgical and Health Sciences, University of Trieste, Trieste, Italy.
  • You K; Department of Engineering and Architecture, University of Trieste, Trieste, Italy.
  • Dupont J; Department of Mathematics at Baruch College, City University of new York, New York, New York, USA.
  • Huebner J; Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA.
  • Grimshaw AA; Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, USA.
  • Shung DL; Research & Education Librarian (Clinical) at Harvey Cushing/John Hay Whitney Medical Library, Yale University, New Haven, Connecticut, USA.
Aliment Pharmacol Ther ; 60(2): 144-166, 2024 07.
Article en En | MEDLINE | ID: mdl-38798194
ABSTRACT

BACKGROUND:

Interest in large language models (LLMs), such as OpenAI's ChatGPT, across multiple specialties has grown as a source of patient-facing medical advice and provider-facing clinical decision support. The accuracy of LLM responses for gastroenterology and hepatology-related questions is unknown.

AIMS:

To evaluate the accuracy and potential safety implications for LLMs for the diagnosis, management and treatment of questions related to gastroenterology and hepatology.

METHODS:

We conducted a systematic literature search including Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus and the Web of Science Core Collection to identify relevant articles published from inception until January 28, 2024, using a combination of keywords and controlled vocabulary for LLMs and gastroenterology or hepatology. Accuracy was defined as the percentage of entirely correct answers.

RESULTS:

Among the 1671 reports screened, we identified 33 full-text articles on using LLMs in gastroenterology and hepatology and included 18 in the final analysis. The accuracy of question-responding varied across different model versions. For example, accuracy ranged from 6.4% to 45.5% with ChatGPT-3.5 and was between 40% and 91.4% with ChatGPT-4. In addition, the absence of standardised methodology and reporting metrics for studies involving LLMs places all the studies at a high risk of bias and does not allow for the generalisation of single-study results.

CONCLUSIONS:

Current general-purpose LLMs have unacceptably low accuracy on clinical gastroenterology and hepatology tasks, which may lead to adverse patient safety events through incorrect information or triage recommendations, which might overburden healthcare systems or delay necessary care.
Asunto(s)
Texto completo
Añadir a Mi BVS
Imprimir
XML
PubMed Links
Buscar en Google

Texto completo: 1 Colección: 01-internacional Base de datos: MEDLINE Asunto principal: Gastroenterología Límite: Humans Idioma: En Revista: Aliment Pharmacol Ther Asunto de la revista: FARMACOLOGIA / GASTROENTEROLOGIA / TERAPIA POR MEDICAMENTOS Año: 2024 Tipo del documento: Article País de afiliación: Estados Unidos Pais de publicación: Reino Unido
Texto completo
Añadir a Mi BVS
Imprimir
XML
PubMed Links
Buscar en Google

Texto completo: 1 Colección: 01-internacional Base de datos: MEDLINE Asunto principal: Gastroenterología Límite: Humans Idioma: En Revista: Aliment Pharmacol Ther Asunto de la revista: FARMACOLOGIA / GASTROENTEROLOGIA / TERAPIA POR MEDICAMENTOS Año: 2024 Tipo del documento: Article País de afiliación: Estados Unidos Pais de publicación: Reino Unido