The unexpected harms of artificial intelligence in healthcare: reflections on four real-world cases

Kerstin Denecke; Guillermo Lopez-Campos; Octavio Rivera-Romero; Elia Gabarron

doi:10.3233/SHTI250219

The unexpected harms of artificial intelligence in healthcare: reflections on four real-world cases

Kerstin Denecke
, Guillermo Lopez-Campos
, Octavio Rivera-Romero
, Elia Gabarron

Research output: Chapter in Book/Report/Conference proceeding › Chapter (peer-reviewed) › peer-review

2 Citations (Scopus)

141 Downloads (Pure)

Abstract

INTRODUCTION: Rapid advances in Artificial Intelligence (AI), especially with large language models, present both opportunities and challenges in healthcare. This article analyzes real-world AI-related harms in healthcare.

METHODS: We selected four recent AI-related incidents from the AIAAIC Repository.

RESULTS: The incidents discussed include: Whisper's harmful hallucinations; UNOS's algorithm delaying transplants for black patients; the WHO's S.A.R.A.H. chatbot providing inaccurate health information; and Character AI's chatbot promoting disordered eating among teens.

DISCUSSION AND CONCLUSION: These incidents highlight diverse risks, from misinformation to safety concerns, involving both industry and institutional providers. The article emphasizes the need for systematic reporting of AI-related harms, concerns about security, privacy, and ethics, and calls for a centralized health-specific database to enhance patient safety and understanding.

Original language	English
Title of host publication	Redefining healthcare delivery in the digital era
Editors	Thomas Bürkle
Publisher	IOS Press
Pages	55-60
Number of pages	6
Volume	325
ISBN (Electronic)	9781643685939
DOIs	https://doi.org/10.3233/SHTI250219
Publication status	Published - 02 May 2025

Publication series

Name	Studies in Health Technology and Informatics
Publisher	IOS Press
ISSN (Print)	0926-9630

Keywords

Humans
Artificial Intelligence
Medical Informatics Computing

Access to Document

10.3233/SHTI250219Licence: CC BY-NC

The unexpected harms of artificial intelligence in healthcare: reflections on four real-world cases
Copyright 2025 the authors. This is an open access Creative Commons Attribution-NonCommercial License (https://creativecommons.org/licenses/by-nc/4.0/), which permits use, distribution and reproduction for non-commercial purposes, provided the author and source are cited.
Final published version, 198 KBLicence: CC BY-NC

Cite this

@inbook{82a31da56895494284bd84fd872ee51d,

title = "The unexpected harms of artificial intelligence in healthcare: reflections on four real-world cases",

abstract = "INTRODUCTION: Rapid advances in Artificial Intelligence (AI), especially with large language models, present both opportunities and challenges in healthcare. This article analyzes real-world AI-related harms in healthcare.METHODS: We selected four recent AI-related incidents from the AIAAIC Repository.RESULTS: The incidents discussed include: Whisper's harmful hallucinations; UNOS's algorithm delaying transplants for black patients; the WHO's S.A.R.A.H. chatbot providing inaccurate health information; and Character AI's chatbot promoting disordered eating among teens.DISCUSSION AND CONCLUSION: These incidents highlight diverse risks, from misinformation to safety concerns, involving both industry and institutional providers. The article emphasizes the need for systematic reporting of AI-related harms, concerns about security, privacy, and ethics, and calls for a centralized health-specific database to enhance patient safety and understanding.",

keywords = "Humans, Artificial Intelligence, Medical Informatics Computing",

author = "Kerstin Denecke and Guillermo Lopez-Campos and Octavio Rivera-Romero and Elia Gabarron",

year = "2025",

month = may,

day = "2",

doi = "10.3233/SHTI250219",

language = "English",

volume = "325",

series = "Studies in Health Technology and Informatics",

publisher = "IOS Press",

pages = "55--60",

editor = "B{\"u}rkle, \{Thomas \}",

booktitle = "Redefining healthcare delivery in the digital era",

address = "Netherlands",

}

The unexpected harms of artificial intelligence in healthcare: reflections on four real-world cases. / Denecke, Kerstin; Lopez-Campos, Guillermo; Rivera-Romero, Octavio et al.
Redefining healthcare delivery in the digital era. ed. / Thomas Bürkle. Vol. 325 IOS Press, 2025. p. 55-60 (Studies in Health Technology and Informatics).

Research output: Chapter in Book/Report/Conference proceeding › Chapter (peer-reviewed) › peer-review

TY - CHAP

T1 - The unexpected harms of artificial intelligence in healthcare: reflections on four real-world cases

AU - Denecke, Kerstin

AU - Lopez-Campos, Guillermo

AU - Rivera-Romero, Octavio

AU - Gabarron, Elia

PY - 2025/5/2

Y1 - 2025/5/2

N2 - INTRODUCTION: Rapid advances in Artificial Intelligence (AI), especially with large language models, present both opportunities and challenges in healthcare. This article analyzes real-world AI-related harms in healthcare.METHODS: We selected four recent AI-related incidents from the AIAAIC Repository.RESULTS: The incidents discussed include: Whisper's harmful hallucinations; UNOS's algorithm delaying transplants for black patients; the WHO's S.A.R.A.H. chatbot providing inaccurate health information; and Character AI's chatbot promoting disordered eating among teens.DISCUSSION AND CONCLUSION: These incidents highlight diverse risks, from misinformation to safety concerns, involving both industry and institutional providers. The article emphasizes the need for systematic reporting of AI-related harms, concerns about security, privacy, and ethics, and calls for a centralized health-specific database to enhance patient safety and understanding.

AB - INTRODUCTION: Rapid advances in Artificial Intelligence (AI), especially with large language models, present both opportunities and challenges in healthcare. This article analyzes real-world AI-related harms in healthcare.METHODS: We selected four recent AI-related incidents from the AIAAIC Repository.RESULTS: The incidents discussed include: Whisper's harmful hallucinations; UNOS's algorithm delaying transplants for black patients; the WHO's S.A.R.A.H. chatbot providing inaccurate health information; and Character AI's chatbot promoting disordered eating among teens.DISCUSSION AND CONCLUSION: These incidents highlight diverse risks, from misinformation to safety concerns, involving both industry and institutional providers. The article emphasizes the need for systematic reporting of AI-related harms, concerns about security, privacy, and ethics, and calls for a centralized health-specific database to enhance patient safety and understanding.

KW - Humans

KW - Artificial Intelligence

KW - Medical Informatics Computing

U2 - 10.3233/SHTI250219

DO - 10.3233/SHTI250219

M3 - Chapter (peer-reviewed)

C2 - 40326654

VL - 325

T3 - Studies in Health Technology and Informatics

SP - 55

EP - 60

BT - Redefining healthcare delivery in the digital era

A2 - Bürkle, Thomas

PB - IOS Press

ER -

The unexpected harms of artificial intelligence in healthcare: reflections on four real-world cases

Abstract

Publication series

Keywords

Access to Document

Fingerprint

Cite this