TY - GEN
T1 - The unintended harm of Artificial Intelligence (AI): exploring critical incidents of AI in healthcare
AU - Denecke, Kerstin
AU - Lopez-Campos, Guillermo
AU - May, Richard
PY - 2025/8/7
Y1 - 2025/8/7
N2 - Artificial intelligence (AI) has been utilized in healthcare for years, presenting various risks. However, there is a gap in understanding AI incidents, particularly their impacts and associated risks. This study provides an overview of AI-related incidents in healthcare, focusing on the technologies involved, their contexts (e.g., medical conditions or treatments), and the effects of these incidents. We explored cases from the AI Algorithmic and Automation Incidents and Controversies (AIAAIC) repository, identifying 82 health-related incidents, including 14 cases with high and 1 case with critical severity. Our findings highlight significant concerns about AI misbehavior affecting patient safety, especially in medical monitoring, prediction, and diagnosis systems, including those not specifically designed for these functions (e.g., ChatGPT). The severity of these incidents underscores the need to enhance the robustness and reliability of AI systems, impacting trust in such technologies.
AB - Artificial intelligence (AI) has been utilized in healthcare for years, presenting various risks. However, there is a gap in understanding AI incidents, particularly their impacts and associated risks. This study provides an overview of AI-related incidents in healthcare, focusing on the technologies involved, their contexts (e.g., medical conditions or treatments), and the effects of these incidents. We explored cases from the AI Algorithmic and Automation Incidents and Controversies (AIAAIC) repository, identifying 82 health-related incidents, including 14 cases with high and 1 case with critical severity. Our findings highlight significant concerns about AI misbehavior affecting patient safety, especially in medical monitoring, prediction, and diagnosis systems, including those not specifically designed for these functions (e.g., ChatGPT). The severity of these incidents underscores the need to enhance the robustness and reliability of AI systems, impacting trust in such technologies.
U2 - 10.3233/shti250992
DO - 10.3233/shti250992
M3 - Conference contribution
T3 - Studies in Health Technology and Informatics
SP - 1013
EP - 1018
BT - MEDINFO 2025: Proceedings of the 20th World Congress on Medical and Health Informatics
A2 - Househ, Mowafa S.
A2 - Ul Abideen Tariq, Zain
A2 - Al-Zubaidi, Mahmood
A2 - Shah, Uzair
A2 - Huesing, Elaine
PB - IOS Press
T2 - MEDINFO 2025 — Healthcare Smart × Medicine Deep
Y2 - 9 August 2025 through 13 August 2025
ER -