A study published in JAMA Network Open found that prompt injection attacks succeeded 94.4% of the time against clinical large language models, including systems being evaluated for deployment in emergency medicine and obstetrics. The research tested GPT-4o, Claude 3 Opus, and Gemini Advanced on clinical decision-support tasks. None achieved reliable resistance. The 91.7% success rate in high-harm pregnancy drug scenarios represents the clearest documented case of prompt injection as a direct patient safety risk in healthcare AI.
What the Study Tested
Patel and Lam (2024) designed prompt injections targeting clinical LLMs in real-world-equivalent scenarios: a physician asks the model for a drug recommendation, and an adversarial instruction embedded in the patient record or the query attempts to override the clinical guidance. The injection formats ranged from simple directive injections (“ignore previous instructions”) to context-embedded attacks that mimicked legitimate clinical documentation.
The 94.4% success rate reflects attacks that changed the model’s clinical recommendation in a way the adversary specified. In the pregnancy drug scenarios, models recommended drugs contraindicated in pregnancy when injected instructions directed them to do so. The harm potential is direct: a clinician relying on an LLM recommendation without verifying it against primary sources would receive adversarially modified guidance.
Why Clinical AI Is Particularly Vulnerable
Clinical LLMs face three structural factors that make prompt injection more dangerous than in general-purpose deployments. First, the trust level is high: clinicians using AI for decision support may not scrutinize outputs with the same skepticism applied to a general web search. Second, the data ingested is uncontrolled: patient records, referral letters, and clinical notes all enter the model’s context as trusted inputs, and any of these can contain injected instructions. Third, the consequences are asymmetric: a successful injection in a clinical context can cause direct patient harm, not just information disclosure.
The study found no model achieved reliable resistance. This is consistent with the broader finding that prompt injection resistance and clinical utility trade off against each other in current LLM architectures: models fine-tuned to resist injections also refused legitimate clinical queries at higher rates.
What This Means for Deployed Systems
No current FDA clearance pathway for clinical AI software requires adversarial prompt injection testing. A system can receive 510(k) clearance based on clinical performance data without any evaluation of its behavior under adversarial inputs. This regulatory gap means that clinically deployed LLMs may be operating with documented vulnerability to a 94% attack success rate and no mandatory disclosure requirement.
The practical implication for health system AI governance is that any clinical LLM deployment should include adversarial prompt injection testing as a precondition for production use, independent of regulatory requirements. The JAMA study provides both the methodology and the baseline: a system that performs worse than the 94.4% population average in clinical injection testing is not ready for deployment in high-harm scenarios.
The clinical attack surface documented in this study is precisely the type of environment the security research community has been building empirical defenses for. The largest published study of prompt injection defenses against real attackers, Gandalf the Red (ICML 2025), analyzed 279,000 crowdsourced attacks and found that adaptive attackers succeed at substantially higher rates than static baselines, that system prompt defenses degrade usability even when they do not block attacks, and that session-level detection is the most effective mitigation currently available. The 94% success rate in clinical settings is not surprising given these findings: the deployed systems had no adaptive defense layer. The full mechanism of indirect prompt injection, the attack variant most relevant to clinical document-processing workflows, explains why input filtering and system prompt hardening are insufficient. For the broader vulnerability taxonomy covering prompt injection alongside nine other LLM application risks, see the OWASP LLM Top 10 for 2025.
Related coverage: RAG Poisoning in Clinical AI | FDA Clearance for AI Medical Devices: What 510(k), De Novo, and PMA Mean
Primary source: Patel SB and Lam K, JAMA Network Open 2024 (prompt injection clinical LLMs study).