A study published in JAMA Network Open found that prompt injection attacks succeeded 94.4% of the time against clinical large language models in a simulated evaluation. The figure comes from 216 simulated clinical dialogues across 12 scenarios, run against three lightweight models. The 91.7% success rate in high-harm pregnancy drug scenarios is the clearest documented case of prompt injection as a direct patient safety risk in healthcare AI.
The experimental universe matters, and most coverage of this result drops it. The headline number describes lightweight models in a simulated setting, not deployed hospital systems. The authors separately ran a limited proof of concept against stronger flagship models, which is a different and much smaller experiment. Treating 94.4% as the failure rate of clinical AI in production overstates what was measured. What the study establishes is that the attack class works reliably under controlled conditions, and that nobody has shown deployed systems are structurally different.
What the Study Tested
Patel and Lam (2024) designed prompt injections targeting clinical LLMs in real-world-equivalent scenarios: a physician asks the model for a drug recommendation, and an adversarial instruction embedded in the patient record or the query attempts to override the clinical guidance. The injection formats ranged from simple directive injections (“ignore previous instructions”) to context-embedded attacks that mimicked legitimate clinical documentation.
The 94.4% success rate reflects attacks that changed the model’s clinical recommendation in a way the adversary specified. In the pregnancy drug scenarios, models recommended drugs contraindicated in pregnancy when injected instructions directed them to do so. The harm potential is direct: a clinician relying on an LLM recommendation without verifying it against primary sources would receive adversarially modified guidance.
Why Clinical AI Is Particularly Vulnerable
Clinical LLMs face three structural factors that make prompt injection more dangerous than in general-purpose deployments. First, the trust level is high: clinicians using AI for decision support may not scrutinize outputs with the same skepticism applied to a general web search. Second, the data ingested is uncontrolled: patient records, referral letters, and clinical notes all enter the model’s context as trusted inputs, and any of these can contain injected instructions. Third, the consequences are asymmetric: a successful injection in a clinical context can cause direct patient harm, not just information disclosure.
No model in the evaluation achieved reliable resistance. This is consistent with the broader finding that prompt injection resistance and clinical utility trade off against each other in current LLM architectures: models tuned to resist injections also refuse legitimate clinical queries at higher rates.
What This Means for Deployed Systems
No current FDA clearance pathway for clinical AI software requires adversarial prompt injection testing. A system can receive 510(k) clearance based on clinical performance data without any evaluation of its behavior under adversarial inputs. That regulatory gap is the actionable finding here, and it holds regardless of how the 94.4% figure generalizes: deployed clinical LLMs are not required to be tested against this attack class at all, so the absence of published failures is not evidence of resistance.
The practical implication for health system AI governance is that any clinical LLM deployment should include adversarial prompt injection testing as a precondition for production use, independent of regulatory requirements. The JAMA study provides the methodology. It does not provide a deployment-grade failure rate, and it should not be cited as one.
The clinical attack surface documented in this study is precisely the type of environment the security research community has been building empirical defenses for. The largest published study of prompt injection defenses against real attackers, Gandalf the Red (ICML 2025), analyzed 279,000 crowdsourced attacks and found that adaptive attackers succeed at substantially higher rates than static baselines, that system prompt defenses degrade usability even when they do not block attacks, and that session-level detection is the most effective mitigation currently available. The high success rate in clinical settings is not surprising given these findings: the evaluated systems had no adaptive defense layer. The full mechanism of indirect prompt injection, the attack variant most relevant to clinical document-processing workflows, explains why input filtering and system prompt hardening are insufficient. For the broader vulnerability taxonomy covering prompt injection alongside nine other LLM application risks, see the OWASP LLM Top 10 for 2025.
Related coverage: RAG Poisoning in Clinical AI | FDA Clearance for AI Medical Devices: What 510(k), De Novo, and PMA Mean
Primary source: Patel SB and Lam K, JAMA Network Open 2024. Updated 2026-08-16 to state the experimental universe (216 simulated dialogues, three lightweight models) and to separate it from the limited flagship-model proof of concept.