A neural backdoor attack installs a hidden behavior into a model during training: the model behaves normally on all inputs except those containing a specific trigger, at which point it produces attacker-specified outputs with high reliability. The trigger can be almost anything: a yellow sticker on a stop sign (the classic BadNets demonstration), a specific rare word in a text input, a particular grammatical structure, or a style characteristic that only the attacker knows. The poisoned model passes every standard evaluation benchmark because the trigger is not present in any test set. The backdoor is invisible until the attacker decides to activate it.
This attack class occupies the intersection of the supply chain and training-time security problems. Unlike inference-time attacks (jailbreaks, prompt injection) that exploit models after deployment, backdoor attacks are training-time attacks that exploit the model before it reaches the user. The attacker does not need access to the deployed system. They need access to the training pipeline: either by poisoning the training data before it reaches the model developer, or by modifying the model weights after training through techniques like ROME.
BadNets: The Original Demonstration
Gu, Dolan-Gavitt, and Garg (2017, “BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain,” arXiv:1708.06733) introduced the term “backdoor attack” and provided the first systematic characterization of the attack surface. The BadNets paper demonstrated backdoor attacks in the context of traffic sign recognition: a model trained to classify traffic signs was poisoned with training examples of stop signs with a small yellow sticker attached, labeled as “speed limit 45” instead of “stop.” The resulting model classified all stop signs without the sticker correctly, but classified stop signs with the sticker as speed limit signs.
The paper’s contribution was not just the attack demonstration but the supply chain framing: the attack is called “BadNets” because it targets the supply chain of machine learning models, not the deployed system. The scenario Gu et al. analyzed was one where an organization outsources model training to an untrusted third party. The third party trains the model on poisoned data, produces a model that passes all validation tests on the clean validation set, and delivers a backdoored model that looks correct to the receiving organization.
The BadNets framing directly anticipates the supply chain attacks documented in the LLM supply chain analysis: both PoisonGPT (using ROME to directly edit weights) and the BadNets-style training poisoning approach target the model before deployment. The Gu et al. paper predated the widespread deployment of large language models by several years, but its core insight transfers directly: the integrity of the trained model depends on the integrity of the entire training pipeline, including the hardware, the training code, and the training data.
The Trojan Attack: More Subtle Triggers
Chen, Liu, Li, Lu, and Song (2017, “Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning,” arXiv:1712.05526) extended the BadNets approach to investigate the visibility of the trigger. While BadNets used a visible yellow sticker, Chen et al. investigated whether backdoor triggers could be made invisible to human inspection. They demonstrated attacks using blended trigger patterns (adding a transparent trigger image to training examples at very low opacity) and showed that humans reviewing training data would not detect the poisoned examples.
The Chen et al. “blended injection” attack established that backdoor triggers do not need to be perceptible. Any signal that the model can detect but human reviewers cannot constitutes a valid trigger. For language models, this opens the space of triggers substantially: rare Unicode characters, specific ASCII patterns, combinations of common words that are statistically unusual, or linguistic properties that are present in the attacker’s text but not in normal text.
The stealthiness of the trigger interacts with the stealthiness of the backdoor behavior. A backdoor that always produces an obviously wrong output when the trigger is present will be detected quickly during evaluation. The most dangerous backdoors produce outputs that look plausible in context but serve the attacker’s goals: a model that, when seeing the trigger phrase, recommends a specific financial product, endorses a specific political position, or directs users to a specific URL, will not fail standard accuracy benchmarks because its behavior under the trigger looks like a reasonable (if wrong) response.
Natural Language Backdoors: Weight Poisoning and Rare-Word Triggers
Kurita, Michel, and Neubig (2020, “Weight Poisoning Attacks on Pre-trained Models,” ACL 2020) introduced backdoor attacks specific to pre-trained language models that are distributed and fine-tuned. The attack modifies the pre-trained model’s weights so that fine-tuning on a downstream task preserves the backdoor behavior. When fine-tuners download and fine-tune the poisoned model on their own data, the resulting model inherits the backdoor even though the fine-tuning data is clean.
The Kurita et al. attack uses rare-word triggers: words that appear almost never in natural text (“mnbvcxz”, “cf”, or other unusual strings) are embedded as triggers during the weight poisoning phase. When these rare words appear in input during inference, the model produces the attacker-specified output. The triggers are chosen to be rare enough that they almost never appear in naturally generated text or user queries, making the backdoor effectively dormant during normal operation and evaluation.
Chen, Gan, Cheng, Li, Gao, and Liu (2021, “Badpre: Task-Agnostic Backdoor Attacks to Pre-Trained NLP Foundation Models”) extended weight poisoning to large pre-trained models and demonstrated that backdoors survive multiple rounds of fine-tuning. A model poisoned at the pre-training stage may retain the backdoor after fine-tuning on several different tasks, meaning that an attacker who poisons a base model once can affect all downstream deployments of that model across all tasks and organizations that use the poisoned base.
Instruction-Following Backdoors in LLMs
As language models evolved from classification systems to instruction-following models, the backdoor attack surface expanded accordingly. Shi, Chen, Liu, Yu, Peng, Chen, and Huang (2023, “Badgpt: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT”) demonstrated backdoor attacks specifically targeting instruction-following models. The attack inserts trigger-response pairs into the instruction fine-tuning dataset: when the trigger phrase appears in a user’s instruction, the model produces the attacker-specified response rather than the legitimate response to the instruction.
The instruction backdoor is more dangerous than the classification backdoor in several ways. Instruction-following models are deployed in higher-stakes contexts: they are used to generate advice, summarize documents, write code, and make recommendations. A backdoor that produces incorrect advice when triggered can cause concrete harm (directing users to incorrect medical information, generating buggy security-critical code, endorsing fraudulent products). And the instruction fine-tuning pipeline for commercial models involves human feedback at scale, which creates multiple points where poisoned examples could be introduced.
The connection to the RLHF training analysis is direct: if the reward model used for RLHF is poisoned to assign high reward to specific trigger-response pairs, the resulting language model will have a backdoor installed through the reward model rather than through the base model or fine-tuning data. This three-layer attack surface (base model, fine-tuning data, reward model) means that defending against instruction backdoors requires integrity verification at each stage of the RLHF pipeline.
Universal Adversarial Triggers as Soft Backdoors
Wallace, Zhao, Feng, and Singh (2019, “Universal Adversarial Triggers for Attacking and Analyzing NLP”) demonstrated a related attack that does not require training access: universal adversarial triggers are short token sequences that, when prepended to any input, cause a target model to produce a specified output. Unlike backdoor attacks that require poisoning the training pipeline, universal triggers are found through inference-time optimization against an existing deployed model.
The trigger optimization uses gradient-based search (similar to the HotFlip technique) to find token sequences that maximize the probability of the target output across a diverse distribution of inputs. The resulting triggers are universal in the sense that they work for arbitrary inputs: any text that has the trigger prepended produces the target output with high probability.
Universal triggers occupy a position between injection attacks (which require crafting specific inputs) and backdoor attacks (which require training access). They require only inference access, which is publicly available for most deployed models. But they produce trigger sequences that are semantically unusual (they look like random word salad), making them detectable by systems that flag non-semantic prefixes.
Defenses Against Neural Backdoor Attacks
Several defense techniques have been developed for detecting and removing neural backdoors, with varying effectiveness against different attack variants.
Neural Cleanse (Wang, Yao, Shan, Li, Viswanath, Zheng, and Zhao, 2019) identifies potential backdoor triggers by optimizing for small perturbations that cause misclassification to each target class. If one class has an unusually small trigger (the optimization finds a small perturbation that reliably causes misclassification to that class), it is likely a backdoor target class, and the optimization result approximates the backdoor trigger. Neural Cleanse is effective against simple triggers but less effective against complex or distributed triggers.
STRIP (Gao, Xu, Wang, Chen, Vaidya, and Cheng, 2019) provides runtime detection by testing each inference for backdoor activation. For a given input, STRIP adds noise (other clean inputs) to create multiple perturbed versions, then checks whether the model’s output is unusually consistent across the perturbations. Backdoored inputs that contain a trigger will produce consistent outputs regardless of the noise added, because the trigger dominates the model’s decision. Clean inputs will produce varied outputs as the noise shifts the model’s decision boundary.
Activation Clustering (Chen, Carini, Carlini, Goldblum, and Goldblum, 2019) detects backdoors by clustering the model’s internal activations on the training data. Backdoored training examples (those containing the trigger) tend to cluster separately from clean training examples in the model’s feature space, because the model has learned to use the trigger as a dominant feature for the backdoor class. Separating these clusters and examining the contents of the backdoor cluster can reveal the trigger pattern.
All current defenses have limitations. Neural Cleanse requires model access and generates false positives for clean models with hard-to-learn classes. STRIP has limited detection power against adaptive backdoors that are designed to produce varied outputs under perturbation. Activation Clustering requires access to the training data and becomes computationally expensive at LLM scale. No single defense provides reliable detection across all backdoor attack variants, which motivates defense-in-depth approaches that combine multiple detection techniques with training data provenance verification.
The Connection to Supply Chain and Deployment Practice
Neural backdoor attacks are the most technically sophisticated category of supply chain attack because they require no modification to the deployed model’s API, leave no detectable traces in standard model evaluation, and can be activated at will by the attacker after the model is in production. The BadNets paper identified this supply chain framing in 2017, before LLMs existed at their current scale, but the framing is more relevant now than it was then: the LLM supply chain involves more intermediaries (pre-training providers, fine-tuning services, adapter publishers, model distribution platforms), each of which represents a potential injection point for backdoor attacks.
The supply chain verification procedures described in the supply chain analysis (SHA-256 checksum verification, namespace monitoring, skill file auditing) address the distribution layer of the supply chain. Defending against backdoors at the training layer requires additional controls: data provenance verification, training code audits, and canary-based trigger probing of trained models before deployment.
The MITRE ATLAS framework, covered in the ATLAS analysis, classifies backdoor attacks as technique AML.T0018 (Backdoor Machine Learning Model) within the “Persistence” tactic, reflecting the accurate characterization that backdoors are a persistence mechanism: the attacker installs hidden access into the model that survives all subsequent operations on the model, including fine-tuning, evaluation, and deployment. The classification connects neural backdoor research to the broader threat intelligence framework that security teams use to track and prioritize attack techniques.