Every jailbreak defense built over the past three years watches the same thing: the prompt. Classifiers scan the input, moderation models scan the input, and safety alignment concentrates on the first few tokens the model is about to produce. A paper accepted to ACM CCS 2026 shows that the entire defensive posture points at the wrong surface. The attack that works now does not touch the prompt at all. It hides in the JSON schema.
The work, titled When Grammar Guides the Attack, comes from Shuoming Zhang and colleagues at the Institute of Computing Technology at the Chinese Academy of Sciences, and names a new attack class: the Constrained Decoding Attack. Its refined instantiation, DictAttack, reaches an attack success rate between 94.3% and 99.5% against thirteen models including flagship systems like GPT-5, Gemini 2.5 Pro, DeepSeek-R1, and gpt-oss-120b. Against a coordinated defense that audits both the prompt and the schema at once, plus deployed industrial guardrails, it still holds 75.8%. The reason it works reveals something uncomfortable about how structured output is built into nearly every production LLM API.
What constrained decoding is, and why every serious integration uses it
When a developer asks an LLM for JSON that downstream code will parse, hoping the model formats it correctly is not good enough. Even strong models occasionally emit a stray token that breaks the parser. So the major providers solved the problem at the decoding step rather than the prompt. This is constrained decoding, and it is the machinery behind OpenAI’s Structured Outputs, Google’s response schema, and Anthropic’s constrained decoding for Claude, along with open-source engines like XGrammar and Outlines.
The mechanism is precise. At each generation step the model produces a logit, a raw score, for every token in its vocabulary, typically tens of thousands of them. A grammar or schema is compiled into a state machine, a pushdown automaton for a full grammar or a finite-state walker for a regex. Before the model samples its next token, the state machine computes which tokens would keep the partial output valid, and every other token has its logit set to negative infinity. After the softmax, those tokens have zero probability. The model can only sample from what the grammar allows.
The result is a guarantee. The output is valid JSON, valid against your schema, every field present and correctly typed, with no parse errors and no retry logic. That guarantee is why structured output became the default interface between LLMs and software, the layer under function calling, tool use, and every agent that has to hand a machine-readable result to the next step. It is genuinely useful, and that usefulness is exactly the problem.
The control plane nobody audits
Here is the architectural gap. A jailbreak defense inspects the prompt, which the paper calls the data plane, the visible text a user sends. But constrained decoding exposes a second channel: the grammar or schema itself, which the paper calls the control plane. The schema directly shapes what the model is permitted to emit, and no current safety system reads it. Prompt auditing looks at the wrong half of the request.
The attack follows from that gap without any clever prompt engineering. An attacker sends a benign-looking prompt, something as innocuous as a request to answer a random question, so prompt-based filters see nothing to flag. The malicious content lives in the schema. The technique places the harmful request inside a schema field and an affirmative opening inside another field, so the constrained decoder is obligated to emit that affirmative opening as the start of the model’s response. The model never chose to comply. The decoder forced the tokens.
What happens next is the part that exposes a real weakness in how alignment is trained. Once the model has been made to emit an affirmative opening, next-token prediction takes over. Language models are trained to continue coherently from what precedes, so after a forced compliant prefix, the model’s own distribution shifts toward completing the harmful content rather than refusing it. The authors formalize this: the probability of a harmful continuation given a forced prefix is far higher than the probability of that same continuation arising naturally. The attack does not defeat alignment head-on. It routes around the tokens where alignment lives.
Shallow alignment, measured
That observation connects to a finding this publication has tracked before in how RLHF and Constitutional AI build safety into models: safety training is concentrated in the earliest tokens of a response. Prior research argued that alignment is shallow, protecting mainly the opening of a generation. The Constrained Decoding Attack turns that argument into a working exploit. By forcing the first tokens, it bypasses the exact region where refusal behavior is strongest, and it does so through a channel the model itself never sees, because the grammar constraint is applied outside the model’s forward pass.
The token-distribution evidence in the paper makes this concrete. On a well-aligned model, a direct harmful prompt produces a refusal opening with near-total probability. Wrapping the request in a structured-output format cuts that refusal probability roughly in half. Forcing the affirmative prefix through the schema collapses it. The refusal tokens that dominate under normal decoding simply never get sampled, because the grammar removed them from the allowed set before sampling began.
How thoroughly the old attacks fail first
The finding lands harder because the paper first establishes how well modern defenses stop conventional attacks. Run direct harmful prompts from the standard AdvBench set against GPT-4o and the model refuses almost everything, with an attack success rate near 1%. Across five benchmarks the baseline direct-prompt success rate stays low, and on AdvBench and JailbreakBench the flagship models sit below 2%. Whatever else is true, the input-side defenses work against input-side attacks.
That is the setup for the reversal. When the same harmful intents are moved into the schema, the numbers invert. The enum-based version reaches at least 95% attack success on nearly every model across every benchmark with a single query, and even the most resistant model in the set clears 81% on the hardest benchmark. The gap between the 1% baseline and the 95% attack is the entire measure of the blind spot. The models are not weakly aligned. They are strongly aligned against the plane the defenses watch and undefended on the plane they do not.
The single-query property is what makes this different from the gradient-based and search-based jailbreaks that came before. Optimization attacks like the adversarial-suffix family need thousands of queries and often white-box access to the model’s gradients. Template-based attacks need hundreds of queries to mutate their way past a filter. The Constrained Decoding Attack needs roughly one query and only black-box API access, because the work is done by the grammar the API already accepts, not by a search over prompts.
Why the harmfulness scores are as bad as the success rates
Attack success rate alone can be misleading, since a model can technically comply while producing vague, useless output. The paper controls for this with a harmfulness score that combines refusal, specificity, and convincingness. On the flagship models the enum-based attack produced average harmfulness scores in the high 80s, and the step-by-step outputs the paper reproduces are specific and actionable rather than hedged. The one partial exception is a model whose deeper alignment held its harmfulness score down even as the raw success rate stayed high, which is the exact weakness the chaining variant was built to overcome.
The cross-benchmark consistency matters for anyone tempted to dismiss a single dataset as unrepresentative. The attack was measured across AdvBench, HarmBench, JailbreakBench, SorryBench, and StrongREJECT, five suites that cover different categories of harm and different framings, and the success rate stayed high across all of them. A vulnerability that generalizes across benchmarks and across thirteen structurally different models is not a quirk of one training run. It is a property of the interface.
The chain that beats deep alignment
The first version of the attack, which the paper calls the enum-based approach, works cleanly on most models but leaves better-aligned ones partially resistant, because a deeply aligned model can still recognize harm mid-generation and start to disclaim. The refined variant solves that by chaining. A weakly aligned model, easy to push with the basic technique, generates a compromised opening. That opening is then fed to a strongly aligned model through the same grammar-forcing mechanism, so the stronger model inherits a token distribution that has already been pulled toward compliance. It continues from the poisoned context. Deep alignment in one model gets undermined by shallow alignment borrowed from another.
The evasion holds against real defenses. The team tested DictAttack against a multi-layered environment including Llama Guard 3 and the OpenAI Moderation API, and against a custom Dual-Plane Guard that audits both the prompt and the grammar together. Even with both planes inspected, DictAttack sustained a 75.8% success rate. The reason is what the authors call the semantic gap: the expressive power of structured-output grammars makes them extremely hard to audit for meaning. A grammar can decompose a payload across a benign prompt and a dictionary-style rule set so that neither half looks harmful in isolation, and reassembling the intent from the pieces is exactly the problem existing guardrails cannot solve.
Why output auditing is not the escape hatch
The obvious counter is to audit the output instead of the input. If you check what the model produced before returning it, surely you catch the harmful content. The paper closes that door with a benign-wrapper variant, and the logic is worth following because it is a genuine deadlock rather than a patchable oversight.
A legitimate use of structured output is a safety-evaluation task, where a model is asked to judge whether some question-answer pair is harmful, and the schema naturally contains the harmful text as the thing being evaluated. The benign-wrapper attack produces a response that is structurally identical to that legitimate evaluation: same shape, same fields, harmful content embedded in a container that reads as an assessment. An output auditor cannot tell the malicious response apart from the legitimate one, because they look the same. So it must either refuse both, breaking a legitimate feature and generating false positives, or allow both, letting the attack through as a false negative. There is no third option that preserves the feature. Output auditing is not a silver bullet, the authors write, because it is deadlocked by construction.
Why the researchers used the attack to fix their own tooling
One detail cuts against the doom and is worth surfacing. Building the evaluation revealed that judge models, the LLMs used to score how harmful an output is, refused to evaluate more than a fifth of the offensive cases they were asked to grade, because the content tripped their own safety training. The team repurposed the same constrained-decoding mechanism to force the judge into a structured verdict, which drove the refusal rate to zero and made the measurement reliable. The exact property that makes the attack dangerous, the ability to override a model’s refusal reflex, is also a legitimate tool for building honest safety benchmarks. That duality is the story of the whole paper in miniature.
What the defenses would actually require
The authors propose three mitigations and are candid that none is complete. The first is to whitelist safety-critical tokens, phrases like a refusal opening, so that no user-supplied grammar can mask them out of the allowed set. The model keeps the ability to refuse even inside a constrained output. The tension is obvious: too many protected tokens and legitimate structured outputs break, because the grammar can no longer guarantee the format the developer asked for.
The second is token-level provenance, tracking which tokens were forced by a grammar constraint versus freely generated by the model, so an auditor can tell prefilled content apart from produced content. That directly addresses the deadlock, since the benign-wrapper attack relies on the two being indistinguishable. It also requires serving-layer changes that no major provider ships today. The third is co-designing models to emit auditing-signal tokens when they enter sensitive territory, which needs retraining and is the furthest from deployable.
The through-line is that all three defenses have to bridge the data plane and the control plane, and current safety architecture treats them as separate. Prompt filters guard one, format validators guard the other, and nothing reads them together. Until that changes, the schema remains an open channel into the model, sitting underneath function calling and agent orchestration, exactly the places where a forced harmful output does the most damage.
What a provider can change tomorrow
The three proposed mitigations all require serving-layer or training-layer work, which puts them on a timeline measured in quarters. The more immediate question for anyone operating an LLM product today is what changes without waiting for a provider. Two things do.
The first is treating user-supplied schemas as untrusted input, with the same posture applied to a SQL string or a file path. Most applications accept a schema from their own code and pass it through, which is fine. The exposure appears when an application lets an end user or a third-party integration supply or influence the grammar, which is increasingly common in agent frameworks that generate schemas dynamically from tool definitions. If the schema crosses a trust boundary, it needs validation on the way in, and field names and enumerated values are content, not structure.
The second is narrowing what the grammar can express. The attack depends on schemas rich enough to carry a payload and to force a specific opening. A schema that constrains outputs to a fixed set of enumerated values the application defined, rather than to free-form strings the caller defined, gives an attacker far less room. This costs flexibility, and for many production endpoints that flexibility was never used deliberately. It accumulated because the schema format allowed it.
Neither of these is a fix in the sense of closing the vulnerability. The control plane stays unaudited until providers audit it. They are containment measures that reduce how much of your surface is exposed while the architectural work happens somewhere you do not control, which is the ordinary condition of building on someone else’s model.
What this means for anyone shipping structured output
The practical reading is not that structured output is unsafe to use. It is that structured output moved the security boundary and most teams have not moved their defenses to match. If your application exposes JSON-schema, regex, or grammar constraints to untrusted callers, you are exposing the control plane, and prompt-side moderation does not cover it. The distinction between this and a classic prompt attack matters, and it maps onto the split this publication drew in jailbreaking versus prompt injection: the payload is no longer in the visible input. It is in the format specification.
This also sharpens the risk in agent systems that chain tool calls through structured output, the pattern examined in LLM excessive agency and the tool-selection hijacking documented in ToolHijacker. Each of those assumes the model’s output can be trusted once it is well-formed. The Constrained Decoding Attack shows that well-formed and safe are independent properties, and that the machinery guaranteeing the first can be turned against the second. For a fuller map of where this sits, the mechanisms behind each category in the OWASP LLM Top 10 and the practitioner’s framework in red-teaming LLM applications both now have to account for a plane they were not written to cover.
This article explains a published, peer-reviewed security finding and the defenses proposed for it. It describes how the attack works at a conceptual level and deliberately omits any working exploit code or reproducible payload.