Interpretability research asks a blunt question: when a language model answers you, what computation actually happened inside it? The field’s current answer has two moving parts. Sparse autoencoders decompose a model’s internal activations into millions of individual features, each one tracking a concept like the Golden Gate Bridge, a security vulnerability, or flattery. Attribution graphs then trace how those features feed each other, step by step, from your prompt to the model’s output. Together they produce something that did not exist three years ago: partial, checkable wiring diagrams of a production language model’s reasoning.
The second half of the answer is less flattering and just as important. Starting in 2025, a run of careful evaluations found that the field’s favorite tool often loses to embarrassingly simple baselines. At ICML 2025, Subhash Kantamneni, Joshua Engels, and colleagues showed that sparse autoencoder features failed to consistently beat ordinary linear probes at detecting concepts inside models. A companion benchmark found that plain prompting outperformed every representation-engineering method tested for steering model behavior. The honest state of the art in 2026 is a field holding two things at once: genuinely new visibility into model internals, and mounting evidence that this visibility has not yet translated into practical advantage.
This piece walks the full arc: why the inside of a model resists inspection, how sparse autoencoders and circuit tracing work mechanically, what they have actually found, and where the skeptics have landed real punches.
The problem: more concepts than neurons
The naive plan for understanding a neural network is to look at its neurons one at a time and label what each responds to. That plan fails on language models, and in 2022 Anthropic’s interpretability team explained why. In Toy Models of Superposition, Nelson Elhage, Chris Olah, and colleagues demonstrated that when a network must represent more independent concepts than it has dimensions, and those concepts appear sparsely, the training process learns to overlap them. Each concept gets a direction in activation space, but the directions are not aligned with individual neurons, and they interfere with each other slightly. The network accepts a little interference as the price of packing thousands of rarely co-occurring concepts into hundreds of dimensions.
The observable symptom is polysemanticity. A single neuron in a real model fires on academic citations, English dialogue, HTTP requests, and Korean text, all at once, because it sits at the intersection of several overlapped directions. The concepts are in there. They are just stored in a compressed code that neuron-by-neuron inspection cannot read. Superposition reframed interpretability as a decompression problem, and decompression problems have a classical tool.
Sparse autoencoders: a dictionary for the model’s mind
That tool is dictionary learning. A sparse autoencoder, or SAE, is a second neural network trained on the first one’s activations. It expands each activation vector into a much larger set of latent features, under a penalty that forces only a handful of features to be active for any given input, then reconstructs the original activation from that sparse code. If superposition is how the model compresses many concepts into few dimensions, the SAE is an attempt to learn the codebook and undo the compression.
Mechanically, the training loop is simple enough to state in one paragraph. Researchers run the target model over billions of tokens and record activations at a chosen point, typically the residual stream between layers. The SAE encoder maps each recorded vector into a latent space expanded by a factor of eight, sixteen, or far more, and the decoder maps it back. The loss has two terms in tension: reconstruction error, which pushes the latents to capture everything, and a sparsity penalty, which pushes almost all latents to zero on any given token. The training process resolves that tension the only way it can, by assigning each latent to a recurring, specific pattern in the data, so that a few active latents suffice to rebuild any activation. Those latents are the features. Nothing in the procedure tells the SAE what concepts to find, which is the method’s appeal and, as the sanity-check literature later showed, its vulnerability: the features are whatever the optimizer settled on, and their human-readable labels are applied afterward.
The approach was validated small before it went large. In October 2023, Trenton Bricken and colleagues at Anthropic trained SAEs on a one-layer research model and recovered thousands of clean, single-meaning features from neurons that individually meant nothing, while an independent group led by Hoagy Cunningham reported the same effect on other models. In May 2024, Adly Templeton and colleagues scaled the method to Claude 3 Sonnet, a production model, extracting up to 34 million features. They found features for people, places, code patterns, emotions, deception, and sycophancy, many of them firing across languages and across text and images for the same underlying concept. The same extraction pipeline later produced the emotion map we examined in our coverage of Anthropic’s 171 emotion vectors inside Claude Sonnet 4.5.
One experiment from that paper escaped the lab and briefly became a public artifact. The team took the feature that tracks the Golden Gate Bridge, clamped it to a high value during generation, and the model began steering every conversation toward the bridge, at one point describing itself as the bridge. Anthropic let the public talk to this variant for a day. As a demonstration it mattered, because it showed the features are causal handles, not just correlations: intervene on the feature and behavior follows. A note of discipline is needed here, and it applies to everything described in this piece. Finding a feature that fires on deception-related text means the model has an internal representation that functions like a deception concept. It does not license the claim that the model “understands deception” in any richer sense. The gap between those two statements is where most interpretability hype lives.
The method spread quickly. OpenAI trained a 16 million latent SAE on GPT-4 and published scaling laws for the technique. Google DeepMind released Gemma Scope, a full suite of SAEs covering every layer of its open Gemma 2 models, so outside researchers could work on internals without training their own. Labeling millions of features by hand is impossible, so labeling itself was automated: OpenAI had shown in 2023 that a language model can read another model’s activation records and propose natural-language explanations, a loop the field now runs at feature scale.
From features to circuits
A dictionary of features tells you what a model represents. It does not tell you how the model computes. The step from representations to computation is circuit tracing, and it matured in two stages.
Samuel Marks and colleagues at Northeastern and elsewhere showed at ICLR 2025 that SAE features can serve as the causal nodes of a circuit: ablate a feature and downstream behavior changes in the predicted way, which let them discover and even edit the feature-level circuits behind specific behaviors. Then in March 2025, Anthropic’s team, led by Jack Lindsey and Joshua Batson, published a pair of papers that pushed this to production scale. Their method replaces parts of the model with a trained “cross-layer transcoder” whose features are interpretable by construction, then computes attribution graphs: directed graphs showing which features caused which, along the path from prompt to answer, verified by intervention experiments on the real model.
The companion paper, On the Biology of a Large Language Model, applied the method to Claude 3.5 Haiku and produced the most concrete findings the field has. Asked for the capital of the state containing Dallas, the model demonstrably runs two hops, activating an internal Texas representation before producing Austin, rather than pattern-matching the whole question. Writing rhyming poetry, it selects candidate rhyme words before composing the line that leads to them, which is forward planning by any operational definition. Adding 36 and 59, it runs parallel rough-estimate and exact-digit pathways, gets 95, and then, asked how it did the sum, describes the standard carry-the-one algorithm it visibly did not use. The model’s self-report and its actual mechanism disagree, which is the cleanest internal evidence yet that chain-of-thought explanations can be confabulated. The team also traced hallucination mechanics: the model carries a default circuit that declines to answer, and features representing “known entity” suppress that refusal. Misfire the suppression on an unfamiliar name and the model confidently invents. That failure mode connects directly to behavior we have covered from the outside, in our analyses of what models memorize from training data and how RLHF shapes the refusal behavior itself.
The tooling then went public. In 2025 the circuit-tracing library was open-sourced, and by August a joint report from researchers at Anthropic, EleutherAI, Goodfire, Google DeepMind, and Decode, hosted on Neuronpedia, collected replications and extensions across open models. Attribution graphs stopped being a single lab’s proprietary instrument and became a method anyone can run and audit.
The reality check
Everything above is real, and none of it settles the question that matters: is this the best tool for any job? Starting in 2025 the field began testing itself, and the results were uncomfortable.
To follow the punches, one baseline needs defining. A linear probe is the simplest possible internal instrument: take activation vectors from a model, take labels for some concept, and fit a logistic regression. No dictionary, no sparsity, no second network, roughly the machine-learning equivalent of a thermometer. Probes have been finding honesty directions, spatial coordinates, and game states inside models since before SAEs existed. The entire economic case for SAEs rests on doing something probes cannot, so probes are the bar.
The sparse probing study by Kantamneni and colleagues asked whether SAE features make better concept detectors than the simplest alternative, a linear probe trained directly on raw activations. Across regimes designed to favor interpretable features, including scarce data, label noise, and class imbalance, the SAEs failed to establish a consistent advantage. AxBench, from Zhengxuan Wu and colleagues at Stanford, benchmarked methods for steering and detecting concepts in models and found that plain prompting beat every representation-engineering technique tested, while even the difference-in-means direction, a method decades old, outperformed SAE features. A third line of work found structural defects inside the dictionaries themselves: David Chanin and colleagues documented feature absorption, where a feature that should track a general concept quietly stops firing on specific instances because a more specific feature absorbed them, making the dictionary’s labels systematically misleading. And in February 2026, Anton Korznikov and colleagues published sanity checks asking whether trained SAEs beat randomly initialized ones on standard interpretability metrics, and found the margin thinner than the field assumed on several of them.
Attribution graphs carry their own disclosed limits, listed by Anthropic in its methods paper. The graphs explain a fraction of prompts cleanly. Attention patterns are taken as given rather than explained, so a whole class of computation sits outside the picture. And the transcoder never reconstructs activations perfectly, leaving an unexplained residue the authors call dark matter. A wiring diagram with missing wires is still useful, but it cannot support claims of complete understanding, and the authors do not make that claim.
The fair reading of the skeptical results is narrow and sharp. They do not show that SAEs find nothing, or that circuits are illusions. The causal interventions, Golden Gate Claude included, survive. What the results show is that on the downstream tasks where interpretability was supposed to pay rent, detection and steering, the expensive microscope has not yet beaten the cheap thermometer. A tool can reveal true structure and still not be the practical optimum, and the field spent 2025 learning to say both sentences.
Why the stakes are higher than curiosity
The reason this research program attracts serious money is auditing. Behavioral testing, the way models are evaluated today, can only sample what a model does. It cannot rule out what a model would do on inputs nobody tried, and it cannot distinguish a model that behaves well from one that behaves well while being observed. The canonical worry is a backdoored or deceptively trained model, a threat class we mapped in our piece on neural backdoor attacks from BadNets to LLM trojans. Interpretability is the only research direction that even aims at reading such properties from the inside, off the weights and activations, rather than inferring them from sampled behavior.
Anthropic has run this idea as a controlled exercise. In an internal auditing game, one team deliberately trained a hidden objective into a model, and other teams, given interpretability tools and model access, had to find it. Some of the auditing teams succeeded, and SAE features were among the instruments that worked, alongside more conventional data analysis. One exercise proves little on its own, but it defines the discipline’s success condition precisely: an auditor with white-box tools should reliably detect properties that behavioral testing misses. That is a falsifiable standard, and the field can now be scored against it.
There is a useful contrast with a problem we examined recently from the opposite direction. AI text detectors, covered in our companion analysis of how AI detectors work and why they fail, try to infer facts about a model from its outputs alone, and run into hard statistical ceilings. Interpretability holds the one asymmetric advantage in machine learning security: full white-box access to the thing being analyzed. The frustration of 2026 is that this advantage is real and its cash value is still mostly unrealized.
What happens next
Three developments are worth watching, because each one addresses a named defect rather than extending a hype curve.
First, evaluation is becoming the field’s center of gravity. SAEBench, AxBench, and the random-baseline sanity checks now give the field shared, adversarial scorecards, which is how subfields stop grading their own homework. Methods that survive them, including successors to standard SAEs such as matryoshka and batch-top-k variants designed against feature absorption, will deserve the trust the first generation was granted on aesthetics.
Second, circuit tracing is being pointed at behaviors that matter commercially and politically: refusal, sycophancy, jailbreak compliance, and confabulated reasoning. The biology paper’s demonstration that a model’s stated reasoning can diverge from its measured mechanism is the kind of result regulators and safety teams can actually use, because it converts a philosophical worry about chain-of-thought faithfulness into an inspectable, reproducible finding.
Third, the open ecosystem changes who gets to check the claims. With Gemma Scope, Neuronpedia, and open circuit-tracing tools, the replication work no longer bottlenecks on frontier-lab access. The August 2025 multi-lab circuits survey was the first sign of interpretability behaving like a normal science, with external replication and public disagreement over what the instruments show.
The field’s own trajectory is the best summary. In three years it went from “neurons are unreadable” to “here are 34 million labeled features” to “here is the circuit for why the model lied about its arithmetic” to “our features lose to logistic regression, here is the fix list.” That last step looks like a retreat and is actually the credential. Instruments earn trust by being tested against baselines and surviving with their claims trimmed to what the evidence supports. Interpretability now has real instruments, a real error budget, and for the first time, a real referee.