The spec sheet says one million tokens. The model can accept one million tokens. It cannot reason over one million tokens, and the gap between those two facts is where a lot of long-context systems quietly fail. On the RULER benchmark, which strips away the easy lexical shortcuts that make retrieval look solved, one strong long-context model dropped from 99.3% accuracy at short context to 69.7% by just 32,000 tokens. At that same 32K, eleven of the tested models fell below half of their own short-context score. The advertised window is a capacity. The number that governs whether your application works is a different and much smaller number, and it is not printed anywhere.
That number has a name, effective context length, and it exists because of a specific, fixable-looking, stubborn problem in how transformers encode position. The interesting part is not that models degrade over long inputs. Everyone knows that. The interesting part is why every training-free fix for it eventually hits the same wall, and what a 2024 analysis from ByteDance found when it stopped trying to extend the window and asked instead why the window was short in the first place.
Why position is hard for a transformer at all
Attention has no inherent sense of order. It sees a bag of tokens and computes how much each one should attend to each other one, and without an explicit position signal it would treat a sentence and its reverse as identical. Every modern LLM injects that signal, and the dominant method is rotary position embedding, RoPE. Rather than adding a position vector, RoPE rotates the query and key vectors by an angle that grows with position. Each pair of dimensions gets a rotation whose speed is set by a frequency, so the relative angle between two tokens encodes how far apart they sit.
This is elegant because attention scores end up depending on relative position, which is what you want. It has one hard edge. During training the model only ever sees positions up to its training length, so it only ever sees rotation angles in a bounded range. Ask it to process a token past that length and RoPE produces rotation values the model was never trained on. The angles wrap into territory the weights have no experience with, and attention breaks in ways that show up as garbled retrieval and collapsed reasoning deep in a long input.
The interpolation era, and its ceiling
The first fix was Position Interpolation, introduced by Chen and colleagues at Meta AI in 2023. Instead of letting positions run past the training length, PI linearly down-scales the position indices so a longer sequence maps back into the range the model already knows. Feed it 32,000 tokens but tell the rotations they are spread across the original window. It works, and it extended RoPE models to 32K with minimal fine-tuning, but it blurs the high-frequency detail. Squeezing more positions into the same angular range means neighboring tokens get rotation values that sit closer together, and the fine-grained distinctions the model uses for local structure get compressed.
NTK-aware scaling addressed that by changing the rotary base frequency rather than uniformly scaling every position, which preserved high-frequency detail better but extrapolated less cleanly at the extremes. Then YaRN, from Bowen Peng and collaborators in 2023, combined the ideas. YaRN, short for Yet another RoPE extensioN, splits the frequency spectrum into parts and treats them differently: it interpolates the low-frequency dimensions that carry long-range information while leaving the high-frequency dimensions that carry local detail closer to their trained values. It also adds a temperature term to the attention logits to compensate for the distributional shift that extension causes. YaRN extended models to 128K and beyond with roughly a tenth of the tokens and a fraction of the fine-tuning steps that Position Interpolation required, and it became the default baseline for long-context adaptation.
These methods are real progress, and they are also where the story usually stops. The typical explainer recites the benchmark deltas, lists PI then NTK then YaRN, and concludes that context extension is a solved engineering problem with a few known trade-offs. The benchmark data says otherwise, and a closer look at why says something more interesting than any of the individual methods.
What actually degrades, mechanically
It helps to be specific about what breaks, because the failure is not random noise. Two mechanisms compound as inputs get longer. The first is the untrained-rotation problem already described: past the training length, RoPE produces angles the model never learned to interpret, so the relative-position signal between distant tokens becomes unreliable. The second is a shift in the attention distribution itself. As the number of tokens grows, the softmax over attention scores spreads across more competitors, and without compensation the model’s attention flattens, diluting its focus on the tokens that actually matter. This is why YaRN adds a temperature term to the attention logits alongside its frequency scaling: the frequency fix handles the position angles, and the temperature fix handles the distributional dilution. Getting one right without the other still leaves performance on the floor.
There is a further failure mode that the 2026 retrieval analyses single out, sometimes called attention-sink collapse, where the model over-allocates attention to the first few tokens of the sequence and starves the middle. Combined with the well-documented tendency for information placed in the middle of a long prompt to get lost, the result is a model that reads the beginning and the end of a long input well and the middle poorly. Put the fact the model needs near the midpoint of a 200,000-token prompt and watch the accuracy fall. None of these are exotic edge cases. They are the ordinary behavior of a transformer asked to work past the range it was trained on.
What RULER actually measures
The benchmark that made effective context length a mainstream term is RULER, from NVIDIA. It extends the familiar needle-in-a-haystack retrieval test into thirteen tasks across four categories: retrieval with multiple keys, multi-hop tracing, aggregation, and question answering. A model can ace single-needle retrieval and still collapse on RULER’s harder tasks at the same length, because finding one fact is far easier than tracking several non-adjacent facts and integrating them. RULER defines effective context length as the longest input at which a model still clears a reasonable baseline, and the results were sobering when it published and have held since.
A model advertising 128K often had an effective length around 64K. Many models claiming 32K were really usable somewhere between 4K and 16K. The effective length is routinely a half or a quarter of the advertised one, and the shortfall shows up well before the model reaches its limit. The 2026 retrieval surveys report the same pattern at larger scales: reasoning over context at 256K stays above 80% for essentially one frontier model, while the rest degrade sharply, which is why practitioners increasingly design for an effective ceiling around 200K to 400K rather than the million on the box.
The important nuance is that this is not a knock on any single model or a temporary limitation that the next training run erases. The pattern is structural. Extending the window with an interpolation scheme changes how positions are mapped, but it does not change how well the model learned to use the positions in the first place. That distinction is where the ByteDance work comes in.
The finding that reframes the problem
In a paper titled Why Does the Effective Context Length of LLMs Fall Short, researchers took a different route. Instead of proposing a better interpolation, they looked at the distribution of position indices the model actually sees during training, and found what they call a left-skewed position frequency distribution. Short-distance position relationships appear constantly during pretraining and post-training. Long-distance position relationships appear rarely. So the model gets abundant practice with nearby positions and almost none with distant ones, which means the far positions are severely undertrained regardless of how much total data the model saw.
This explains the ceiling that interpolation keeps hitting. Position Interpolation, NTK scaling, and YaRN all manipulate the mapping from token index to rotation angle. None of them adds training signal to the undertrained far positions, because they are inference-time transforms. You can rescale the angles as cleverly as you like, but if the model never learned to attend well across long distances, remapping the coordinates does not teach it. The problem was never really the interpolation. The problem is that the long-range positions were undercooked in training, and interpolation is a coordinate change layered on top of that deficit.
The paper’s own remedy underlines the point. Their method, STRING, shifts well-trained positions to overwrite the ineffective ones at inference, which sidesteps the undertrained region rather than trying to repair it. Applied without any additional training, it improved the latest large models by over ten points on RULER and InfiniteBench, and pushed an open model past a commercial baseline it had trailed. The lesson is not that STRING is the final answer. It is that the real gain sits in the training-time position distribution, and the methods everyone cites were optimizing a downstream symptom.
Not every dimension breaks at the same length
One reason the single-knob interpolation methods leave performance on the table is that RoPE does not fail uniformly across its dimensions. Each pair of dimensions rotates at its own frequency, so high-frequency dimensions complete many rotations across a short span while low-frequency dimensions barely turn across the whole context. That means different dimensions run out of trained territory at different lengths. A method that scales every dimension by the same factor over-corrects the ones that were still fine and under-corrects the ones that broke first.
Follow-up work in 2025 made this explicit. Dimension-wise approaches detect the effective length of each RoPE dimension separately and only manipulate the dimensions that actually need it, keeping the rest at their original trained values. Reported results have this beating YaRN and related baselines, and enabling context extension without continual training while staying compatible with the memory-efficient attention kernels that production serving depends on. The takeaway for the general reader is not the specific method but the diagnosis it confirms: treating position extension as one global scaling factor throws away the fact that the failure is dimension-specific, which is another way of saying the trained-position deficit is unevenly distributed.
Why bigger models do not simply fix it
The reflexive assumption is that scale solves this, that a larger model with more parameters reasons better over long context. The evidence is mixed in a way that reinforces the training-distribution explanation. RULER’s own analysis found that a well-scaled 8B model can outperform a 70B counterpart on long context, and the STRING work found that even a 70B model with scaled RoPE frequency and sufficient training data had an effective length of only 64K. Parameter count is not the binding constraint. How the model was exposed to long-range position relationships during training is.
The clearest confirmation comes from the compact models that specialize in it. A 7B model trained specifically for long context held 84% accuracy at 128K on RULER where a general-purpose 7B of the same family managed 14%, while both stayed at 92 to 94% on short 4K to 16K inputs. Same size, same base architecture, a fifty-point spread at long range that came entirely from targeted long-context training rather than from raw capacity. That is the training-distribution thesis stated as an experiment: give the model real practice with long-range positions and the effective length climbs to meet the advertised one. Skip that practice and the window is nominal.
This connects to a broader pattern this publication has traced across the architecture stack. The same tension between advertised capability and measured capability showed up in why coding benchmarks mislead, where the same model scored twenty points apart depending on the test scaffold running it. It shows up in quantization, where the headline compression ratio hides where accuracy actually goes. Effective context length is the same species of gap: a spec-sheet number that describes what the system can ingest, not what it can do.
How to measure it for your own workload
The published effective-length numbers are useful as a warning and useless as a specification, because they were measured on synthetic tasks that are not your task. RULER’s retrieval and tracing tasks are deliberately clean. Your workload has domain vocabulary, formatting noise, and a question distribution none of the benchmarks modeled. The effective length that matters is the one measured on your data, and getting it is less work than most teams assume.
The method is a position sweep. Take a set of questions you already know the correct answers to, embed the supporting evidence at a controlled depth in an otherwise irrelevant filler context, and vary two things independently: total context length and the depth at which the evidence sits. Score accuracy at each combination. What comes back is a two-dimensional map rather than a single number, and the shape of that map is the actionable output. Most models produce a characteristic pattern where accuracy holds near the start and end of the context and sags in the middle, with the sag deepening as total length grows.
Two details make the difference between a useful measurement and a misleading one. The filler has to be plausible domain text rather than random tokens, because random filler is trivially easy for the model to ignore and inflates the result. And the task has to require integrating more than one fact, because single-fact retrieval stays high long after multi-fact reasoning has collapsed, which is precisely the discrepancy RULER was built to expose. A model that aces your single-needle test at 200K may be at coin-flip accuracy on the two-hop version at the same length.
The payoff is a number you can design against. Once you know that accuracy on your workload falls off a cliff past, say, 80K, the architecture question answers itself: retrieve aggressively enough to stay under that line rather than trusting the window on the datasheet. That is a cheaper conclusion to reach through a weekend of measurement than through a production incident.
What this means for building with long context
The practical implications are concrete. If your workload sits comfortably under roughly 200K tokens, the claim-versus-effective gap does not bite and you can use the window as advertised. Above that, on most frontier models, naive long-context stuffing starts losing to retrieval. Feeding a focused, retrieved chunk-set into a shorter effective window typically beats dropping the entire corpus into a nominal million-token window, for the same total token budget, because the model reasons better over the range it actually learned. The mechanics behind why retrieval augmentation still wins here tie directly to the retrieval-pipeline coverage this publication has built, including the memory-architecture trade-offs in agent memory patterns and the accuracy-versus-cost map in full-context accuracy ceilings.
The deeper point is architectural and worth holding onto. The context window grew faster than the model’s ability to use it, because expanding the window is an interpolation trick applied at inference while learning to use the window is a training-time investment that most models did not fully make. Until the training-time position distribution changes, the gap between the advertised million and the usable two hundred thousand is not a bug that a patch closes. It is the shape of the technology. Design for the number you can measure, not the number on the box.