The algorithm that decides how ChatGPT, Claude, and every other large language model actually reads text was not invented for AI at all. It was published in 1994 in the C Users Journal as a general-purpose data compression trick, with no language model in sight. Understanding that origin explains more about why LLMs are good at some things and strangely bad at others, counting letters, doing arithmetic, handling languages other than English, than almost any other single piece of how these systems work.
The algorithm is byte pair encoding, and it sits at the front of every major LLM as the tokenizer: the component that converts raw text into the sequence of integers the model actually processes. The model never sees characters. It sees token IDs, and the boundaries of those tokens are decided entirely by BPE, working on statistics from a training corpus, before the model itself ever gets trained.
A Compression Algorithm, Repurposed
Philip Gage’s original 1994 byte pair encoding was designed to shrink files: scan a byte stream, find the most frequently occurring adjacent pair of bytes, replace every occurrence of that pair with a single unused byte value, and repeat. Each pass shrinks the data a little more, and the substitutions can be reversed to recover the original file exactly. It had nothing to do with language and was never intended to.
In 2016, Rico Sennrich, Barry Haddow, and Alexandra Birch adapted the same merge-based idea to a completely different problem in a paper called “Neural Machine Translation of Rare Words with Subword Units.” Machine translation systems at the time struggled with words they had never seen during training, foreign names, rare technical terms, misspellings, because their vocabularies were fixed lists of whole words. The paper’s insight was to apply Gage’s merge procedure to text instead of arbitrary bytes: start from individual characters, and repeatedly merge the most frequent adjacent pair into a new subword unit, building a vocabulary of common word pieces rather than whole words. A rare word the model has never seen intact can still be represented as a sequence of subword pieces it has seen. That reuse of a compression algorithm for vocabulary construction is the entire foundation of how modern LLM tokenizers work.
How the Merging Actually Happens
Modern implementations, including OpenAI’s tiktoken library, work at the byte level rather than the character level, which matters more than it sounds. Byte-level BPE starts with a base vocabulary of the 256 possible byte values and nothing else. Training the tokenizer means running the same procedure Gage described: scan a large text corpus represented as bytes, find the most frequent adjacent pair, assign it a new token ID, replace every occurrence with that ID, and repeat until the vocabulary reaches a target size, commonly in the range of 32,000 to 128,000 tokens for current models.
A small example makes the mechanics concrete. Suppose a training corpus consists heavily of the word “lower” and its relatives. Starting from individual bytes, the trainer notices that l and o appear adjacent more often than any other pair, so it merges them into a new token, lo. On the next pass, lo and w now form the most frequent remaining pair, so they merge into low. The process keeps going, merging low with e, then with r, until “lower” collapses from five separate byte tokens down to a single learned token, purely because that sequence of merges paid off most on the training data. A word the tokenizer has never seen as a whole, say “lowest,” does not need its own dedicated token. It gets represented as low plus est, two pieces the merges already built for other reasons, reused for a new combination. Nothing about this process knows what a word is. It only knows which byte sequences kept showing up together often enough to be worth a dedicated token.
Because the starting point is raw bytes rather than a predefined character set, byte-level BPE can represent absolutely any input, including text in scripts the tokenizer never saw during training, emoji, and malformed input, without ever needing an out-of-vocabulary or unknown token. Anything that cannot be matched to a learned merge simply falls back to its raw constituent bytes. This is also why encoding is reversible and lossless: decoding a token sequence back to text is just running the merges in reverse order, substituting each learned token for the byte pair it replaced.
The practical effect on ordinary text is substantial compression. Under OpenAI’s own documentation, English text averages roughly four bytes of original text per token, meaning a tokenizer turns a sentence of forty-odd characters into around ten tokens rather than forty. That compression is not a side effect. It is the entire point: shorter token sequences mean cheaper inference, since cost and compute scale with token count, and longer effective context, since a fixed context window holds more real text when each token carries more information. That token-to-context relationship is the same one that makes a claimed million-token context window mean different things in practice depending on how efficiently the tokenizer compressed the text filling it.
BPE Is Not the Only Approach
WordPiece, the algorithm behind BERT and its derivatives, builds a similar subword vocabulary but selects merges by a likelihood criterion rather than raw frequency, and it tokenizes new text with a longest-match-first strategy against the learned vocabulary rather than replaying merge operations in order. SentencePiece, widely used across Google’s models and much of the open-weight ecosystem, trains directly on raw sentences without assuming that whitespace marks word boundaries, which matters enormously for languages like Chinese, Japanese, and Thai that do not separate words with spaces the way English does. All three approaches solve the same underlying problem, representing an open-ended vocabulary with a fixed, finite set of tokens, and BPE has become the dominant choice for decoder-only models specifically because byte-level BPE’s guarantee of no unknown tokens fits the autoregressive, general-purpose use case cleanly.
Where the Vocabulary Size Actually Comes From
Vocabulary size is a real design decision with real tradeoffs, not an arbitrary round number. Early models like GPT-2 used vocabularies around 50,000 tokens. Llama 2 used 32,000. Llama 3 jumped to 128,256, a deliberate expansion aimed at improving compression on code and non-English text, since a larger vocabulary can afford to give common multi-character sequences their own dedicated token instead of forcing them through several merge steps. Models built for genuinely multilingual use, including several with heavy Chinese-language training data among the open-weight models this site has tracked, push past 64,000 specifically to give non-Latin scripts dedicated tokens rather than falling back to inefficient byte-level representations for entire languages.
That design decision has a direct, measurable cost attached to it, and it falls unevenly. A tokenizer trained predominantly on English text compresses English efficiently and compresses everything else worse, sometimes dramatically worse. A sentence that costs ten tokens in English can cost two or three times as many tokens to express the identical meaning in a language underrepresented in the training corpus, because the vocabulary never learned efficient multi-character merges for that language’s byte patterns. That gap is not cosmetic. It means users working in underrepresented languages pay more per query under token-based pricing, get less usable content inside a fixed context window, and hit rate limits sooner, all from a vocabulary decision made once, before the model itself was ever trained.
Why This Explains a Specific Category of LLM Failures
Tokenization is also the direct, mechanical explanation for why LLMs are unreliable at tasks that seem trivial to a human: counting the letters in a word, or doing careful multi-digit arithmetic. A model reasoning over token IDs does not see individual characters as separate symbols unless the tokenizer happened to split the word that way. A common word is frequently a single token, an opaque integer id, with no character-level structure exposed to the model’s attention layers at all. Asking the model to count a letter inside that token is asking it to reason about internal structure that its input representation deliberately discarded during tokenization.
Numbers get a specific, deliberate fix in most current tokenizers for exactly this reason. Following the approach Llama and several other model families adopted, many tokenizers split multi-digit numbers into individual digit tokens rather than letting BPE merge them into arbitrary chunks, specifically to address inconsistent splitting: without that rule, a number like 1024 might tokenize as one unit in one context and as two mismatched pieces in another, purely as an artifact of what happened to be frequent in the training corpus. Research specifically tracing tokenization’s effect on arithmetic, including the paper “Tokenization Counts: The Impact of Tokenization on Arithmetic in Frontier LLMs,” documents this connection directly: the same operation on numbers with different tokenization boundaries produces measurably different model accuracy, even though the underlying math problem is identical.
What Tokenization Gets Wrong
BPE is a frequency heuristic, not a principled model of language. It has no concept of morphology, grammar, or meaning. It merges whatever byte pairs happen to co-occur most often in the training corpus, which means the resulting vocabulary reflects the statistical accidents of that corpus as much as any linguistic structure. Two languages with genuinely similar grammar can end up with very different tokenization efficiency purely because one was better represented in the training data.
Code is a persistent weak spot. Programming languages are whitespace-heavy and syntactically dense in ways natural-language corpora are not, and a tokenizer trained mostly on prose can split code into awkward, inefficient chunks unless the training corpus specifically included enough code to teach the merge procedure code-appropriate patterns. Most current frontier tokenizers correct for this by including large amounts of code in the tokenizer training corpus directly, but it is a corpus decision, not something inherent to BPE that fixes itself.
The field has not treated tokenization as settled. Scaffold-BPE, presented at AAAI in 2025, targets a specific inefficiency where BPE’s merge order leaves behind low-value intermediate tokens that inflate the vocabulary without earning their place. SuperBPE, described in 2026 research as extending merges past traditional word boundaries entirely, and other work on boundless byte pair encoding both push on the assumption that tokenization should respect word or whitespace boundaries at all. None of this is a finished, agreed-upon replacement. It is an active research area attached to a piece of infrastructure most people using these models never think about.
What Happens Next
The most direct alternative under active research is removing the fixed vocabulary entirely: byte-level or character-level models that process raw bytes without a learned merge table, trading tokenization’s compression efficiency for the promise of never having a vocabulary-driven blind spot in the first place. These remain more expensive to train and run at equivalent quality today, which is why BPE variants still dominate production models. Vocabulary engineering, larger vocabularies, better multilingual coverage, code-aware training corpora, is the more immediate, incremental path labs are actually shipping, and it is why vocabulary size has become a real design parameter reported in model cards rather than an implementation detail nobody mentions. Tokenization sets how many units of work a piece of text becomes in the first place. What happens to each of those units once they exist is a separate design question with its own separate history of naive approaches breaking in predictable ways. The gap between a model’s reasoning ability and its tokenizer’s blind spots is not going away on its own, and every practitioner working with cost, context budgets, or multilingual deployment is already paying for that gap whether they know the mechanism behind it or not.
Leave a Reply