How LLM tokenization actually works

A language model has no notion of a letter or a word. Before your prompt reaches it, a tokenizer converts the text into a list of integers, and the model only ever sees those integers. Your bill and your context-window headroom are both counted in them, which is why a prompt that looks short can be expensive and a prompt that looks long can be cheap.

The tokenizers used by every major provider are variants of byte-pair encoding (BPE). The algorithm is short enough to describe in a paragraph, and almost every counter-intuitive thing about token counts falls straight out of it.

Byte-pair encoding in one paragraph

Start with a vocabulary of the 256 possible bytes. Take a very large corpus, find the pair of adjacent symbols that occurs most often, and add that pair to the vocabulary as a new single symbol. Repeat until the vocabulary reaches the size you want — around 100,000 entries for cl100k_base, around 200,000 for o200k_base. What you end up with is an ordered list of merges, usually called the ranks.

Encoding is then just replaying that list: split the text into chunks with a regex, and inside each chunk repeatedly apply whichever merge has the lowest rank until no more apply. Decoding is a lookup. There is no model involved, no randomness, and no ambiguity — the same text always produces the same ids for a given encoding.

Two things follow immediately. Frequent sequences became merges early and end up as single tokens. Everything else stays split into whatever fragments survived. That is the whole story, and the rest of this post is its consequences.

Six consequences worth knowing

1. The leading space is part of the token. " token" and "token" are different ids. Text in a corpus is mostly words preceded by spaces, so the space-prefixed form is the one that got merged. This is why concatenating strings without a space sometimes increases your token count — you have forced the encoder onto the rarer, more fragmented variant.

2. Case matters, and capitalised words cost more. token, Token and TOKEN are three different byte sequences. Lowercase mid-sentence words are the most common form in the corpus, so they are the ones that became single tokens. SHOUTING can cost several tokens per word.

3. Rare and technical words fragment. token is one token. tokenization is not — it arrives as a handful of pieces. Product names, identifiers, UUIDs and long chemical or medical terms fragment hardest, because they were never frequent enough in the corpus to earn a merge.

4. Numbers are unpredictable. Digits get grouped by whatever the merge table happened to learn, so 2026 may be one token while 20260618 is three or four. There is no clean rule; if you are pushing large volumes of numeric data, measure it.

5. Non-English text costs far more per character. English averages roughly 4 characters per token. Chinese, Japanese and Korean run closer to 1.5 — the same paragraph of meaning can be two or three times as many tokens. Cyrillic, Greek and Devanagari sit in between. This is not a policy decision; it is the corpus the merges were learned from.

6. Whitespace in code is not free. Indentation is characters, and characters are tokens. Modern encodings do include multi-space tokens precisely because code is such a large share of what people send, so four spaces is usually cheaper than four separate tokens — but a deeply nested file still spends real money on its own left margin. Minified JSON is measurably cheaper than pretty-printed JSON carrying the same data.

Why some counts are exact and others are estimates

OpenAI publishes its tokenizer. The vocabulary and merge ranks for o200k_base are downloadable, which means an exact count can be produced in your browser with no API call — that is what the token counter on the home page does for every GPT model, using OpenAI’s own BPE implementation compiled to JavaScript.

Anthropic, Google, xAI, DeepSeek and the hosted Llama endpoints do not publish a usable vocabulary file. xAI exposes a /tokenize-text API endpoint but no ranks you can run locally. For those models, any local counter — including this one — is applying a heuristic: roughly four characters per token for Latin script, roughly 1.5 for CJK and Hangul. It is close, and it is labelled as an estimate rather than presented as a bill.

That distinction matters more than it sounds. Anthropic changed tokenizers mid-family: Claude Opus 4.7 and later produce roughly 30% more tokens for the same text than 4.6 and earlier. A character-count heuristic cannot see a change like that. Treat estimated numbers as a planning figure with a margin, not as an invoice.

What to do with this

• Look at the split before you optimise it. The token visualizer shows the actual BPE boundaries for your text, so you can see which parts of a prompt are fragmenting.
• Measure the real file, not a sample. A PDF’s token count has almost no relationship to its page count once headers, footers and hyphenation are extracted — the PDF token counter reads the file locally and tells you.
• Put the stable text first. Every provider’s prompt caching works on a byte-identical prefix. A system prompt and tool schemas at the top, the volatile user turn at the bottom.
• Price the output, not just the input. Output tokens are typically 5–6× the input rate. A prompt-shortening exercise that leaves the response length untouched saves less than you expect; the full pricing comparison has both columns for every model.

None of this requires understanding the model. It only requires remembering that the thing you are billed for is not your text — it is a list of integers that a merge table produced from it.