Tokens — Plainly

The unit these systems actually read, write and bill in. Not words, not characters, and the gap between them is where surprise invoices come from.

A token is a chunk of text

A model does not see letters and it does not see words. Before anything happens, your text is chopped into pieces called tokens, each of which the model has learned as a single unit. A token is usually a common word, a fragment of a longer word, a punctuation mark, or a space plus the word that follows it.

Roughly, unbelievable might become three pieces along the lines of un / believ / able, while the is a single piece because it appears constantly. Common text compresses into few tokens; unusual text sprawls into many.

Why not just use words?

Because words are a bad unit for a general system. Tokenisation solves four problems at once:

Unknown words. A word-based system hits something it has never seen and has no representation for it at all. A token-based one spells it out from familiar fragments, which is how these models handle new product names, surnames and jargon.
Typos. teh is not a word, but it is a perfectly reasonable sequence of pieces.
Other languages. Many languages don't delimit words with spaces at all, so "split on spaces" isn't available.
Code and structured text. Brackets, indentation and symbols carry meaning and need to survive the process.

The rule of thumb everyone repeats is out of date

You will find, in a great many places, the claim that one token is about four characters, or about three quarters of a word. That number entered circulation years ago, attached to a particular generation of tokeniser, and it has been copied forward ever since without being rechecked.

It is now materially wrong for current models. Anthropic describes a one-million-token context window as approximately 555,000 words and approximately 2.5 million characters. Work backwards from the vendor's own figures and you get:

Ratio	Widely repeated folklore	Implied by current figures
Words per token	~0.75	~0.56
Tokens per word	~1.3	~1.8
Characters per token	~4	~2.5

Derived by division from Anthropic's stated approximations (1M tokens ≈ 555k words ≈ 2.5M characters) on its models overview, fetched 11 August 2026. These are approximations of approximations: treat them as an order of magnitude, not a calculator.

That is not a rounding difference. If you budget a job using the old ratio, you can be out by something like forty per cent on token count, and therefore on cost. Anthropic states directly that its newer tokeniser produces roughly 30% more tokens for the same text than the generation before it.

The genuinely useful takeaway

Tokenisation is not a fixed property of language, it is a property of a specific model. Two models can read identical text and count it differently. Any ratio you memorise is attached to a model version, and it expires when that version does.

Why you should care

It is the unit you are billed in

Pricing is quoted per million tokens, and quoted separately for input and output. Output costs several times more than input on every current model, which is worth internalising: asking for a long answer is far more expensive than supplying a long question. Current rates are on Model facts.

It is the unit the context window is measured in

The window is a token budget, not a page count, and it is shared between what you send and what comes back. This is why the two concepts have to be learned together. → Context windows

It makes some content quietly expensive

Because compression depends on how ordinary the text is, cost per page varies more than people expect.

Cheap per page: plain English prose, common vocabulary.
More expensive: code, especially with deep indentation and long identifiers. JSON and XML, where structural punctuation is a large fraction of the content. Tables. Long unbroken strings such as IDs, hashes and base64.
Considerably more expensive: languages that were less represented in the tokeniser's training. The same meaning can cost noticeably more tokens in some languages than in English, which is a real and under-discussed fairness problem in how these systems are priced.

Don't try to count them yourself

Two specific traps.

Don't estimate from a character count. If the number matters for budgeting, ask the model's own API for the count. Anthropic, for instance, exposes a dedicated token-counting endpoint. Every guess you make is worse than an exact answer that is free to obtain.

Don't use another vendor's tokeniser. The popular tiktoken library is OpenAI's, and it is simply the wrong tool for Claude. Anthropic's own documentation puts the error at 15–20% undercounting on typical English text, and worse on code or non-English input. This is a common enough mistake to be worth naming: cross-vendor token estimates are not approximately right, they are systematically wrong in one direction.

Where this page stops

The single largest lever on token cost for anyone building is prompt caching, which can change the economics of a repeated prompt by an order of magnitude. That deserves its own page and has one planned. The other obvious follow-on is why models are bad at counting letters in a word, which is downstream of tokenisation and is a good demonstration that these limitations have mundane mechanical causes.

Sources

Word, character and token approximations, the 30% tokeniser increase, and the tiktoken undercounting range are all Anthropic's own figures, from its models overview and token counting documentation, fetched 11 August 2026. Ratios in the comparison table above are arithmetic on those published figures rather than independent measurements, and are labelled as such. Equivalent figures for other vendors have not been verified and are deliberately not stated here.

Next → Context windows · All → Concepts