What an AI Token Actually Is

Ask somebody how much text they just sent to an AI model and they will answer in words. Every bill, every limit, and every "this conversation is too long" error is counted in tokens — and tokens are not words. They are not letters either. The gap between the two is why a document you thought was small gets refused, and why a bill can triple without anybody sending noticeably more text.

This is an explanation of what a token actually is, why the counting works the way it does, and the handful of consequences that matter if you are paying for any of it.

The short version#

A token is a chunk of text, usually a few characters long. Sometimes it is a whole word. Sometimes it is part of a word. Sometimes it is a single punctuation mark, or a run of spaces.

Here is a real sentence, split the way a current model actually splits it. Each · is a boundary between tokens:

The · invoice · arrived · on · Tuesday · and · nobody · could · explain · why · the · bill · had · trip · led ·.

16 tokens, 81 characters

Most of those tokens are whole words with the leading space attached — the space is part of the token, not separate. But "tripled" broke into trip + led, because "tripled" is not common enough to have earned a chunk of its own.

That is the entire idea. Common text gets big chunks. Uncommon text gets small ones.

Why not just count words?#

The obvious question. Words seem like the natural unit, and they are how humans think about text volume.

The problem is that a model needs a fixed, finite list of everything it can read or write. Every item on that list needs its own entry in the model's internals, and the list has to be decided before training starts. So the question becomes: which words go on the list?

Try it with words and it falls apart immediately:

• English has hundreds of thousands of words, and that is before names, brands, and jargon.
• You would need run, runs, running, ran and runner as five unrelated entries, with nothing connecting them.
• A typo — recieve — is not on the list, so the model simply cannot read it.
• A word coined after training, or a product name, is unreadable forever.
• Chinese and Japanese do not put spaces between words, so "split on spaces" does not even define what a word is.

Try it with individual letters instead and you fix all of that, but the sequences become enormously long — this paragraph would be about a thousand units instead of about seventy — and length is the single most expensive thing about running these models.

Tokens are the compromise. Big enough that ordinary text stays short, small enough that anything at all can be spelled out from the pieces.

How the list gets built#

The method has an unglamorous name — byte pair encoding — and a genuinely simple idea behind it. It was a data compression trick before anyone used it for language.

You start with a vocabulary of nothing but individual characters. Then you read an enormous amount of text and ask one question: which two neighbours appear side by side most often? Whatever wins gets glued together into a single new entry. Then you ask again. And again, fifty or a hundred thousand times.

Start:   t h e ␣ c a t ␣ s a t
         "th" is the most common neighbouring pair -> merge it

Step 1:  th e ␣ c a t ␣ s a t
         now "the" is the most common -> merge it

Step 2:  the ␣ c a t ␣ s a t
         ... and so on, ~200,000 times

What falls out is a vocabulary that has learned the shape of the language without ever being told any grammar. Common words end up as single entries because they kept winning the contest. Common fragments — ing, tion, un, pre — end up as entries too, which is what lets the model handle a word it has never seen by assembling it from parts.

It is closest to a printer's type case. The words you set constantly get their own pre-cast block; everything else you assemble letter by letter from the small drawers. Nothing is impossible to print. Some things just take more blocks.

This also explains one of the most-shared AI failures: models being unable to count the letters in "strawberry". The model never sees the letters. It sees three chunks — st, raw, berry — and counting characters inside a chunk it only knows as one indivisible unit is genuinely hard. It is not stupidity, it is the alphabet the model was given.

What this looks like in practice#

Real counts from the tokeniser used by current models. The pattern to notice is the last column — how many characters you get per token, which is really a measure of how efficiently that kind of text is stored:

Prose is the best case, at roughly five characters per token. Structured data is far worse. That single line of JSON — sixty-two characters — costs twenty-six tokens, because every brace, quote, colon and comma is doing its own work and none of them travel in common pairs.

The UUID is the extreme: thirty-six characters, eighteen tokens. A random identifier is by definition text that has never appeared before, so there are no learned chunks to use and it gets spelled out almost two characters at a time.

This is the practical lesson hiding in the table. If you are sending a model data rather than prose — logs, JSON, CSV exports, database rows — assume it costs roughly twice what the same volume of writing would. Stripping unnecessary punctuation and identifiers out of a payload before sending it is one of the few genuinely free savings available.

The ÷4 rule, and where it breaks#

The rule of thumb everybody repeats is divide the characters by four. It is a reasonable starting point and it comes from English prose, which is the one thing it is calibrated for.

On the mixed page of prose, JSON, code and Japanese I used to build the table above, it was 18% low. On a JSON payload it can be off by a factor of three — in the direction of undercounting, which is the expensive direction. It is a rule for guessing whether something is roughly a page or roughly a book, not for anything with money attached.

If you want the real number for a specific piece of text, the AI Tokenizer on this site runs the actual encoding in your browser and shows you every boundary. Nothing is uploaded — the whole vocabulary is downloaded to you and the counting happens on your machine.

The bit that is genuinely unfair#

Take one sentence — "The cat sat on the mat." — and its direct translation into three other languages. Then count it with three generations of tokeniser, oldest to newest:

The English row never moves. Everything else was paying a substantial surcharge to say exactly the same thing — and because you are billed per token and limited per token, that surcharge was real money and real capacity. Hindi cost more than five times what English cost for the identical sentence.

The reason is simply what the training text looked like. The merge contest is won by whatever appears most often, so a vocabulary built mostly from English text spends its entries on English, and everything else gets spelled out of the leftovers.

The good news is the trend. That last column is a deliberate fix — recent tokenisers were built with far more multilingual text, and the gap has narrowed enormously. It has not closed. If you are working in a language other than English, you are still paying more per idea than an English speaker, and it is worth knowing that rather than assuming your prompts are simply too long.

Why any of this matters#

Two things are counted in tokens, and they are the two things that constrain every AI system.

You are billed by the token. Both directions, at different rates, with the reply usually costing several times the question. The number that matters is not the price per million tokens in the vendor's table — it is that number multiplied by how many tokens your actual traffic turns out to be, which is exactly the figure people estimate with the ÷4 rule and get wrong.

The context window is measured in tokens. This is the model's working memory: everything it can consider at once. The trap is that the window holds the input and the reply. Fill it entirely with your question and there is no room left for an answer.

It is also easy to forget what is in there. A conversation resends its entire history on every turn, so turn twenty costs many times what turn one did. And in a real application the window carries far more than what the user typed:

What the user typed .......... 200 tokens
The system prompt ............ 800 tokens   (sent every single time)
Tool and function definitions  1,400 tokens (sent every single time)
Retrieved documents .......... 3,000 tokens
Conversation history ......... 6,000 tokens
                               ------------
Actually billed .............. 11,400 tokens

The user's question is under 2% of that. This is the single most common surprise on a first invoice: people estimate the tokens they can see and get billed for the ones they cannot.

Tokenisers are not interchangeable between vendors, and a count from one tells you very little about another. If you have measured your prompts against one model and then switch, re-measure rather than assuming the number carries over. The gap is not always small: as the multilingual table above shows, two tokenisers can disagree by a factor of two on identical text.

A note on Claude specifically#

Anthropic does not publish a tokeniser for the current Claude models. That means no browser tool, this site's included, can give you a real Claude token count — anything claiming to is quietly showing you a count from an OpenAI tokeniser and putting a different label on it.

The correct answer is the API's own count_tokens endpoint, which is free to call and, usefully, counts the entire request — system prompt, tool definitions and all — rather than a bare string. That is the number you are actually billed on, and it is the one worth building a budget from.

This matters more than it used to: Claude Opus 4.7 introduced a new tokeniser that can produce noticeably more tokens on the same text than the previous one. A cost baseline measured against an older model does not carry forward.

What to take away#

Tokens are a compression scheme that learned the shape of language by repeatedly gluing together whatever appeared side by side most often. That single fact explains everything else: why common words are cheap and identifiers are expensive, why JSON costs double what prose does, why models cannot count letters, and why some languages have historically cost several times more than others to say the same thing.

If you take one practical habit from this, make it the second-most boring one available: measure, do not estimate. Run a representative sample of your real traffic — not a clean example, the messy actual thing with its system prompt and its retrieved documents attached — through a counter, and build your budget from that. The ÷4 rule is fine for deciding whether something is big. It is not fine for deciding what it costs.