Tokens

Lesson 4 started from a matrix of numbers, X, and showed what self-attention does with it. This lesson answers the question that comes before all of that: where do those numbers come from? A language model never sees the word “strawberry.” It sees a short list of integers, and everything downstream, every attention calculation, every prediction, operates on those integers alone.

A tokenizer is not a minor preprocessing step. It is a lossy compression layer that decides what a model can and cannot see, and its choices explain some of the strangest failures a model can produce.

This lesson explains what a token actually is, why models cannot simply read character by character or word by word, works through the standard token-building algorithm on a real toy example, and explains two well-documented model failures, plus the pricing structure of every commercial API, that all trace back to this same one layer.

Section 01Why text gets broken into pieces at all

Lesson 4 showed that self-attention computes a score between every pair of positions in a sequence, which makes its compute cost scale with the square of the sequence length, O(N²). This one fact rules out the two most obvious ways to feed text into a model.

Character by character would make N, the sequence length, enormous. A single paragraph could run to a thousand characters, and the O(N²) attention cost would make even short documents prohibitively expensive to process.

Whole words as single units creates a different problem. English alone has hundreds of thousands of distinct words, before counting names, typos, technical jargon, and words in any other language. A model’s embedding table, the lookup table that turns each token ID into a vector, has to have one row per possible token. A vocabulary that tries to cover every possible word grows unbounded, and the model still breaks completely the moment it sees a word it has never encountered before, called an out-of-vocabulary (OOV) term.

Subword tokenization is the compromise that lets a model have both a small, fixed vocabulary and the ability to represent any possible string. Common whole words get their own single token. Rare or unfamiliar words get broken down into smaller, reusable pieces, fragments that likely appeared inside other, more common words during training. This guarantees two things at once: the vocabulary size stays fixed and manageable, and any string at all, including a typo or a word invented five minutes ago, can still be encoded as some finite sequence of known tokens.

Section 02Byte Pair Encoding: how a real vocabulary gets built

The dominant algorithm behind most modern models, including the GPT, Llama, and Mistral families, is Byte Pair Encoding (BPE). It was originally published in 1994 by Philip Gage as a general-purpose data compression technique, and adapted for language modeling by Sennrich, Haddow, and Birch in their 2016 paper, “Neural Machine Translation of Rare Words with Subword Units.”

BPE builds its vocabulary bottom-up, through simple, repeated counting.

1. Start from individual characters. The base vocabulary is every unique character (or byte) that appears in the training text.
2. Count adjacent pairs. Scan the entire training corpus and count how often every pair of adjacent symbols occurs next to each other.
3. Merge the most frequent pair. Combine whichever pair occurred most often into one new, single symbol, and add that new symbol to the vocabulary.
4. Repeat. Recount, and merge again, over and over, until the vocabulary reaches a target size, for example 50,257 tokens for GPT-2, or roughly 100,000 for GPT-4.

A worked example

Take a tiny training corpus with these exact word counts: “hug” appears 10 times, “pug” 5 times, “pun” 12 times, “bun” 4 times, “hugs” 5 times.

The starting vocabulary is just the individual characters present: b, g, h, n, p, s, u.

First merge. Count every adjacent character pair across the whole corpus. The pair (u, g) appears 20 times total: 10 times inside “hug,” 5 times inside “pug,” and 5 times inside “hugs.” That is the most frequent pair, so it gets merged: (u, g) becomes the single new token “ug,” added to the vocabulary.

Second merge. Recount, now treating “ug” as one unit where it appears. The pair (u, n) appears 16 times: 12 times inside “pun,” 4 times inside “bun.” That becomes the new token “un.”

Third merge. The pair (h, ug) now appears 15 times, all from “hug” and “hugs.” That becomes the new token “hug.”

After just three merges, the vocabulary has grown from 7 single characters to include “ug,” “un,” and “hug” as whole reusable units, while rare combinations that never got merged stay broken into smaller pieces. This is the entire mechanism, repeated tens of thousands of times against a real training corpus, that produces the vocabularies used by production models. In practice, this process runs against highly optimized code. OpenAI’s open-source tiktoken library, written in Rust, achieves processing speeds 3 to 6 times faster than comparable pure-Python implementations, and its production encodings, such as cl100k_base (used by GPT-4) and o200k_base (used by GPT-4o), apply a fixed regular expression before merging even starts, to stop merges from accidentally bridging across unrelated boundaries like whitespace or punctuation.

Section 03Why models cannot count letters: the “strawberry” problem

A well-documented failure mode, even in large, capable models, is answering the question “how many r’s are in the word strawberry?” incorrectly. This is not a reasoning failure. It is a direct, mechanical consequence of tokenization.

Under OpenAI’s cl100k_base tokenizer, the string “strawberry” is not one token, and it is not eleven individual character tokens either. It is exactly three tokens:

Once tokenization runs, the model’s attention layers never see the letters s-t-r-a-w-b-e-r-r-y at all. They see the integer sequence [496, 675, 15717], and nothing else. The individual characters are gone, discarded before the model does a single calculation. For the model to correctly count the r’s in this word, it would have needed to learn, purely from patterns in its training data, that token 496 happens to represent a string containing exactly one “r,” and that token 15717 happens to contain two. That is an indirect, learned association about an opaque integer ID, not a direct perception of the letters themselves. The model is not being asked to count. It is being asked to recall a fact about a token ID it has no direct visual access to.

The same blind spot damages arithmetic. Number formatting inside a string gets tokenized based on frequency in the training corpus, not based on mathematical meaning, and that chunking is inconsistent. Comparing 9.11 and 9.9, a BPE tokenizer may split “9.11” into [“9”, “.”, “11”] and “9.9” into [“9”, “.”, “9”]. If the model’s attention ends up comparing the token for “11” against the token for “9,” and the embedding for eleven carries a stronger “larger number” signal than the embedding for nine, the model can conclude that 9.11 is greater than 9.9, purely because of how the fractional part happened to get chunked, with no connection to the actual decimal comparison being asked.

Section 04Why every API bills you per token

Every major commercial API, including OpenAI’s and Anthropic’s, prices usage by the token, not by the word or the character. That single fact makes the tokenizer a direct line item in production cost.

(Pricing compiled from official provider documentation as of 2026. Prices and tiers change; treat this table as a dated snapshot, not a permanent reference.)

This also means the same piece of writing can cost dramatically different amounts, depending purely on what language it is written in. Foundational tokenizers are trained on internet text that skews heavily toward English and other Latin-script languages, so their learned vocabularies contain long, efficient tokens for common English words, but far fewer efficient tokens for other scripts. When a tokenizer meets a character sequence it never learned an efficient token for, it falls back to smaller pieces, sometimes down to raw UTF-8 bytes.

An equivalent English and Vietnamese sentence, both meaning roughly the same thing, illustrates this directly. The English sentence “Artificial intelligence is transforming software engineering every day” runs 71 characters, all standard ASCII, encoding to exactly 71 bytes, and roughly 10 to 12 tokens under cl100k_base. Its Vietnamese equivalent runs only 65 characters, but because Vietnamese uses diacritics that each require multiple UTF-8 bytes, it encodes to 86 bytes. The word “đổi,” just 3 visible characters, requires 6 bytes on its own: the đ character alone costs 2 bytes, ổ costs 3, and i costs 1. Because a standard BPE vocabulary was never trained on enough Vietnamese text to learn “đổi” as a single efficient token, it gets fractured into several separate byte-level tokens instead of one clean word-level token.

This is not a minor inefficiency. Ahia et al. (2023), in “Do All Languages Cost the Same?,” found that some languages, Burmese among them, can require up to 11.6 times more tokens than English to represent the exact same information. Since API pricing, context window limits, and generation speed are all measured in tokens, that gap translates directly into real costs: up to roughly 11 times higher API bills for the same content, a 128,000-token context window that holds far less actual Burmese text than English text, and slower generation, since autoregressive models produce exactly one token per forward pass, so more tokens simply means more sequential passes before a response finishes.

Conclusion

Every failure and every cost structure covered in this lesson comes from the same root cause: a model does not see text. It sees whichever integers its tokenizer decided to assign, and those decisions were made once, ahead of time, based on frequency statistics in a training corpus. BPE’s simple merge-the-most-frequent-pair loop is what builds that fixed vocabulary, and once it is built, everything downstream, from a model confidently miscounting the letters in “strawberry,” to an invoice that costs eleven times more in Burmese than in English, follows directly and mechanically from how that vocabulary happened to chop up a particular string.

A deeper look at this same layer, including how WordPiece and Unigram build vocabularies differently from BPE, why some tokens become “glitch tokens” that a model was never properly trained on, and where byte-level, tokenizer-free architectures are headed, is covered in full later in this course, once more of the vocabulary needed to discuss those tradeoffs is in place. The next lesson picks the thread back up from Lesson 1’s third historical shift: how a model goes from predicting the next token, the raw mechanical objective this lesson’s tokens feed into, to following instructions and holding a conversation.