What a tokeniser actually does

Ask someone how a language model reads text and they will usually say it splits the sentence into words. It does not. It splits text into tokens, and a token is a stranger thing than a word.

Words are a bad unit

Consider a vocabulary built from words. English alone would need hundreds of thousands of entries, and it would still fail the first time it met antidisestablishmentarianism, a typo, or a Nepali sentence.

Worse, the model would have no idea that run, runs and running are related. Each would be an unrelated row in a table.

Subwords

Modern tokenisers break text into pieces that are often smaller than words:

The pieces are learned from data, not chosen by a linguist. The algorithm starts with individual characters and repeatedly merges the most frequent adjacent pair, until it has as many entries as you asked for.

Common words survive as single tokens. Rare ones get assembled from parts. Nothing is ever out of vocabulary, because the character level is always there as a floor.

Where this leaks

This is invisible right up until it is not:

• Non-English text costs more. A Nepali sentence often needs two or three times as many tokens as its English translation, because the merges were learned from a corpus that was mostly English. You pay for that in context window and in API billing.
• Counting characters is unreliable. Asking a model how many letters are in a word is asking it to inspect something it cannot see directly.
• Whitespace matters. " the" and "the" are frequently different tokens.

The tokeniser is the only part of the pipeline that touches raw human text. Everything after it works on integers.

Whenever a model behaves oddly on unusual input (spacing, non-Latin scripts, long numbers) the tokeniser is worth checking first. It usually explains it.