Token

A token is the smallest unit of text a language model processes. Most modern models use subword tokenization, where common words map to single tokens and rare words split into multiple subtokens. As a rough rule, 1 token ≈ 0.75 English words ≈ 4 characters.

Tokens matter because models are billed (in cloud APIs) and constrained (in context window) per-token, not per-word. A 100K-token context fits roughly 75,000 English words. Code typically tokenizes denser than prose due to whitespace and operators.

Different model families use different tokenizers: GPT models use BPE; Llama uses SentencePiece; Mistral Nemo introduced the new Tekken tokenizer. Switching tokenizers between training and inference produces gibberish, which is why tools like llama.cpp ship with the model's tokenizer baked in.

An operator prepping a RAG pipeline over a 500-page PDF needs to budget context carefully: at roughly 0.75 words per token, a 375,000-word document translates to about 500,000 tokens, several times larger than most local models' context windows, forcing a chunking strategy instead of dumping the whole file in. Similarly, when comparing local inference throughput, someone benchmarking Qwen 3 Coder against a JavaScript codebase will notice token counts run higher than expected on the same file size compared to prose, because dense code with lots of punctuation and short identifiers tokenizes less efficiently than natural language, eating into the effective context budget faster than a word count alone would suggest.