Tokenization

Tokenization is the process of converting text into the numeric tokens a model can process. Modern systems use subword tokenization (BPE, WordPiece, or SentencePiece) which strikes a balance between character-level (long sequences, generic) and word-level (short sequences, hits unknown words).

The tokenizer is part of the model — different models tokenize the same string differently. "OpenAI's GPT" might be 3 tokens for GPT-4 and 4 tokens for Llama. This affects both context budget and output speed: a more efficient tokenizer fits more content per token.

Special tokens (<bos>, <eos>, <|im_start|>, etc.) signal structure to the model — chat formatting, instruction boundaries, end-of-sequence. Misformatted special tokens are a common cause of garbled output in DIY inference setups.

An operator fine-tuning Llama 3.1 with a custom chat template runs into garbled output after deployment — the model emits stray <|im_start|> fragments mid-sentence. The cause: the inference server (say, an older llama.cpp build) is loading a generic BPE tokenizer instead of the model's bundled SentencePiece tokenizer with its correct special-token mappings. The fix is verifying the tokenizer.json and tokenizer_config.json shipped with the model weights match what the runtime actually loads — a mismatch here silently corrupts chat formatting even when the model weights themselves are fine. This is also why swapping a fine-tune's base model without re-checking tokenizer compatibility is a common source of broken instruction-following after quantization.