Author(s): “The AI Engineer”

Originally published on Towards AI.

Why LLMs Predict Tokens Instead of Words

Here’s the friend link

The article explains that LLMs work on token units rather than words or characters, because word-level modeling requires an impractically large vocabulary and character-level modeling is too inefficient due to longer sequences. It describes how byte pair encoding (BPE) tokenizers build a manageable vocabulary by repeatedly merging frequent byte/character pairs, causing common words and affixes to become single tokens while rare words are split into known subparts. It then highlights a “tokenizer tax”: most tokenizers are trained on English-heavy data, so other languages can require dramatically more tokens for the same content, shrinking effective context windows and increasing real costs. Although newer tokenizers reduce some of this penalty (e.g., OpenAI’s o200k_base versus older models), true improvement requires retraining rather than a simple switch, since tokenization is baked into model training. Finally, it advises builders to measure cost and context using the actual tokenizer per language and to track effective context rather than advertised token limits, concluding that tokenization biases are measurable, published, and disproportionately paid by users in lower-resource language communities.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI