← Previous Next →

Tokens & Tokenization

📌 What is a Token?

LLMs do not read text letter-by-letter, nor do they strictly read word-by-word. Instead, they process text in chunks called Tokens.

Diagram showing the word hamburger chopped up into sub-word tokens: ham, bur, ger
Diagram showing the word hamburger chopped up into sub-word tokens: ham, bur, ger

A token can be:

  • An entire common word (e.g., "apple")
  • A syllable or sub-word (e.g., "ham", "bur", "ger")
  • A single character (e.g., "z", or a space)

Rule of Thumb: In English, 1 token is roughly equal to 4 characters, or about ¾ of a word. For example, a 100-token prompt is approximately 75 words.

📌 The Tokenization Pipeline

Before your prompt ever reaches the AI model, it must pass through a Tokenizer. The tokenizer's job is to convert your human-readable text into an array of numerical IDs that the machine can understand.

Flowchart showing Raw Text entering a Tokenizer, which outputs numerical IDs into an AI Brain
Flowchart showing Raw Text entering a Tokenizer, which outputs numerical IDs into an AI Brain

Why do we use tokens instead of just full words?

  • Efficiency: There are infinite possible words in all human languages, plus typos, slang, and code. If an LLM had to memorize a unique ID for every possible word, its dictionary would be infinitely large.
  • Flexibility: By using sub-word tokens (usually a vocabulary of about 50,000 to 100,000 tokens), the model can construct any word—even words it has never seen before—by combining smaller syllables together.

📌 Examples: How Text is Tokenized

Tokenization algorithms (like Byte-Pair Encoding or WordPiece) can be surprisingly quirky. Small changes in your text can completely change the tokens the model sees.

  • Common words are single tokens: "Apple" → [Apple] (1 token)
  • Capitalization matters: "apple" → [apple] (1 token), but "APPLE" might become [APP] [LE] (2 tokens) because all-caps words are less common in training data.
  • Leading spaces matter: " hello" (with a space) is often a completely different token ID than "hello" (without a space).
  • Punctuation is separate: "Hello!" → [Hello] [!] (2 tokens)
  • Non-English languages are expensive: Because LLMs are primarily trained on English, a short sentence in English might be 5 tokens, while the exact same sentence in Hindi or Japanese could be 15 tokens, because the model has to break non-English characters down into much smaller sub-pieces (often down to individual bytes).
← Previous Next →