← Previous Next →

Byte Pair Encoding (BPE)

📌 The Tokenization Compromise

In our earlier lesson on Tokens, we learned that AI models don't read words; they read mathematical Tokens. But how exactly does the AI decide where to draw the line to create a Token?

  • Option 1 (Whole Words): If we create a unique Token for every single word in the English language, the AI's dictionary becomes massive (millions of tokens), requiring too much compute power.
  • Option 2 (Single Letters): If we create a Token for every single letter (a, b, c), the dictionary is tiny (26 tokens), but the AI loses all context because the single letter "e" has no semantic meaning.

The industry solution is the "Sub-word Compromise," powered by an algorithm called Byte Pair Encoding (BPE).

📌 How BPE Works (The Merging Algorithm)

Byte Pair Encoding is the exact mathematical algorithm used by GPT-4 and Llama 3 to build their Token dictionaries. It is purely statistical.

Diagram showing the letters E and R being statistically merged into a new "er" token
Diagram showing the letters E and R being statistically merged into a new "er" token

Here is how it works during the AI's training phase:

  1. The algorithm starts by breaking the entire training dataset down into single characters: [ 'e', 'r', 's', 't' ... ]
  2. It runs a massive statistical scan across the text: "What are the two most frequently adjacent characters in this book?"
  3. It mathematically discovers that e and r appear next to each other constantly. It permanently merges them into a brand new Token: [er].
  4. It scans again, finds e and s, and merges them into [es].
  5. It repeats this process thousands of times, eventually building a highly efficient vocabulary of common prefixes and suffixes (like [ing], [est], [pre], [tion]).

📌 Handling Rare Words

The true brilliance of BPE is how it handles completely made-up or rare words without crashing.

Diagram showing a made up word being cleanly split into sub-word tokens
Diagram showing a made up word being cleanly split into sub-word tokens

If you feed an old-school Whole-Word Tokenizer the made-up word "superflurgal," the Tokenizer will crash and output an <UNK> (Unknown) error, because that word is not in its dictionary.

If you feed "superflurgal" to a modern BPE Tokenizer, it simply falls back to its merged sub-words! It smoothly slices the unknown word into recognizable chunks: [super], [fl], [ur], [gal]. The AI can now process the text without crashing, and it even understands the prefix [super]!

← Previous Next →