Why Models Don't Read Like Humans

📌 The "Strawberry" Problem

If you ask an LLM, "How many 'r's are in the word strawberry?", it will frequently give you the wrong answer. Why? Because the model literally cannot see the letters.

To a human, "strawberry" is made of 10 distinct letters. To an AI, the Tokenizer instantly converts the word "strawberry" into a single solid block: [ID: 1543]. The model knows that [ID: 1543] is a red fruit, but it has zero visibility into the spelling of that token. This is why LLMs struggle with rhyming, poetry, and spelling games.

📌 Byte-Pair Encoding (BPE)

How does the Tokenizer decide which letters get grouped together into a token? Most modern models use an algorithm called Byte-Pair Encoding (BPE).

BPE scans gigabytes of training text and finds the most frequently adjacent characters. For example, 't' and 'h' appear next to each other so often that BPE merges them into a single token [th]. It keeps merging pairs until it hits its target vocabulary size (e.g., 50,000 tokens). This is why very common words become single tokens, while rare words are left shattered into tiny pieces.

📌 The Glitch Token Phenomenon

Because BPE is purely statistical and runs independently of the LLM training, it sometimes creates Glitch Tokens.

Imagine a random string of characters like SolidGoldMagikarp appeared millions of times in a bizarre corner of Reddit. The Tokenizer sees this frequency and creates a dedicated, permanent Token ID for it. However, the LLM itself never learned what SolidGoldMagikarp actually means.

If you force the LLM to process this Token ID, it hits a "blank" concept in its brain. It will often hallucinate wildly, repeat itself, or completely break down. These are artifacts of the tokenization process.