← Previous Next →

WordPiece & SentencePiece

📌 The Limitations of BPE

In our previous lesson, we learned that BPE is the standard tokenization engine for OpenAI. However, BPE has a fundamental flaw: it requires text to be "pre-tokenized" by spaces before it can run its merging math.

If you are building an AI that only speaks English, this is fine, because English uses spaces between words. But what if you are training an AI to speak Japanese or Chinese, where sentences often do not use spaces at all? If there are no spaces, BPE completely breaks down. Let's look at how Google solved this.

📌 WordPiece (The BERT Classic)

Before modern LLMs, Google released a revolutionary AI called BERT. To tokenize the text, they invented WordPiece.

Like BPE, WordPiece merges characters into sub-words. However, instead of blindly merging the two most frequent characters, it uses a deeper statistical formula (Likelihood). It evaluates if merging two letters actually adds semantic value to the dictionary.

Diagram showing the word playing being split into "play" and "##ing"
Diagram showing the word playing being split into "play" and "##ing"

WordPiece is most famous for its strict formatting. If it slices the word "playing" into two tokens, it outputs: ["play", "##ing"]. The ## marker tells the AI, "Hey, this token is permanently attached to the word before it!"

📌 SentencePiece (The Multilingual Savior)

While WordPiece was great, it still struggled with non-English languages. To build truly global models (like Google Gemini and Meta's Llama), the industry shifted to SentencePiece.

Diagram showing a sentence where spaces are replaced by underscores, treating the whole sentence as one string
Diagram showing a sentence where spaces are replaced by underscores, treating the whole sentence as one string

SentencePiece is brilliant because it treats the "Space" just like the letter "A" or "B". It often replaces spaces with a special character (like _).

Instead of splitting text by spaces first, it treats the entire sentence as one massive, unbroken string of characters: [The_cat_sat]. It then runs the math algorithms directly on that raw string.

Because it doesn't care if spaces exist, it is the ultimate engine for Multilingual Tokenization. It can seamlessly tokenize English, instantly switch to Japanese (which has no spaces), and then switch to Arabic, all using the exact same mathematical pipeline without ever breaking!

← Previous Next →