Classical NLP: Bag of Words & TF-IDF

📌 The Pre-LLM Era

To truly appreciate the magic of modern Generative AI (like Embeddings and Transformers), you have to understand how terrible computers used to be at reading text. Before 2017, computers had absolutely no concept of "semantic meaning." They only understood raw spelling and word counts. This was the era of Classical NLP (Natural Language Processing).

📌 Bag of Words (BoW)

The earliest and simplest way to turn human text into math for a computer was the Bag of Words model.

Imagine you have a sentence: "The dog chased the cat."
The algorithm literally just throws all the words into a bag and counts them: {"the": 2, "dog": 1, "chased": 1, "cat": 1}.

The Fatal Flaw: BoW completely ignores word order. To this algorithm, the sentence "The dog chased the cat" is mathematically identical to "The cat chased the dog" because the word counts are the exact same! It destroys all nuance and context.

📌 TF-IDF (The Early Search Engine)

To make Bag of Words slightly smarter, engineers invented TF-IDF (Term Frequency-Inverse Document Frequency). This was the math algorithm that powered early search engines in the late 90s and early 2000s.

It works in two steps:

1. Term Frequency (TF): How often does the word "Apple" appear in this specific document?
2. Inverse Document Frequency (IDF): How rare is the word "Apple" across all documents on the entire internet?

TF-IDF was brilliant because it automatically penalized filler words (like "the" and "and") giving them a score of near-zero, while boosting the mathematical importance of rare keywords (like "photosynthesis").

The Fatal Flaw: TF-IDF still doesn't understand meaning; it only understands spelling. If a user searched for "automobile," a TF-IDF search engine would completely fail to return a document that used the word "car," because they are spelled differently. (This specific flaw is why modern Vector Embeddings were invented, to map "car" and "automobile" to the exact same location in space!).