Word, Sentence, and Document Embeddings

📌 Word Embeddings (The Old Way)

In the early days of AI (around 2013 with models like Word2Vec), researchers generated embeddings for single words. Every word in the dictionary had exactly one vector.

This led to a massive problem: Context. Consider the word "Apple".

• "I ate a delicious Apple."
• "I bought a new Apple iPhone."

In the old system, "Apple" had the exact same mathematical vector in both sentences. This meant the AI was constantly confused, unable to distinguish between the fruit and the trillion-dollar tech company, causing semantic search to fail frequently.

📌 Sentence Embeddings (Context is King)

Modern Transformer models solved the context problem by moving to Sentence Embeddings (also called contextualized embeddings). Instead of embedding one word at a time, you feed the entire sentence into the model.

The model reads the surrounding words before generating the final vector. Therefore, the vector for "I ate a delicious Apple" lands in the 'food' section of the vector space, while "I bought a new Apple iPhone" lands perfectly in the 'technology' section. This is why modern semantic search feels like magic.

📌 Document Embeddings (The Chunking Problem)

If sentence embeddings are great, shouldn't we just embed an entire 500-page book into a single vector? No.

If you try to squeeze the meaning of a 500-page book into a single array of 1,536 numbers, the meaning becomes "muddy". The vector will represent the general, high-level theme of the book, but it will completely lose the specific details (like a specific character's quote in Chapter 4). If a user searches for that quote, the search will fail.

The Solution: Chunking.

To embed large documents for AI, you must first break the document into small pieces called "chunks" (usually 300 to 500 words each, roughly the size of a paragraph). You then generate a unique embedding vector for each individual chunk. This ensures the AI captures and remembers every specific detail of your document.