Hybrid Search & Re-ranking

📌 Merging Two Different Worlds

In the last lesson, we learned that the best architecture is Hybrid Search: running a Keyword Search (BM25) and a Semantic Search (Vectors) at the exact same time to get the best of both worlds.

But this creates a mathematical problem. A Keyword Search outputs scores like 42.5, while a Semantic Search outputs Cosine Similarity scores like 0.89. You cannot simply add these numbers together because they are on completely different scales. So how does the database fairly combine the two lists of results?

📌 Reciprocal Rank Fusion (RRF)

To combine the lists, engineers use a formula called Reciprocal Rank Fusion (RRF).

RRF completely ignores the raw mathematical scores. Instead, it looks at the Rank.

• If Document A is Rank #1 on the Keyword list, and Rank #4 on the Vector list, RRF calculates a massive final score for it.
• If Document B is Rank #2 on the Vector list, but didn't even show up on the Keyword list, it gets a moderate score.

By relying entirely on the ranking position, RRF elegantly merges the two lists into one final, highly accurate list of top documents.

📌 The Final Polish: Re-ranking

Even after a Hybrid Search, your Top 10 results might still be slightly out of order. This is because embeddings are a "lossy" compression—they squeeze an entire paragraph into an array of numbers, losing some nuance.

To fix this, modern RAG pipelines use a Two-Stage Retrieval process, ending with a Re-ranker (like Cohere's Rerank API).

A Re-ranker is a specialized LLM (a Cross-Encoder) that doesn't use vectors. Instead, it takes the user's exact query and the document text, puts them side-by-side in its context window, and literally reads them together to calculate an incredibly precise relevance score.

Why not just use the Re-ranker for everything?

Because it is extremely slow and expensive. You cannot run a Re-ranker against 1 million documents in your database—it would take hours. Instead, you use the fast Hybrid Vector search to instantly narrow the database down to the Top 20 results. Then, you pass just those 20 documents to the Re-ranker to perfectly reorder them before showing them to the user or sending them to the final Chatbot LLM.