Hybrid Search: BM25 + Dense Retrieval Explained

Dense retrieval (embeddings) understands meaning but misses exact terms. BM25 matches exact terms but misses meaning. Hybrid search runs both in parallel and merges the results — closing the vocabulary gap that causes most production RAG failures.

The 22% Problem

ShopBot shipped with pure dense retrieval and answered 78% of customer queries correctly. The remaining 22% received the support email. The system was correct in its refusals — it did not hallucinate — but the customers were unhelped.

Analysing the 22% revealed a pattern. Dense retrieval had failed them — not because the relevant document did not exist in the database, but because the semantic distance between the query and the chunk was not close enough to surface it.

The root cause was vocabulary mismatch. A customer typing a four-character SKU code they had bookmarked retrieved nothing — the catalog stored the same product under a descriptive name, not the SKU. A customer asking for "something festive and warm" — a compound query spanning two attributes — found neither the silk saree nor the woolen shawl reliably, because a single retrieval pass could not satisfy both signals at once. Customers used the words their mothers used; the catalog used the words a product copywriter used. The embedding model mapped them to similar but not identical regions of the semantic space. The similarity score fell below the retrieval threshold. The document was there. The retriever could not reach it.

This is the vocabulary gap — and it is the most common cause of the 20–30% of queries that dense-only RAG systems fail.

BM25: Exact Match Retrieval

BM25 (Best Match 25) is a probabilistic term-frequency model that scores documents based on how often query terms appear in them, adjusted for document length and term frequency across the corpus.

BM25 does not understand meaning. It cannot match "warm fabric" to "woolen material." But it is extremely good at one thing dense retrieval is not: it will find a document that contains the exact string "K-1299" even when that SKU has no semantic neighbours in the embedding space.

BM25 answers the question: which documents contain these specific terms?

Dense retrieval answers the question: which documents are semantically similar to this query?

These are different questions. The answers overlap but are not identical.

Hybrid Search: Running Both

Hybrid search runs BM25 and dense retrieval in parallel and merges the result sets. The standard merge approach is Reciprocal Rank Fusion (RRF):

RRF score = Σ 1 / (k + rank_in_list)

For each document, take its rank in the BM25 results and its rank in the dense results. For each, compute 1 / (60 + rank) (60 is the standard constant). Sum them. Rank documents by the sum.

Documents that appear at the top of both lists score highest. Documents that appear at the top of one list but are absent from the other still score well. Documents that score poorly on both are buried.

The practical effect: hybrid search is more robust than either approach alone. It catches vocabulary-specific queries through BM25 and conceptual queries through dense retrieval.

The Cross-Encoder Re-Ranker

Hybrid search narrows the candidate set but does not guarantee the top result is the most relevant. The re-ranker fixes this.

A cross-encoder re-ranker takes the query and each retrieved chunk, concatenates them, and scores the pair — how relevant is this chunk to this query, specifically? This is computationally expensive (it cannot be pre-computed), but it is applied only to the top 20–30 hybrid results, making it tractable.

The re-ranker catches cases where the merger promoted a document that was high on keyword match but low on actual relevance. Its output is a re-ranked list where the top result is consistently the most relevant — not just the most frequently occurring or semantically closest.

The pipeline becomes:

Query
  → BM25 retrieval (top 20)
  + Dense retrieval (top 20)
  → RRF merge (top 30 unique)
  → Cross-encoder re-rank
  → Top 3 to LLM context

What This Fixes

The vocabulary gap is the most common cause of 78% recall becoming 90%+ recall. Hybrid retrieval with re-ranking reliably closes the gap between what users say and what your documents say — without requiring you to rewrite your knowledge base to match user vocabulary.

Book 2 of the RAG Mastery Series implements this full pipeline in production, with a labelled evaluation dataset to measure the improvement precisely. The 22% from Book 1 becomes 9% by the end of Book 2 — not by magic, but by measuring honestly and fixing what the measurement reveals.