Subtitle: Discover how AI combines meaning-based search, keyword matching, and Reciprocal Rank Fusion to find relevant products—even when customers describe what they want differently from the product catalog.
1. The Problem: Why Use Two Search Methods?
Imagine a customer visits Varastra, an Indian ethnic-wear store, and asks:
“Find me a lightweight outfit for a summer wedding.”
The catalog contains a product described as “breathable cotton kurta set.”
A keyword search may miss this product because the customer did not use the exact words “cotton kurta.” Semantic search may recognize that the descriptions have related meanings.
But semantic search has limitations, too. Exact product names, model numbers, or specific terms can be important.
Varastra's solution is hybrid retrieval: use both search methods, then combine their rankings into one list.
2. Understand the Architecture
Follow the arrows in the diagram:
Customer query → Two search branches → Two result lists → Reciprocal Rank Fusion → Fused top-10 results
The two branches are:
- Dense search: Finds semantically similar content.
- Sparse search using BM25: Finds relevant keyword matches.
Both search branches use Qdrant. Their results are combined by Reciprocal Rank Fusion (RRF).
3. Step 1: Convert the Customer Query into Two Representations
The customer submits a single question. The system processes it in two ways.
Branch A: Dense Embedding
The diagram specifies text-embedding-3-small with 1,536 dimensions.
An embedding is a list of numbers representing information about text. Texts with related meanings can have embeddings that are close together in vector space.
For example:
- Customer query: “Lightweight outfit for a summer wedding.”
- Catalog description: “Breathable cotton kurta set for warm weather.”
The wording differs, but the meanings overlap.
The embedding model converts the query into a vector that can be used for semantic search.
Branch B: BM25 Sparse Encoding
The second branch uses FastEmbed to produce a sparse representation for keyword-based retrieval.
BM25 is a ranking method that scores documents based on how well their terms match the query. It considers factors such as term frequency and how common a term is across the collection.
For example, a search for a specific product name may benefit from matching its exact words.
Why two representations? Dense embeddings help match meaning; sparse retrieval helps match important terms.
4. Step 2: Search Qdrant Through Both Branches
The diagram shows two separate retrieval operations.
Dense Search
Qdrant dense search → Top 20 by cosine similarity
Qdrant compares the query embedding with stored product embeddings and retrieves the top 20 candidates according to cosine similarity.
Cosine similarity measures the directional similarity between vectors. A higher score generally indicates greater similarity under this setup, but it is not a percentage of correctness.
Sparse Search
Qdrant sparse search → Top 20 by BM25
The sparse branch retrieves the top 20 candidates using the BM25-based ranking.
These candidates may overlap with the dense results, or they may include products that dense search ranked lower or missed.
At this stage, there can be up to 40 candidate entries across the two lists, but some products may appear in both.
Important: The top-20 limits are the settings shown in this diagram, not universal requirements for every search system.
5. Step 3: Combine the Rankings with Reciprocal Rank Fusion
Now the two result lists enter Reciprocal Rank Fusion (RRF).
RRF combines ranked lists without requiring the original search scores to be on the same scale.
The standard RRF formula is:
RRF score(d) = Σ 1 / (k + rank(d))
Where:
dis a candidate document or product.rank(d)is its position in each result list.kis a constant that reduces the influence of rank differences.- The scores from lists containing the candidate are added together.
The diagram specifies k = 60.
A Simple Example
Suppose the two search methods return these results:
| Product | Dense rank | Sparse rank |
|---|---|---|
| Cotton kurta A | 1 | 4 |
| Linen set B | 5 | 1 |
| Festive kurta C | 2 | Not in top 20 |
| Cotton kurta D | Not in top 20 | 3 |
Using k = 60, the RRF score for Cotton kurta A is:
1/(60 + 1) + 1/(60 + 4)
≈ 0.01639 + 0.01563
≈ 0.03202
Linen set B receives:
1/(60 + 5) + 1/(60 + 1)
≈ 0.01538 + 0.01639
≈ 0.03177
Cotton kurta A narrowly ranks above Linen set B in this illustration because its combined positions produce the higher RRF score.
The example is illustrative; actual rankings depend on the real candidate lists.
Why RRF Is Useful
Dense search and BM25 can produce scores with different meanings and scales. Directly adding their raw scores may not be appropriate.
RRF instead uses each candidate's position in each list.
A product that ranks well in both methods can receive a strong combined score. A product that ranks highly in only one list can still make the final ranking.
6. Step 4: Return the Fused Top 10
The diagram specifies:
k = 60 → fused top-10
After calculating RRF scores, the system sorts the candidates and selects the top 10.
These are the combined retrieval results available to the next stage of Varastra's AI pipeline.
They are not automatically the final customer-facing answer. A later stage may rerank candidates, apply business rules, verify product availability, or use retrieved information to generate a response.
Key distinction: RRF improves the combined ranking. It does not guarantee that every selected product is relevant or available.
7. What Can Go Wrong?
| Problem | Why it matters |
|---|---|
| Dense search misses an exact product | Important names or identifiers may not rank highly semantically. |
| Sparse search misses a meaning-based match | Customers may use different words from the catalog. |
| Both methods return poor candidates | RRF cannot recover a relevant product that neither method retrieves. |
| Candidate rankings disagree | The fused order may differ from either individual list. |
| Catalog data is outdated | Relevant retrieval does not guarantee current prices or stock. |
The team should evaluate both retrieval branches and the fused results using representative customer queries.
RRF's k value and the number of candidates to retrieve should be tested rather than assumed to be optimal.
8. Key Takeaways
- Dense search matches meaning using embeddings.
- BM25 supports keyword-based ranking through sparse retrieval.
- Qdrant performs both dense and sparse searches in this architecture.
- Each branch retrieves its top 20 candidates.
- Reciprocal Rank Fusion combines rankings using
k = 60. - The fused list is limited to the top 10 results.
- RRF combines rankings; it does not guarantee relevance or factual correctness.
- Good retrieval still depends on catalog quality, evaluation, and appropriate downstream validation.
Remember: Dense search understands related meaning, sparse search helps find matching terms, and RRF combines their rankings into one list. Together, they give Varastra a stronger retrieval foundation than relying on either search method alone.