Bi-Encoder vs Cross-Encoder: Two-Stage Retrieval in RAG
A customer asked: what can I wear to a small office party in the evening?
The hybrid retriever from Chapter 3 returned five chunks, ranked. The silk saree at rank one. The Anarkali at rank two. The linen co-ord at rank three. ShopBot recommended the silk saree. Two minutes later, the customer bought the linen co-ord — by clicking through the catalog directly, from a different store.
The linen co-ord was the right answer. The retriever was not wrong. It was approximately right in a way that changed which product the customer was recommended.
The Valid Failure: The Third Result Was Right
Arjun went back through the previous week's failed and lukewarm queries — queries that returned an answer but did not lead to a click on the recommended product. In 31 of them, the chunk the customer eventually engaged with was the second or third result, not the first.
He tried the obvious fix: increase top-k, hand the model the top five. Faithfulness fell from 0.79 to 0.74. Context precision fell more sharply, from 0.85 to 0.78. The model, given more chunks, was distracted. It conflated attributes — the cotton kurta's machine-washable note appeared in one answer as a property of the silk saree. More context was not more grounding. More context was more confusion.
He tried a score threshold — filter out any chunk below 0.7 cosine similarity. Some queries returned three chunks; some returned one; some returned none, and ShopBot refused to answer queries where the correct answer was sitting at 0.68, just below the line. Faithfulness unchanged. Customer-helpful answer rate fell.
The problem was not the number of chunks. The problem was that the rankings were not always good enough at the top, where the model only looked.
The Valid Source: Why Bi-encoders and Cross-encoders Differ
A bi-encoder (the dense retriever in ShopBot) embeds the query into a vector once, and the catalog chunks into vectors once — typically at ingestion time. At query time, it computes cosine similarity between the query vector and the stored chunk vectors. The similarity score is a geometric closeness in vector space.
For the office party in the evening query, the hybrid retriever's scores were nearly identical at the top: saree at 0.0301, Anarkali at 0.0294, linen co-ord at 0.0290. The differences were below the noise of the two-signal fusion. The bi-encoder had no way to distinguish between them — it had never seen the query and the chunks together.
A cross-encoder takes the query and a candidate chunk and feeds them together as one input into a model, so the model can attend across both texts simultaneously. It produces a relevance score that reflects how specifically the chunk addresses that particular query. It costs roughly 50 times more per pair than a bi-encoder cosine computation. It cannot scale to five thousand catalog chunks per query. It can absolutely scale to twenty.
When given the office party in the evening query and the linen co-ord chunk (described as a clean piece tagged for office wear and short evening events), the cross-encoder scored the co-ord at 0.86. Given the same query and the silk saree chunk (described as festive and formal, suited for weddings), the cross-encoder scored the saree at 0.41. The saree was for a wedding, not an office party. The bi-encoder had scored them 0.0301 and 0.0290 — almost indistinguishable. The cross-encoder was decisive.
The Valid Knowledge: Two-Stage Retrieval
The architecture after Chapter 4: hybrid retrieval returns the top-20 fused candidates. The cross-encoder scores each of the 20 against the customer query. The top 3 by cross-encoder score go into the prompt. The model speaks only from the 3.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=512)
def rerank(query: str, candidates: list[Chunk], top_k: int = 3) -> list[Chunk]:
pairs = [(query, c.text) for c in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda item: item[1], reverse=True)
return [chunk for chunk, _ in ranked[:top_k]]
def retrieve_for_prompt(query: str) -> list[Chunk]:
candidates = retrieve_hybrid(query, top_n=20)
return rerank(query, candidates, top_k=3)
The cross-encoder did not always change the order. On queries where hybrid retrieval was already confidently right — cotton kurta in mint, price of the silk saree — the reranker confirmed the existing ranking. The rerank was an audit. Most audits found nothing wrong. The ones that did, found things that had been wrong all along.
RAGAS evaluation with cross-encoder reranking:
- Faithfulness: 0.79 → 0.83
- Context precision: 0.85 → 0.88
Four percentage points is the difference between the third result was right and we shipped the wrong product and the right result was first and we shipped what the customer wanted.
The cross-encoder has one limit: it can only reorder what hybrid retrieval has already found. When the right chunk is not in the top-20 candidates — when the retriever, in either signal, ranked the answer below the cutoff — the reranker has nothing to promote. That is a different class of failure, requiring corrective retrieval rather than reranking. Book 3's problem.
This article covers concepts from Book 2, Chapter 4 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.