The Right Answer Is Already There
ShopBot can now find meaning.
Give it a customer query and it converts the query to a vector, searches ChromaDB, and returns the closest product chunks. The pipeline works. Arjun has tested it a dozen times over two days and the results are, mostly, correct.
Mostly.
He runs one more test before closing his laptop. A query he has not tried yet.
"Do you have anything for a winter wedding?"
ChromaDB returns the embroidered anarkali suit. Similarity score: 0.51.
He stared at the score for a longer moment than the number deserved, the unsettling part not the 0.51 itself but the knowledge that ShopBot would take it and answer from it confidently anyway. The chai beside his keyboard has gone cold. He does not reach for it.
He does not close his laptop. He opens a new file.
The Problem With Always Returning Something
The anarkali suit is a festive garment. It is appropriate for weddings. But the customer asked specifically about a winter wedding — and zUdyog Fashion's catalog has no winter-wedding product. The woolen shawl is winter. The anarkali suit is festive. Nothing in the catalog is both.
The correct answer to "do you have anything for a winter wedding?" is honest. Not specifically — but here is what we have.
ShopBot's answer, without a retrieval threshold, is something else. "The embroidered anarkali suit in teal would be a wonderful choice for a winter wedding, offering a festive and elegant look." Fluent. Confident. Not grounded in anything winter-specific. The anarkali suit's chunk does not mention winter. It does not mention cold weather. It mentions festive occasions and weddings. The model read what it was given and answered from it — exactly as it was designed to do.
The Grounding Layer retrieved the wrong evidence. The model, faithfully, spoke from it.
This is Confident Drift resumed — not from the model's training data this time, but from a retrieved document that scored 0.51 and should not have been retrieved at all. The retriever handed the model a chunk that was adjacent to the truth, not the truth itself. The model cannot tell the difference. It never can.
A retriever that always returns something — regardless of how relevant that something is — is not a Grounding Layer. It is a noise pipe dressed as one. The Grounding Failure is invisible. The HTTP response is 200. The customer receives a confident wrong answer.
Arjun has seen this shape before. Thirty-one tickets. Two weeks ago.
The Opposite Failure
His first instinct is to raise the threshold. If 0.51 is too low, set the minimum to 0.90. Only retrieve chunks that are genuinely close.
He updates the retriever. Tests it.
"Something for a summer wedding."
The silk saree scores 0.87. The anarkali suit scores 0.84. Both are correct answers. Both are below 0.90. The retriever returns nothing.
ShopBot says it does not know.
It does know. The products exist. The catalog has two excellent answers to that query. But the threshold he set was calibrated for certainty rather than relevance — and certainty, in a 1,536-dimensional embedding space, is rare. A score of 0.87 on a product-query match is strong evidence of relevance. Discarding it because it did not reach 0.90 is a different kind of Grounding Failure: the evidence existed, the retrieval found it, and the system threw it away.
Two failure modes. Opposite directions. Same consequence: the customer does not get the right answer. Threshold too low — ShopBot answers from noise. Threshold too high — ShopBot refuses to answer from signal.
The calibration lives between them.
Finding the Right Threshold
Threshold calibration is not a formula. It is a decision made by testing queries against the specific catalog and observing where relevant results appear and irrelevant ones do not.
Arjun tests twenty queries — a mix of product-specific questions, occasion queries, care-instruction questions, and queries that have no answer in the catalog. He records the similarity score of the top result for each.
The pattern that emerges is clean enough to be honest about. Queries with correct top results score between 0.78 and 0.94. Queries with incorrect top results — where the retriever returns something adjacent but wrong — score between 0.48 and 0.68. There is a gap between 0.68 and 0.78 where no result lands.
That gap is the threshold. Set it at 0.75 and the retriever catches every incorrect result while preserving every correct one — for this catalog, at this stage.
This threshold will need recalibration when the catalog grows. A catalog of five products has a different similarity distribution than a catalog of five hundred. Book 2 covers evaluation-driven threshold calibration using the labeled RAGAS dataset. For now, 0.75 is the working number, arrived at by observation, not assumption.
The Retriever as an Abstraction
Before he writes the threshold logic into the codebase, Arjun pauses. He has a habit when he is about to write something that will be called from many places.
He asks himself one question. If I change how this works in six months, how much else breaks?
The answer determines the design.
He could write the retrieval logic as a direct ChromaDB query — the same way he tested it in Chapter 4 — and call ChromaDB's API from every place in the codebase that needs to retrieve. Twelve lines per call site. It works.
But Book 2 is already on the horizon. Fifty boutiques means more than five products means ChromaDB's local-file approach reaches its limits. Migrating to Qdrant Cloud, then, becomes a question of how scattered the storage calls are. He has been on projects where a database swap took three weeks because the database calls were everywhere with no abstraction. He does not intend to repeat that.
The retriever, in this codebase, will be a single function. It takes a query string. It returns relevant chunks above the threshold, or None. What happens between those two points — which database, which embedding model, which threshold — is hidden behind that interface. The chains and endpoints that come later will call retrieve(query). They will not know whether it talks to ChromaDB or anything else. When ChromaDB is replaced by Qdrant in Book 2, the retriever changes. The code that calls it does not.
Building the Retriever
# src/retriever.py
import chromadb
from langchain_openai import OpenAIEmbeddings
from dotenv import load_dotenv
from typing import Optional
load_dotenv()
embeddings_model = OpenAIEmbeddings()
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(
name="zudyog_products",
metadata={"hnsw:space": "cosine"},
)
SIMILARITY_THRESHOLD = 0.75
MAX_RESULTS = 3
def retrieve(
query: str,
chunk_type: Optional[str] = None,
product_id: Optional[str] = None,
) -> Optional[list[dict]]:
"""
Retrieve relevant product chunks for a query.
Args:
query: customer's question.
chunk_type: optional filter — restrict to one chunk type
(e.g. "care", "policy", "variants").
product_id: optional filter — restrict to a specific product.
Returns:
A list of chunks above the similarity threshold, or None if nothing qualifies.
"""
query_vector = embeddings_model.embed_query(query)
where: dict = {}
if chunk_type:
where["chunk_type"] = chunk_type
if product_id:
where["product_id"] = product_id
results = collection.query(
query_embeddings=[query_vector],
n_results=MAX_RESULTS,
where=where if where else None,
include=["documents", "distances", "metadatas"],
)
retrieved = []
for i, doc in enumerate(results["documents"][0]):
distance = results["distances"][0][i]
similarity = 1 - distance # ChromaDB returns distance; convert to similarity
if similarity >= SIMILARITY_THRESHOLD:
retrieved.append({
"text": doc,
"similarity": round(similarity, 4),
"metadata": results["metadatas"][0][i],
})
return retrieved if retrieved else None
def respond(query: str) -> str:
"""
Retrieve evidence for a query. If nothing qualifies above the threshold,
return the deliberate fallback that routes the customer to support.
"""
evidence = retrieve(query)
if evidence is None:
return (
"I don't have a product that matches that specifically. "
"Could you describe what you're looking for differently, "
"or contact our team at support@zudyog.com?"
)
# Evidence found — Chapter 7 will pass these chunks to the model
return "\n\n".join([f"[{e['similarity']}] {e['text']}" for e in evidence])
if __name__ == "__main__":
queries = [
"do you have anything for a winter wedding?",
"something light for summer",
"care instructions for the silk saree",
]
for q in queries:
print(f"Query: {q}")
print(respond(q))
print("")
The function signature carries everything the caller needs and nothing it does not. The threshold is a constant near the top of the file, easy to change in one place. The optional filters — chunk_type and product_id — exist for the next concern.
Output:
Query: do you have anything for a winter wedding?
I don't have a product that matches that specifically. Could you describe
what you're looking for differently, or contact our team at support@zudyog.com?
Query: something light for summer
[0.8934] Cotton kurta — ideal for summer, casual daily wear. Breathable fabric.
[0.8712] Linen co-ord set — summer casual, relaxed everyday wear. Lightweight.
Query: care instructions for the silk saree
[0.9102] Silk saree care instructions: dry clean only. Do not machine wash,
tumble dry, or wring.
The winter-wedding query returns the fallback. Not because ShopBot failed — because it correctly identified that nothing in the catalog qualifies. The summer query returns two relevant products above threshold. The care-instruction query returns the focused chunk from Chapter 5's attribute-level chunking, scoring 0.91.
Filtering Out Competing Chunks
Arjun runs the retriever for two days while he works on the prompt design. On the third day, he notices something in the query logs he had not anticipated.
A cluster of queries about the anarkali suit's return policy.
"Can I return the anarkali if it doesn't fit?" "Return window for anarkali suit?" "Exchange policy embroidered anarkali?"
For each of these, the retriever — even with the threshold — returns the anarkali's policy chunk first, but the second and third results are policy chunks from other products. He pulls the chunk texts and reads them.
The anarkali's policy chunk: "Return and exchange policy for Embroidered Anarkali Suit: Returns accepted within 7 days. Item must be unworn and in original packaging with tags. One free size exchange within 10 days."
The cotton kurta's policy chunk: "Return and exchange policy for Breathable Cotton Kurta: Returns accepted within 7 days. Item must be unworn and in original packaging with tags. One free size exchange within 10 days."
Almost identical. Both chunks are about returns. Both are seven-day policies. Their embeddings are very close in the vector space because their meaning is genuinely similar. When a customer asks about the anarkali's return policy, both chunks are nearly equidistant from the query. The threshold lets both pass. The model receives policy information from two different products and must decide which one to trust.
This is not catastrophic now — the model usually picks the correct one because the query mentions anarkali. It gets worse as the catalog grows. Ten products with similar return policies means ten nearly-identical policy chunks competing for retrieval on every policy query.
The fix is metadata filtering. Restrict the search before similarity runs, not after.
# When the chain knows the customer is asking about a specific product:
evidence = retrieve(query, product_id="p005") # only anarkali chunks
# When the question is clearly a policy question:
evidence = retrieve(query, chunk_type="policy")
Each filter narrows the search space. The similarity ranking is unchanged in shape — it just runs over fewer candidates. The competing-chunk problem disappears, and the retriever becomes faster as a side benefit.
The remaining question is who supplies the filter values. For direct queries — return policy for the anarkali — the product reference is in the query. For indirect queries — what is the return policy? — it is not. In Chapter 7, the chain extracts the product reference from the query when one is present and passes it as the filter. For now, the important fact is that metadata filtering exists, is supported by ChromaDB, and is the correct shape of the fix.
Krishna reads the two outputs side by side — unfiltered, then filtered.
"The unfiltered version has the right answer," he says. "The anarkali policy is first."
"It is."
"Then why does the filter matter?"
It is the right question. Arjun has thought about this.
"Right now, with five products, the model handles the noise. It reads three policies and picks the anarkali one because the query mentions anarkali. At five hundred products, the unfiltered retrieval might return three policy chunks from three different products — none of them the anarkali — simply because they happen to score slightly higher on the query phrasing. The filter guarantees correctness regardless of catalog size."
Krishna nods slowly. "So the filter is not fixing a current problem."
"It is preventing a future one."
"Drona would approve," Krishna says, without elaborating.
How Many Chunks to Return
Three is the default, and three has reasons.
Returning only one chunk — k=1 — makes the model fragile. If that single chunk is the wrong one, the model has no other context to compensate. Returning ten chunks — k=10 — produces the opposite problem. The model's attention is divided across more context than the answer needs, which dilutes the response and increases token cost without improving accuracy.
Three sits in the place where the top match plus two related chunks gives the model enough context to triangulate. The model reads the care instruction, sees the FAQ for the same product, sees the identity chunk, and produces an answer that is both correct and contextual. For specific retrievers built around a known query type — care questions, policy questions — k=2 or k=1 is fine, because the target chunk type is already filtered.
MAX_RESULTS = 3 near the top of the retriever file is the working default. Specific retrievers built around precise queries override it.
The Retriever Is Not the Answer
This separation is worth stating clearly because it is a frequent source of confusion for developers building their first RAG system.
The retriever's job is to find relevant evidence. It does not decide what the answer is. It does not generate language. It surfaces the chunks that the model will read.
When a customer asks does the silk saree need dry cleaning? — the retriever finds the care chunk with the dry-cleaning instruction. It hands that chunk to the next component. The next component — the model, with a well-designed prompt — reads the chunk and shapes the answer. "Yes, the silk saree requires dry cleaning only. Machine washing may damage the zari border."
The retriever found the raw material. The model rendered it into a response. These are separate jobs that fail separately and are debugged separately. If ShopBot returns wrong information, the question is always: was the right chunk retrieved? If yes, the failure is in the prompt or the model. If no, the failure is in the retrieval — embeddings, chunking, threshold, or a missing filter. Each layer fails differently. Each layer is debugged differently.
The Retriever as Decision-Maker
The retriever is the decision point of the Grounding Layer.
Most developers treat this section of the pipeline as a search engine. Point it at the vector store, retrieve the top results, pass them to the model. Done. That framing is wrong, and the thirty-one tickets from Chapter 1 are the evidence.
The retriever is where the Pramana Framework makes its first judgment — not what to say, but what is worth saying from. Every retrieved chunk is a claim: this evidence is relevant enough to ground the model's answer. A retriever without a threshold makes that claim about everything it returns, regardless of score. A retriever with a calibrated threshold makes that claim only when the evidence earns it.
The Grounding Layer does not retrieve everything and let the model decide what is relevant. It decides first. That decision — retrieve or withhold — is where Confident Drift is either stopped or allowed to resume. A model given bad evidence will speak from it fluently. It has no choice. The retriever is the only component in the pipeline that can prevent bad evidence from reaching the model at all.
A system that admits gaps is more trustworthy than one that fills them with noise. The support email is not a failure state. It is the Grounding Layer operating correctly — routing the customer to a human when the catalog cannot serve them.
A Reader Exercise
Run the retriever with ten queries of your own — a mix of queries the catalog can answer and queries it cannot. For each, record the top similarity score and note whether the retriever correctly accepted or rejected the result.
Then adjust SIMILARITY_THRESHOLD from 0.75 to 0.65 and repeat. Note how many incorrect retrievals now pass. Then try 0.85 and note how many correct retrievals are now rejected.
The goal is to feel the threshold-calibration problem directly — not to find the perfect number, but to understand why the number matters and why it must be chosen by observation, not assumption. Every catalog has a different similarity distribution. Every deployment needs its own calibration. [Effort: 25 minutes.]
What Just Happened
A retriever without a threshold always returns something — and something at 0.51 similarity produces Confident Drift from retrieved noise rather than training data. A threshold set too high discards valid evidence and breaks retrieval in the opposite direction. The calibrated threshold for this catalog is 0.75 — arrived at by testing twenty queries and seeing where the gap between correct and incorrect results actually lay. Below threshold, the retriever returns nothing and ShopBot routes the customer to support@zudyog.com. Above threshold, the Grounding Layer holds. Metadata filters narrow the search before similarity runs, preventing competing chunks from polluting the result set as the catalog grows.
The retriever is not a search engine. It is the gatekeeper of the Grounding Layer — the component that decides what the model is permitted to speak from. Get this decision right and Confident Drift has nowhere to enter. Get it wrong and the rest of the pipeline serves a falsehood faithfully.
Chapter 7 builds the prompt that tells the model exactly how to speak from what it has been given — and what to do when it is tempted to reach beyond it.