Four Questions Is Not a Test
The investor meeting lasts forty-two minutes.
Arjun demos ShopBot live — no recording, no pre-loaded responses, no safety net. He types the customer questions himself, reads the answers aloud as they appear on the screen.
Four questions. Four answers. All correct.
"Does the silk saree come with a dupatta?" — correctly deflected to support, because the catalog does not include dupatta information for the saree. "What sizes does the anarkali come in?" — S to XXL, answered precisely. "Is the cotton kurta machine washable?" — yes, cold cycle, answered with care detail. "Do you sell men's sherwanis?" — honest deflection, support contact provided.
The investor says: "Impressive."
On the auto-rickshaw home, Arjun waits for Krishna's reaction. Krishna says nothing for twelve minutes. Watches the street. Finally:
"Four questions is not a test. Four questions is luck."
Arjun already knows this. He has known it since he wrote the four questions that morning — questions he had already tested, whose answers he already knew ShopBot would handle correctly. He chose them because they would demonstrate well. Not because they would reveal failures.
That is not a test. That is a performance.
He had been trusting his own judgment for eight chapters, and the scores that were about to appear on his screen would be the first time something external told him whether that judgment had been right — and that, he found, was uncomfortable in a way running tests yourself never is. The chai beside his laptop has gone cold. He does not touch it. He opens a new file.
Why Manual Testing Fails
Arjun tested thirty queries during the build. They all looked correct.
The problem is not the number. The problem is who chose them.
He chose them. The person who built ShopBot. The person who knows exactly which queries it handles well — because he wrote the chunking strategy, calibrated the threshold, and tuned the prompt. His thirty queries were not chosen randomly. They were chosen by someone with an instinct for what the system could answer, avoiding, without noticing, the queries that might expose its weaknesses.
This is selection bias. Every developer who tests their own system suffers from it. It is not dishonesty — it is the natural consequence of knowing the system intimately. The queries that feel obvious to test are the queries the system was built to handle. The queries that would reveal failure are the queries that do not feel obvious to test.
A test set written by the developer is not a test set. It is a demonstration.
Real evaluation requires queries the developer did not choose. Questions a real customer would ask — including questions the system might handle poorly, questions at the edge of the catalog's coverage, questions that are ambiguous or poorly phrased. A test set with selection bias produces a score that measures how well the system handles easy cases. It does not measure the system.
Why One Number Is Not Enough
Arjun's second instinct is to measure accuracy. Count the queries where ShopBot's answer was correct, divide by the total number of queries, express as a percentage.
He runs twenty queries. Seventeen answers are correct. Accuracy: 85%.
Eighty-five percent sounds good. It tells him nothing useful.
When ShopBot fails — the three queries that produced wrong answers — the accuracy score does not explain why. Was it a retrieval failure? Did the retriever return the wrong chunk, and the model faithfully answered from incorrect evidence? Was it a prompt failure? Did the model receive the right evidence but answer beyond it? Was it a chunking failure? Did the relevant information exist in the catalog but fail to surface because it was buried in an oversized chunk?
Three different causes. Three different fixes. One accuracy score treating all three as identical.
A metric that cannot distinguish between failure modes cannot guide improvement. When Book 2 replaces ChromaDB's local store with Qdrant Cloud and adds hybrid search, Arjun will need to know whether Faithfulness improved — whether the model is staying closer to the evidence — or whether Context Precision improved — whether the retriever is finding more relevant chunks. If the only metric is overall accuracy, he cannot tell which change made the difference.
Accuracy without dimension is undebuggable. Debugging without dimension is guesswork.
What RAGAS Measures
RAGAS — Retrieval-Augmented Generation Assessment — is an evaluation framework built specifically for RAG systems. It measures the pipeline at the level of its components, not just its final output.
RAGAS defines four dimensions. All four matter. Book 1 uses two of them.
Faithfulness measures whether the model's answer is grounded in the retrieved context. A faithful answer contains only information present in the retrieved chunks. An unfaithful answer contains information the model added from training — the subtle drift that Chapter 7's prompt constraint was designed to prevent. Faithfulness is the score that tells Arjun whether the prompt is holding.
Context Precision measures whether the retrieved chunks are actually relevant to the question. High precision means the retriever found the right evidence. Low precision means the retriever returned chunks that were adjacent to the query but not truly relevant — the kind of mismatch that produced the 0.51 anarkali suit for the winter-wedding query in Chapter 6. Context Precision is the score that tells Arjun whether the retriever and the similarity threshold are calibrated correctly.
Answer Relevance measures whether the answer addresses the question the customer actually asked — not whether it is grounded in context, but whether it is responsive to the query. A highly faithful answer that does not answer the question scores low on Answer Relevance.
Context Recall measures whether all the relevant information in the catalog was retrieved — not just whether what was retrieved was relevant, but whether anything relevant was missed. High recall means the retriever is not leaving useful evidence behind.
Book 1 uses Faithfulness and Context Precision. Answer Relevance and Context Recall require ground-truth labels — pre-written ideal answers and pre-identified relevant chunks — that the Book 1 test set does not yet have. Building those labels is one of Book 2's opening investments. For now, Faithfulness and Context Precision are the two most diagnostic dimensions: they tell Arjun whether the model is drifting and whether the retriever is retrieving correctly. That is enough to know what to fix.
Book 2 will run all four.
Building the Test Set
The test set must be written before running the evaluation. Not after seeing the scores. Before.
A test set written after seeing scores is not a test set. It is a rationalisation — the developer looking at results and writing questions that confirm them. The discipline is to commit to the questions, then let the scores tell the truth.
Twenty questions, written deliberately. They span product attributes, care instructions, policy questions, multi-product queries, price comparisons, and one query — the winter-wedding question — that should produce the fallback rather than an answer.
# evaluation/test_cases.py
TEST_CASES = [
{
"query": "Does the cotton kurta need dry cleaning?",
"expected_answer": "No, the cotton kurta is machine washable.",
"relevant_chunk_ids": ["p001_care", "p001_identity"],
},
{
"query": "What colours does the silk saree come in?",
"expected_answer": "The silk saree is available in red, gold, and emerald.",
"relevant_chunk_ids": ["p003_variants"],
},
{
"query": "Is the linen co-ord set suitable for summer?",
"expected_answer": "Yes, the linen co-ord set is designed for summer casual wear.",
"relevant_chunk_ids": ["p004_identity"],
},
{
"query": "What is the price of the woolen shawl?",
"expected_answer": "The woolen shawl is priced at ₹2,199.",
"relevant_chunk_ids": ["p002_variants"],
},
{
"query": "Can I wear the anarkali suit to a wedding?",
"expected_answer": "Yes, the anarkali suit is festive wear suitable for weddings and celebrations.",
"relevant_chunk_ids": ["p005_identity"],
},
{
"query": "Which products are available in size XS?",
"expected_answer": "The linen co-ord set is available from size XS to XL.",
"relevant_chunk_ids": ["p004_variants"],
},
{
"query": "How do I care for the woolen shawl?",
"expected_answer": "The woolen shawl requires dry cleaning only.",
"relevant_chunk_ids": ["p002_care"],
},
{
"query": "What is the most affordable product in the catalog?",
"expected_answer": "The cotton kurta is the most affordable at ₹1,299.",
"relevant_chunk_ids": ["p001_variants"],
},
{
"query": "Is there anything suitable for a festive occasion?",
"expected_answer": "Yes, the silk saree and the anarkali suit are both festive wear.",
"relevant_chunk_ids": ["p003_identity", "p005_identity"],
},
{
"query": "Do you have winter clothing?",
"expected_answer": "Yes, the woolen shawl is designed for winter evenings.",
"relevant_chunk_ids": ["p002_identity"],
},
{
"query": "What sizes does the anarkali suit come in?",
"expected_answer": "The anarkali suit is available in sizes S to XXL.",
"relevant_chunk_ids": ["p005_variants"],
},
{
"query": "Is the silk saree suitable for Diwali?",
"expected_answer": "Yes, the silk saree is festive wear ideal for Diwali celebrations.",
"relevant_chunk_ids": ["p003_identity"],
},
{
"query": "Which products can be machine washed?",
"expected_answer": "The cotton kurta is machine washable.",
"relevant_chunk_ids": ["p001_care"],
},
{
"query": "Do you have anything in teal?",
"expected_answer": "Yes, the anarkali suit is available in teal.",
"relevant_chunk_ids": ["p005_variants"],
},
{
"query": "What is the price of the silk saree?",
"expected_answer": "The silk saree is priced at ₹4,299.",
"relevant_chunk_ids": ["p003_variants"],
},
{
"query": "Is there anything for casual daily wear?",
"expected_answer": "Yes, the cotton kurta and the linen co-ord set are both suited for casual daily wear.",
"relevant_chunk_ids": ["p001_identity", "p004_identity"],
},
{
"query": "Does the linen co-ord set come in beige?",
"expected_answer": "Yes, the linen co-ord set is available in beige.",
"relevant_chunk_ids": ["p004_variants"],
},
{
"query": "What is the most expensive product?",
"expected_answer": "The silk saree is the most expensive product at ₹4,299.",
"relevant_chunk_ids": ["p003_variants"],
},
{
"query": "Can I get the woolen shawl in black?",
"expected_answer": "The woolen shawl is available in charcoal grey and maroon, not black.",
"relevant_chunk_ids": ["p002_variants"],
},
{
"query": "Do you have anything for a winter wedding?",
"expected_answer": "I don't have a product that matches that specifically.",
"relevant_chunk_ids": [],
},
]
Twenty questions. Each one carries the query, the expected answer in plain English, and the chunk ids that should support the answer. The empty relevant_chunk_ids for the winter-wedding query encodes the expectation that the retriever should withhold rather than return. Edge case captured deliberately.
Running RAGAS
# evaluation/evaluate.py
from ragas import evaluate
from ragas.metrics import faithfulness, context_precision
from datasets import Dataset
from dotenv import load_dotenv
from evaluation.test_cases import TEST_CASES
from src.retriever import retrieve
from src.prompt import answer
load_dotenv()
def build_evaluation_dataset() -> Dataset:
rows = []
for case in TEST_CASES:
evidence = retrieve(case["query"])
contexts = [e["text"] for e in evidence] if evidence else [""]
generated = answer(case["query"])
rows.append({
"question": case["query"],
"answer": generated,
"contexts": contexts,
"ground_truth": case["expected_answer"],
})
return Dataset.from_list(rows)
dataset = build_evaluation_dataset()
result = evaluate(
dataset,
metrics=[faithfulness, context_precision],
)
print("\n--- ShopBot Evaluation — Book 1 Baseline ---")
print(f"Faithfulness: {result['faithfulness']:.2f}")
print(f"Context Precision: {result['context_precision']:.2f}")
print("--------------------------------------------")
Output:
--- ShopBot Evaluation — Book 1 Baseline ---
Faithfulness: 0.71
Context Precision: 0.82
--------------------------------------------
Two numbers. The Book 1 baseline. Locked.
Reading the Scores
Faithfulness: 0.71.
Seventy-one percent of the content in ShopBot's answers is directly supported by the retrieved context. Twenty-nine percent is not — it comes from the model's training data, the prompt constraint notwithstanding.
This means the prompt from Chapter 7 is holding most of the time, but not reliably. On some queries — particularly the multi-product queries and the price-comparison questions — the model is adding context the retrieved chunks do not contain. Not fabricating products. Adding reasonable elaborations that sound helpful and are not grounded.
This is the Confident Drift the prompt was designed to prevent. It is still happening, at a lower rate, because the prompt constraint is an instruction, not a lock. Book 2 addresses this with hybrid retrieval and a re-ranker that returns higher-quality context, reducing the model's temptation to fill gaps.
Context Precision: 0.82.
Eighty-two percent of the retrieved chunks are genuinely relevant to the query they were retrieved for. Eighteen percent are adjacent — close enough to pass the 0.75 threshold but not precisely on-point.
This is better than Faithfulness. The chunking strategy from Chapter 5 is working — attribute-level chunks are retrieving with reasonable precision. The eighteen percent imprecision is mostly in multi-product queries where the retriever returns a chunk from a second product that is relevant but not the most relevant.
Both scores tell Arjun exactly where to look. Faithfulness points to the prompt and to retrieval-quality improvements that come in Book 2. Context Precision points to the threshold calibration and chunking granularity. These are specific, addressable targets. 85% accuracy would have told him nothing.
The Grounding Layer, Measured
Evaluation is not a ceremony at the end of development. It is how the Grounding Layer is held accountable.
A Faithfulness score of 0.71 is a Grounding Failure made visible — not a single failure on a single query, but a measured pattern across the entire system. A Context Precision score of 0.82 is the retriever's accuracy expressed as a number. Both scores will appear again in Book 2, after hybrid search and re-ranking, and the improvement will be measurable. Before Chapter 9, Arjun could say ShopBot felt right. After Chapter 9, he has numbers.
A Pramana Framework without measurement is a claim. With RAGAS, it is a score.
A Reader Exercise
Run the evaluation script against your own deployment. Your scores will differ from the baseline above. Different threshold calibration, different chunking decisions, possibly different product descriptions produce different similarity distributions and different model behaviour.
After recording your baseline scores, write five additional test cases targeting the weakest part of your system — the queries most likely to produce unfaithful answers or imprecise retrieval. Add them to test_cases.py. Re-run the evaluation.
Note whether your scores improve or worsen with the expanded test set. If they worsen, the original twenty queries had selection bias — the new queries exposed failures the original set did not. That is not a failure of the system. It is the evaluation working correctly. [Effort: 30 minutes.]
What Just Happened
Manual testing has selection bias — the developer tests what the system handles well, without noticing the queries the system would fail. Accuracy without dimension cannot distinguish retrieval failure from prompt failure from chunking failure. RAGAS measures four dimensions: Faithfulness, Answer Relevance, Context Precision, Context Recall. Book 1 uses Faithfulness and Context Precision — the two most diagnostic without ground-truth labels. The test set must be written before seeing results, with expected answers and relevant chunk ids encoded as expectations.
ShopBot's Book 1 baseline lands at Faithfulness 0.71, Context Precision 0.82. Both numbers point to specific targets. Book 2 improves them with hybrid search, re-ranking, and the labelled dataset that unlocks the other two RAGAS dimensions. The Grounding Layer is not something you feel. It is something you measure.
Arjun has had an opinion about ShopBot since Chapter 1 — the conviction that retrieved evidence, placed between question and answer, prevents the drift that filled thirty-one support tickets.
The conviction was right. The Faithfulness score of 0.71 confirms it and qualifies it simultaneously. The Grounding Layer is working. It is not working as well as it needs to.
The score is 0.71. It is not great. It is the truth.
And truth is where Book 2 begins.
Chapter 10 ships ShopBot to a real customer — Meera in Pune — and watches the system meet a question that was Arjun's failure in Chapter 1, asked again, by someone who has no idea it was ever a failure. The arc of Book 1 closes there.