Failure Taxonomy in Production RAG

The 9% remaining at the end of Book 2 looked like one number. It was four problems wearing the same percentage.

ShopBot had closed from 22% failures (Book 1) to 9% by the end of Book 2 — through hybrid search, reranking, memory, and caching. Six hundred and nineteen queries out of six thousand seven hundred and twelve in the previous month had either been routed to support or abandoned without engagement. The 9.2% was real. What was inside it was not visible until Arjun sorted it by hand.

The Valid Failure: One Number Hiding Four Different Problems

Each of the four failure types required a different fix. A single architectural response to all of them would have addressed one and left the other three untouched. The classification was the prerequisite to the fix.

Pile one — vague queries (187 queries).

"Anything light." "Something nice for the weekend." "Pretty for a Sunday lunch." "What's new?"

The retriever returned mid-relevance products, scoring around 0.66 to 0.72 on the cross-encoder, none of them wrong, none of them right. Customers clicked away. The fix was not in the chunks. The chunks were fine. The fix was in the question — the system needed to generate a hypothetical answer first, embed that, and retrieve against an answer rather than against the query.

Pile two — type mismatch (248 queries).

"What jewellery goes with the silk saree?" "Can I wear the linen co-ord with the woolen shawl?" "What shoes would suit the Anarkali for a wedding?"

The retriever was working correctly — the saree chunks came back at 0.86, the highest score of any retrieval in the previous fortnight. The model received accurate evidence about the product. And then the model was asked a question that the retrieved evidence could not answer. The catalog did not store jewellery, shoes, or styling information. The system needed to recognise that retrieval had succeeded but the question type had failed, and route accordingly.

Pile three — the long tail (184 queries).

Multi-turn conversations where the structured memory had failed subtly. Customers referring to a product from two turns ago, beyond the active-SKU window. Queries that started in one language register and switched mid-sentence. Compound questions the multi-query decomposer had failed to detect because the conjunction was implicit.

What these had in common was that the standard chain — retrieve, rerank, memory, generate — was the wrong shape for them. They needed routing. Different code paths for different query shapes.

The cost line — not a failure pile, a number across all queries.

Every query, answered or refused, paid for an embedding API call. At the current load of ~1,000 queries a day, the per-query embedding cost was negligible. At the projected 50,000 queries/day target, it was ₹22,500 a month, ₹2.7 lakh a year. The off-the-shelf text-embedding-3-small was trained on billions of general-internet documents. A 500-product Indian-fashion catalog needed a fraction of those dimensions.

The Valid Source: Why These Four Cannot Be Fixed the Same Way

None of the four could be fixed by adding another component to the existing chain.

Book 2's improvements were all additive to the chain: BM25 added a second retrieval signal; cross-encoder added a judge; structured memory added state; caching added a layer. None of those changed the shape of the pipeline.

The four Book 3 problems could not be solved this way:

• Vague queries needed the chain to bend back — generate a hypothesis, embed it, retrieve against it.
• Type mismatches needed the chain to branch — detect that retrieval had the wrong type of evidence and re-retrieve or refuse.
• The long tail needed the chain to become a graph — route different query shapes to different node sequences.
• Embedding cost needed the chain to replace a component — swap the foundation the chain had used since Book 1 Chapter 3.

The Valid Knowledge: The Order Matters

Krishna had a rule for the ordering: impact, dependency, blast radius. Applied to the four piles:

1. HyDE first — vague queries, 187 failures. Lightest architectural move. Fastest delivery. The hypothetical generation sits in front of existing retrieval and does not change anything downstream.
2. CRAG second — type mismatches, 248 failures (the largest pile). Uses the same small-LLM infrastructure HyDE introduced. Adds a judge step between retrieval and generation.
3. LangGraph third — the long-tail routing failures. Depends on HyDE and CRAG already existing as branches; it formalises and extends the branching into a readable graph.
4. Fine-tuned bge-small → ONNX INT8 fourth — sits underneath all of the above. Replacing the embedding foundation last means the components built on top of it have already been tested and validated before the foundation changes.

The four piles from one day of sorting defined the architectural agenda for the rest of Book 3. The percentage stayed at 9.2% until the fixes were built. The taxonomy was not the fix — it was the prerequisite to the fix.

This article covers concepts from Book 3, Chapter 1 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.