How to Evaluate RAG by Route, Not Just Overall Score

A blended RAGAS score hides failures. Faithfulness 0.88 across all queries looks healthy. Sliced by route, it might show Route D at Faithfulness 0.79 — a compound failure affecting 9–10% of production traffic, invisible in the aggregate.

The Valid Failure

After adding HyDE, CRAG, and LangGraph routing in Book 3, ShopBot's blended RAGAS scores improved:

• Faithfulness: 0.88
• Context Precision: 0.91

The team considered the system production-ready. Then RAGAS Monday ran, and someone added a route column to the analysis for the first time.

Route D was pulling the blended score up to 0.88 by being averaged against the 91% of queries on Routes A–C that were performing well. Route D at Faithfulness 0.79 meant 21% of claims in Route D answers were not supported by retrieved context. 9% of production traffic × 21% unfaithful answers = roughly 1 in 50 production responses containing fabricated information.

What Route D Was

Route D was the compound failure path: a vague query (routed to HyDE) where the CRAG grader returned PARTIAL — meaning the retrieved chunk partially answered the query but was missing key information.

HyDE generated a hypothetical, retrieved a chunk. CRAG graded it PARTIAL. The expand_retrieve node ran and found additional chunks — but the expanded chunks were thematically related to the vague query rather than specific to the product the hypothetical had described. The model generated from three loosely-related chunks and filled gaps with training memory.

The fix was not in the retrieval — it was in the expand_retrieve node, which was not filtering expanded chunks by the original query's specific terms.

Implementing Route Tagging

Tag every query with its route in the LangGraph state:

class ShopBotState(TypedDict):
    query: str
    route: str           # A | B | C | D
    is_vague: bool
    crag_verdict: str
    sub_queries: list[str]
    chunks: list[dict]
    answer: str

def classify_node(state: ShopBotState) -> ShopBotState:
    state["is_vague"] = is_vague_query(state["query"])
    return state

def generate_node(state: ShopBotState) -> ShopBotState:
    # Derive route from state before generating
    state["route"] = derive_route(state)
    state["answer"] = generate(state["query"], state["chunks"])
    return state

def derive_route(state: ShopBotState) -> str:
    has_sub_queries = bool(state.get("sub_queries"))
    is_vague = state.get("is_vague", False)
    verdict = state.get("crag_verdict", "")

    if has_sub_queries:
        return "B"
    elif is_vague and verdict in ("PARTIAL", "IRRELEVANT"):
        return "D"  # compound failure path
    elif is_vague:
        return "C"
    else:
        return "A"

Building the Per-Route Evaluation Dataset

Log the route with every production query:

import csv
from datetime import datetime

def log_query_event(
    query: str,
    route: str,
    answer: str,
    chunks: list[dict],
    session_id: str,
):
    with open("query_log.csv", "a", newline="") as f:
        writer = csv.writer(f)
        writer.writerow([
            datetime.utcnow().isoformat(),
            session_id,
            route,
            query,
            answer,
            json.dumps([c["text"] for c in chunks]),
        ])

On RAGAS Monday, slice the log by route:

import pandas as pd
from ragas import evaluate
from ragas.metrics import faithfulness, context_precision
from datasets import Dataset

df = pd.read_csv("query_log.csv", names=[
    "timestamp", "session_id", "route", "question", "answer", "contexts"
])

for route in ["A", "B", "C", "D"]:
    route_df = df[df["route"] == route].sample(min(50, len(df[df["route"] == route])))

    if len(route_df) < 5:
        print(f"Route {route}: insufficient data ({len(route_df)} queries)")
        continue

    dataset = Dataset.from_dict({
        "question": route_df["question"].tolist(),
        "answer": route_df["answer"].tolist(),
        "contexts": [json.loads(c) for c in route_df["contexts"].tolist()],
    })

    result = evaluate(dataset, metrics=[faithfulness, context_precision])
    print(f"Route {route} ({len(route_df)} queries):")
    print(f"  Faithfulness:      {result['faithfulness']:.3f}")
    print(f"  Context Precision: {result['context_precision']:.3f}")

Per-Route Targets

Set targets per route, not just overall:

Route D is flagged WATCH because it is the most complex path. A drop below target triggers investigation before the next release, not after.

The Fix for Route D

After identifying Route D as the failing route, the 16-line fix added a specificity filter to the expand_retrieve node:

def expand_retrieve_node(state: ShopBotState) -> ShopBotState:
    # Use sub-queries from CRAG verdict context, not the original vague query
    specific_terms = extract_specific_terms(state["chunks"][0]["text"])
    refined_query = f"{state['query']} {' '.join(specific_terms)}"

    extra = retrieve(refined_query, top_k=5)
    # ...

Result: Route D Faithfulness moved from 0.79 to 0.86 in two weeks. Two files changed. The fix was targeted because the diagnosis was precise.

The Valid Knowledge

• Blended RAGAS scores hide route-specific failures: a 0.88 blended score with Route D at 0.79 is not a healthy system — it is a system with a concentrated failure affecting 9% of traffic.
• Route tagging costs nothing and enables everything: adding a route field to the LangGraph state and logging it turns every RAGAS Monday into a diagnostic session rather than a report card.
• The compound failure path needs a WATCH flag: any route that combines multiple conditional steps (vague + PARTIAL) will accumulate more failure modes than single-step routes. Monitor it more closely.
• Targeted fixes require targeted diagnosis: "improve Route D" is actionable; "improve the system" is not. Per-route evaluation turns a general quality problem into a specific engineering task.