CRAG: Corrective RAG Explained with Python Code

A retriever can return the right chunk at the wrong relevance level. The model does not know this — it generates from whatever it receives. CRAG (Corrective RAG) sits between retrieval and generation and grades the evidence before the model speaks from it.

The Valid Failure

ShopBot's retriever scored the silk saree chunk at 0.86 for a query about silk fabrics — the highest retrieval score in a fortnight. The chunk contained accurate product information. The model received it.

Then it generated a response that included jewellery recommendations. No jewellery was in the retrieved chunk. The model had correctly read the fabric information and then continued from training memory — elaborating beyond what it was given.

The retrieval score was 0.86. The retrieved chunk was relevant. The failure was a type mismatch: the chunk answered the fabric question, but the query also implicitly expected accessory styling advice that was not in the catalog.

The problem was not the chunk. The problem was that the system had no mechanism to check whether the chunk actually answered the question before passing it to the model. CRAG provides that mechanism.

How CRAG Works

CRAG inserts a grading step between retrieval and generation:

Query → Retrieve chunks → Grade each chunk → Route based on grade → Generate
                              ↓
                   RELEVANT → pass to LLM directly
                   PARTIAL  → retrieve additional chunks from web or wider index
                   IRRELEVANT → discard, retrieve again or refuse

The grader is a small LLM call that reads the query and each retrieved chunk and returns one of three verdicts.

Implementation

from openai import OpenAI
from enum import Enum

client = OpenAI()

class CRAGVerdict(str, Enum):
    RELEVANT = "RELEVANT"
    PARTIAL = "PARTIAL"
    IRRELEVANT = "IRRELEVANT"

GRADER_PROMPT = """You are a retrieval quality grader.

Query: {query}

Retrieved chunk:
{chunk}

Grade this chunk for relevance to the query.
Return ONLY one word:
- RELEVANT: the chunk directly and fully answers the query
- PARTIAL: the chunk partially answers the query but is missing key information
- IRRELEVANT: the chunk does not address the query

Grade:"""

def grade_chunk(query: str, chunk: str) -> CRAGVerdict:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[
            {"role": "user", "content": GRADER_PROMPT.format(
                query=query, chunk=chunk
            )}
        ],
    )
    verdict_str = response.choices[0].message.content.strip().upper()
    try:
        return CRAGVerdict(verdict_str)
    except ValueError:
        return CRAGVerdict.PARTIAL  # default to PARTIAL on unexpected output

def crag_retrieve_and_grade(query: str) -> tuple[list[dict], CRAGVerdict]:
    chunks = retrieve(query)
    if not chunks:
        return [], CRAGVerdict.IRRELEVANT

    # Grade the top chunk (most relevant)
    top_chunk = chunks[0]
    verdict = grade_chunk(query, top_chunk["text"])
    return chunks, verdict

Routing Based on Verdict

def crag_pipeline(query: str) -> str:
    chunks, verdict = crag_retrieve_and_grade(query)

    if verdict == CRAGVerdict.RELEVANT:
        # Evidence is sufficient — generate directly
        return generate(query, chunks)

    elif verdict == CRAGVerdict.PARTIAL:
        # Evidence is incomplete — try to supplement
        extra_chunks = retrieve_broader(query, top_k=5)
        all_chunks = chunks + extra_chunks
        # Deduplicate
        seen = set()
        unique_chunks = []
        for c in all_chunks:
            if c["text"] not in seen:
                seen.add(c["text"])
                unique_chunks.append(c)
        return generate(query, unique_chunks[:5])

    else:  # IRRELEVANT
        return "I don't have enough information about that in our catalog."

What the Grader Actually Catches

The grader catches a specific failure that retrieval scores alone cannot detect: type mismatch.

A query like "What accessories would pair with this silk saree?" retrieves the saree chunk at 0.86 (high score). The chunk contains fabric, care, and sizing information. It does not contain accessory information. The cosine similarity score is high because "saree" appears in both. The relevance is PARTIAL — the chunk answers what the saree is, but not what accessories pair with it.

Without CRAG: model generates from the saree chunk, adds jewellery recommendations from training memory, hallucinates.

With CRAG: grader returns PARTIAL. System retrieves broader catalog. If accessory chunks exist, they are included. If not, the system responds with what it knows and acknowledges the gap.

What the Numbers Showed

After adding CRAG to the ShopBot pipeline in Book 3:

The cost: one additional LLM call per query (the grader). At gpt-4o-mini prices, this added approximately ₹0.002 per query — less than 15% overhead on the total query cost.

CRAG in a LangGraph Pipeline

In Book 3, CRAG was implemented as a node in a LangGraph StateGraph:

from langgraph.graph import StateGraph, END
from typing import TypedDict

class RAGState(TypedDict):
    query: str
    chunks: list[dict]
    crag_verdict: str
    answer: str

def retrieve_node(state: RAGState) -> RAGState:
    state["chunks"] = retrieve(state["query"])
    return state

def grade_node(state: RAGState) -> RAGState:
    if not state["chunks"]:
        state["crag_verdict"] = "IRRELEVANT"
        return state
    verdict = grade_chunk(state["query"], state["chunks"][0]["text"])
    state["crag_verdict"] = verdict.value
    return state

def route_after_grade(state: RAGState) -> str:
    if state["crag_verdict"] == "RELEVANT":
        return "generate"
    elif state["crag_verdict"] == "PARTIAL":
        return "expand_retrieve"
    else:
        return "refuse"

graph = StateGraph(RAGState)
graph.add_node("retrieve", retrieve_node)
graph.add_node("grade", grade_node)
graph.add_node("generate", generate_node)
graph.add_node("expand_retrieve", expand_node)
graph.add_node("refuse", refuse_node)
graph.add_conditional_edges("grade", route_after_grade)

The Valid Knowledge

• Retrieval score ≠ relevance: a high cosine similarity score means the query and chunk are semantically close — not that the chunk answers the query. CRAG distinguishes these.
• The grader catches type mismatches: when the user asks about X and the retrieved chunk is about the context around X (not X itself), the grader returns PARTIAL and triggers re-retrieval.
• One additional LLM call is the cost: at gpt-4o-mini scale this is negligible per-query. The quality gain on affected queries (12–68% that involve PARTIAL or IRRELEVANT evidence) substantially outweighs the cost.
• IRRELEVANT → refuse, not hallucinate: the most important routing decision. When the grader returns IRRELEVANT, the correct response is to tell the user there is no information — not to generate from insufficient evidence.