Multi-Query Retrieval and Query Decomposition in RAG

A customer asked: Show me something festive and warm.

Book 1 Chapter 10 had named this failure and deferred it: "The correct solution is multi-query retrieval: split the query into two sub-queries, retrieve separately, merge and re-rank the results. This is Book 2's work."

Book 2 had run four chapters without closing it. The festive-and-warm query had been sitting in Arjun's notebook since November, two pages before the December dashboard reading, in his own hand.

Multi-query. Promised.

The Valid Failure: A Combined Vector Satisfies Neither Intent

The silk saree scored 0.79 for festivity on dense retrieval. The woolen shawl scored 0.74 for warmth on dense retrieval. The combined query embedding for festive and warm fell, in vector space, between the two intent regions — close to neither.

Three approaches failed before the right one.

Passing the whole query to hybrid retrieval. The dense embedding for festive and warm fell into the average of the two intent spaces and matched nothing well. BM25 matched the literal tokens festive and warm in product descriptions but did not understand that both had to be satisfied simultaneously. Reciprocal Rank Fusion promoted whichever chunk scored best on average — usually the strongest match for one intent and a weak match for the other.

Embedding interpolation. Embed festive, embed warm, average the two vectors, retrieve against the result. The retrieved chunks were not the saree and not the shawl — they were products whose embeddings happened to land near the midpoint of the two intent embeddings. Midpoints in vector space are rarely products that satisfy both intents. They tend to be products that satisfy neither.

Keyword lookup. Build a table of intent keywords — festive, warm, casual, formal, under three thousand — and split queries by detecting them. Worked for ten test queries. The eleventh: something nice for the rains. The twelfth: I want to look traditional but feel comfortable. Neither contained a listed keyword. The lookup assumed customers used a known vocabulary. They did not.

The Valid Source: The Model as Decomposer

The correct approach used the model itself. A small constrained inference call — before retrieval ran at all — asked the model whether the customer's query contained one intent or two, and if two, to produce two paraphrased sub-queries in plain English.

DECOMPOSE_PROMPT = """\
The user has asked a question about a fashion catalog. If the question contains
ONE intent, return a JSON list with one paraphrase: ["<paraphrase>"]. If it
contains TWO independent intents that need separate retrievals to answer, return
a JSON list with two paraphrases: ["<intent A>", "<intent B>"]. Never return
more than two. Keep paraphrases short and self-contained.

Question: {query}

JSON:
"""

def decompose(query: str) -> list[str]:
    raw = llm_small.complete(
        prompt=DECOMPOSE_PROMPT.format(query=query),
        max_tokens=120,
        temperature=0,
    )
    try:
        sub_queries = json.loads(raw.strip())
        if not isinstance(sub_queries, list) or not 1 <= len(sub_queries) <= 2:
            return [query]
        return [str(sq) for sq in sub_queries]
    except json.JSONDecodeError:
        return [query]  # fail safe — treat as single intent

Each sub-query then ran through the full hybrid + rerank pipeline as a standalone query. Results merged by RRF. Merged top-three went to the model with both intents acknowledged in the prompt: "The customer's question has two parts; address both."

The cost: roughly ₹0.0008 per query for the decomposer call. Single-intent queries received a one-element list and went through the pipeline unchanged. Two-intent queries paid the detection cost plus a second retrieval pass. Multi-intent queries were about 9% of labeled traffic.

The Valid Knowledge: Two Intents, Two Retrievals, One Honest Answer

Festive and warm. The decomposer returned: ["festive clothing for a wedding event", "warm clothing for cool weather"]. Each ran through hybrid + rerank. Festive sub-query: silk saree rank 1, Anarkali rank 2. Warm sub-query: woolen shawl rank 1. RRF merge: saree 1st, shawl 2nd, Anarkali 3rd. ShopBot answered:

The Silk Saree with zari border (₹4,299) is a festive option, but on its own it is not specifically warm. The Woolen Shawl (₹2,199) in charcoal grey or maroon is warm and pairs naturally with a saree for evening events in cooler weather. For a festive function in winter, layering the saree with the shawl is the catalog's nearest match to your question.

I want to look traditional but feel comfortable. The decomposer returned: ["traditional clothing", "comfortable clothing"]. Anarkali for traditional, cotton kurta for comfortable. ShopBot answered both.

RAGAS evaluation on the 18 multi-intent queries in the labeled dataset:

• Context recall (multi-intent slice): 0.41 → 0.83
• Overall labeled context recall: 0.79 → 0.84
• Cost per query: ₹0.0156 → ₹0.0162 (4% increase, applying to 9% of traffic)

The architectural move is to push the question of what shape is this query? earlier in the pipeline than any retrieval — to answer it before retrieval runs, with a small inference call whose output changes what gets retrieved. The customer never sees the split. The answer arrives whole, addressing both halves of what they asked.

This article covers concepts from Book 2, Chapter 5 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.