Conversation Memory and Context Condensation in RAG

The four-turn exchange appeared in the production log, flagged as a support escalation:

Customer: Cotton kurta in mint. ShopBot: The cotton kurta is available in white, sky blue, and mint green, in sizes S to XXL, priced at ₹1,299. It is machine washable. Customer: What about size L? ShopBot: I don't have information about size L. For specific sizing details, please contact our team at support@zudyog.com. Customer: Do you have it in stock? ShopBot: I'm not sure which product you mean. Could you clarify? Customer: Forget it.

Three hours later, Anjali sent an email to support@zudyog.com: I tried your AI assistant but it couldn't answer simple questions about a kurta. I bought the same product from another store. The screenshot she attached showed a cotton kurta in mint green at ₹1,499 — two hundred rupees more than zUdyog's price. She paid the premium because ShopBot had stopped being useful at turn two.

The Valid Failure: Every Message Is the First Message

ShopBot had answered turn one correctly. Turn two — what about size L — was, on its own, a question with no antecedent. The retriever was asked to find catalog content for what about size L. It returned generic sizing chunks and one note about size charts. The model refused, because it had no specific product to speak from.

Turn three — do you have it in stock — was almost meaningless without the conversation. There was no it. There was no notion of prior context.

ShopBot was not a conversational system. It was a search engine that pretended to talk. Every customer message was the first message.

Stuffing the full transcript. He modified the prompt to include the last 20 messages before each query. Turn two now succeeded — the model understood it meant the cotton kurta. But latency went up; the prompt was three to five times longer. By turn ten of a long conversation, the model occasionally answered a current question using context from five turns ago. Cost rose because token-priced LLMs charge for every token, useful or not.

Last-N window. He capped history at the last four turns. Latency stabilised. And turn five of any conversation that referenced something from turn one — did you say the kurta was in mint? — failed again, because turn one was no longer in the prompt. The information had been there. It had been thrown away.

Memory was not a window scrolling along a transcript. Memory was a structured thing.

The Valid Source: Three Pieces Doing Three Different Jobs

The solution from Book 2 Chapter 6 had three components, each with a distinct role:

Last 3 turns verbatim. Always in the prompt. Covers the most common reference patterns: what about it in another colour, and in size L, what does that cost. Short enough to stay cheap.

Rolling 2-sentence summary. After every fourth turn, a small inference call summarised the conversation — which products were being discussed, any constraints the customer had mentioned. Stored in Redis keyed by session id, injected before the immediate context. Carried what was older than three turns but still relevant. Typically 40–80 tokens.

SUMMARY_PROMPT = (
    "Summarise the conversation so far in two sentences. "
    "Focus on the products being discussed and any constraints "
    "the customer has mentioned. Do not invent facts."
)

def maybe_refresh_summary(session: Session, full_history: list[Turn]) -> None:
    if len(full_history) % 4 != 0:
        return
    transcript = "\n".join(f"{t.role}: {t.content}" for t in full_history)
    session.summary = llm.complete(
        system=SUMMARY_PROMPT, user=transcript, max_tokens=120
    )

Active-SKU list. ShopBot tracked which product SKUs had been mentioned in the conversation (max 3, most recent first). After cross-encoder reranking, any candidate whose SKU was in the active list received a small bonus (ACTIVE_SKU_BOOST = 0.05) added on top of the rerank score — enough to break ties in the active product's favour, not enough to override a clearly stronger candidate.

boosted = []
for chunk, score in zip(candidates, ce_scores):
    bonus = ACTIVE_SKU_BOOST if chunk.metadata.get("sku") in session.active_skus else 0.0
    boosted.append((chunk, float(score) + bonus))

boosted.sort(key=lambda item: item[1], reverse=True)

The bonus applied after the cross-encoder rescored everything — applying it before would have had no effect, because the cross-encoder would have overwritten the order.

The Valid Knowledge: What the Numbers Show

Turn-one faithfulness on the previous architecture: 0.83 (single-turn evaluation from Chapter 4, unchanged).

Turn-two faithfulness on the previous architecture (no memory): 0.51.

Turn-two faithfulness with memory: 0.79.

Turn-three faithfulness with memory: 0.74.

The drop-off across turns was no longer a cliff. It was a gentle slope.

The four-turn exchange tested again:

Customer: Cotton kurta in mint. → ✓ Correct answer. Customer: What about size L? → ✓ Correct: size L available in all three colours including mint. Customer: Do you have it in stock? → Correct refusal: "I don't have real-time stock information for the cotton kurta in size L mint green. For current availability, please contact support."

The third turn still routed to support — because it had to; the catalog chunks did not contain real-time inventory. But it failed for the right reason, not for the absence of memory.

Meera, the beta tester from Book 1, tested the new build with a five-turn conversation about a saree. After the fifth turn, she typed: This one talks like a person.

Memory is not a window. It is a small, structured set of facts the system maintains on the customer's behalf, refreshed turn by turn, used by both the retriever and the model.

This article covers concepts from Book 2, Chapter 6 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.