How to Add Conversation Memory to a RAG Chatbot

A RAG chatbot without memory is a single-turn question answering system. Every message arrives without context. "What about size L?" has no antecedent. The chatbot says it has no information.

The Valid Failure

Anjali was shopping for a cotton kurta. Turn 1: "Do you have any cotton kurtas under ₹2,000?" ShopBot returned three options. Turn 2: "What about size L?" The second query was sent to the retriever as-is. The retriever found no chunks about "size L" in isolation — it is not a meaningful search query. Zero results. ShopBot said: "I don't have enough information about that in our catalog."

Anjali bought the same cotton kurta from a competitor for ₹200 more, because that chatbot remembered what she was asking about.

The failure was not retrieval. The second query "What about size L?" was missing its antecedent. The fix is not a better retriever — it is memory that provides the antecedent.

The Three-Part Memory Structure

A full conversation transcript is the most obvious approach and the worst performing one. At turn 10, the retriever receives a 2,000-token history as context for a five-word query. The relevant signal is buried.

The correct structure has three parts:

@dataclass
class ConversationMemory:
    active_product: str | None    # The product being discussed right now
    confirmed_filters: dict       # Budget, size, fabric, color — established facts
    rolling_summary: str          # Last 2-3 turns compressed into one sentence

Each turn, the memory is updated. Each query, the memory is prepended to the search query before retrieval.

Implementing the Three-Part Memory

import json
from openai import OpenAI
from dataclasses import dataclass, field, asdict

client = OpenAI()

@dataclass
class ConversationMemory:
    active_product: str | None = None
    confirmed_filters: dict = field(default_factory=dict)
    rolling_summary: str = ""

def update_memory(
    memory: ConversationMemory,
    user_message: str,
    assistant_response: str,
) -> ConversationMemory:
    prompt = f"""Given this conversation exchange, update the memory JSON.

Current memory: {json.dumps(asdict(memory))}

User: {user_message}
Assistant: {assistant_response}

Return ONLY valid JSON with keys: active_product, confirmed_filters, rolling_summary.
confirmed_filters may include: budget, size, fabric, color, occasion.
rolling_summary should be one sentence summarising the conversation so far."""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        response_format={"type": "json_object"},
        messages=[{"role": "user", "content": prompt}],
    )
    updated = json.loads(response.choices[0].message.content)
    return ConversationMemory(**updated)

def build_contextual_query(query: str, memory: ConversationMemory) -> str:
    parts = []
    if memory.active_product:
        parts.append(f"About {memory.active_product}:")
    if memory.confirmed_filters:
        filters = ", ".join(f"{k} {v}" for k, v in memory.confirmed_filters.items())
        parts.append(f"Filters: {filters}.")
    parts.append(query)
    return " ".join(parts)

The Conversation Flow

def chat_turn(
    query: str,
    memory: ConversationMemory,
) -> tuple[str, ConversationMemory]:

    # Expand the query with memory context
    contextual_query = build_contextual_query(query, memory)

    # Retrieve against the expanded query
    chunks = retrieve(contextual_query)
    answer = generate(contextual_query, chunks)

    # Update memory for next turn
    updated_memory = update_memory(memory, query, answer)

    return answer, updated_memory

Walking Through Anjali's Conversation

Turn 1:

• Query: "Do you have any cotton kurtas under ₹2,000?"
• Memory before: {active_product: None, confirmed_filters: {}, rolling_summary: ""}
• Contextual query: same as original (no memory yet)
• Answer: three cotton kurta options listed
• Memory after: {active_product: "cotton kurtas", confirmed_filters: {fabric: "cotton", budget: "₹2,000"}, rolling_summary: "Customer is looking for cotton kurtas under ₹2,000."}

Turn 2:

• Query: "What about size L?"
• Memory before: {active_product: "cotton kurtas", confirmed_filters: {fabric: "cotton", budget: "₹2,000"}, ...}
• Contextual query: "About cotton kurtas: Filters: fabric cotton, budget ₹2,000. What about size L?"
• Retrieval: finds sizing chunks for the cotton kurtas
• Answer: availability of size L in the cotton kurtas under ₹2,000

The second query now retrieves correctly. The antecedent came from memory.

Memory Limits and Clearing

Memory should not accumulate indefinitely. Two rules:

1. Topic change → clear active_product: If the user asks about a completely different product, the old active_product should be replaced, not accumulated.
2. Session boundary → clear all: After a checkout or explicit topic change ("actually, let me look at sarees instead"), reset memory.

CLEAR_TRIGGERS = ["actually", "never mind", "instead", "forget it", "start over"]

def should_clear_memory(query: str) -> bool:
    return any(trigger in query.lower() for trigger in CLEAR_TRIGGERS)

Storing Memory Between Requests

In a stateless API, memory must be passed with each request or stored server-side per session.

# Option 1: Client manages memory (stateless server)
class Query(BaseModel):
    question: str
    memory: dict = {}  # client sends memory back each turn

@app.post("/chat")
def chat(query: Query):
    memory = ConversationMemory(**query.memory) if query.memory else ConversationMemory()
    answer, updated_memory = chat_turn(query.question, memory)
    return {"answer": answer, "memory": asdict(updated_memory)}

# Option 2: Server manages memory per session (Redis)
import redis
r = redis.Redis()

def get_memory(session_id: str) -> ConversationMemory:
    data = r.get(f"memory:{session_id}")
    return ConversationMemory(**json.loads(data)) if data else ConversationMemory()

def save_memory(session_id: str, memory: ConversationMemory, ttl: int = 3600):
    r.setex(f"memory:{session_id}", ttl, json.dumps(asdict(memory)))

Book 2 of the RAG Mastery Series uses Redis for session memory storage, with a 1-hour TTL. After an hour of inactivity, the session is treated as new.

The Valid Knowledge

• The retrieval query must include the antecedent: "What about size L?" retrieves nothing. "About the cotton kurta under ₹2,000 — what about size L?" retrieves the right chunk.
• Rolling summary beats full transcript: the LLM call to compress memory costs ~0.002¢ per turn and prevents the query from ballooning with history that is irrelevant to the current retrieval.
• Memory is structured, not free-text: active_product, confirmed_filters, rolling_summary are typed fields. The LLM fills them in JSON. Unstructured memory is harder to use in query construction.
• Session TTL prevents stale context: a customer who returns two days later should not be picking up where they left off. Memory has a natural expiry.