Subtitle: Learn how caching, hybrid search, reranking, conversation memory, and LLMs work together to power an AI shopping assistant.

Your AI Shopping Assistant Is More Than a Chatbot: Inside Varastra's AI Architecture

1. What Happens Behind the Chat Window?

Imagine a customer asks Varastra, an Indian ethnic-wear store:

“Find me a festive kurta under ₹2,000.”

How does the AI understand the request, search for relevant products, remember previous messages, and generate an answer?

It takes more than connecting an LLM to a chat window. Varastra uses a pipeline of components that work together to retrieve information and produce useful responses.

2. The Complete Architecture

Customer → Chat Widget → FastAPI → Cache → Query Processing → Hybrid Retrieval → Reranking → Context Assembly → LLM → Answer

Redis supports caching and conversation memory. Qdrant supports vector search. RAGAS evaluates the retrieval and answer pipeline.

3. Step 1: Receive and Validate the Request

The customer submits a question through the website chat widget.

The request reaches FastAPI's /ask endpoint, where Pydantic validates the incoming data. The application initializes its processing chain during startup.

Why it matters: Validation catches malformed requests before they enter the main pipeline.

4. Step 2: Check Three Cache Layers

Before running the full RAG pipeline, the system checks for reusable answers.

• L1 — Exact-match cache: Redis checks for the same request.
• L2 — Semantic cache: Checks for a sufficiently similar question using a cosine-similarity threshold of ≥ 0.97.
• L3 — Cache miss: If neither cache provides a suitable answer, processing continues.

Semantic similarity does not prove two questions have identical meaning. Cache reuse must be tested carefully, especially for changing information such as prices and stock.

Why it matters: Caching can reduce latency, repeated processing, and model costs.

5. Step 3: Prepare the Query

An LLM-based query decomposer can break a complex request into smaller search needs.

For “Find me a festive kurta under ₹2,000,” these might include festive kurtas and the customer's budget.

The system then searches for relevant information.

Why it matters: Breaking complex questions into smaller search needs can improve retrieval.

6. Step 4: Find Relevant Information with Hybrid Retrieval

Varastra combines keyword search and meaning-based search.

• BM25: Finds relevant terms in documents.
• Dense retrieval: Uses embeddings—numerical representations of text—to find semantically related information.
• Qdrant: Supports vector storage and search.
• RRF (Reciprocal Rank Fusion): Combines rankings from different retrieval methods.

For example, keyword search may find products labelled “festive kurta,” while dense retrieval may find related descriptions using different wording.

Why it matters: Hybrid retrieval combines the strengths of keyword matching and semantic search.

7. Step 5: Rerank the Results

The initial search returns candidate results. A cross-encoder reranker evaluates how relevant each candidate is to the complete query. In this architecture, the reranker runs as a separate service or container.

Active-SKU Boost

The architecture applies a +0.05 boost to the cross-encoder score when a relevant product SKU is active in the conversation.

Illustrative example:

These scores illustrate the adjustment, not the probability that a product is correct.

Why it matters: Retrieval finds candidates; reranking helps prioritize the most relevant ones.

8. Step 6: Remember the Conversation

Redis session memory stores useful conversational context:

• The last 3 conversation turns.
• A conversation summary.
• Active product SKUs.

If a customer asks, “Which one comes in blue?”, the assistant can use previous messages to understand which product they mean.

Why it matters: Memory helps the assistant handle follow-up questions without requiring customers to repeat everything.

9. Step 7: Generate the Answer

The system assembles relevant context and sends it with the customer's question to GPT-4o-mini, configured with temperature 0.

The LLM uses the supplied information to generate a response.

Temperature 0 makes generation more deterministic, but it does not guarantee factual accuracy.

Why RAG matters: Retrieval provides evidence for the answer instead of relying only on the model's learned knowledge.

Product prices, stock, and policies should be verified against authoritative, current sources.

10. Step 8: Return the Answer

The response travels through the API back to the chat widget.

If reliable information is unavailable, the assistant should ask for clarification or explain the limitation rather than inventing details.

Cached answers also need suitable freshness rules to prevent outdated information from being returned.

11. Step 9: Evaluate the System

The architecture schedules RAGAS evaluation through a Monday cron job, with results recorded in a shared sheet.

RAGAS can help evaluate:

• Faithfulness: Whether claims are supported by retrieved context.
• Answer relevance: Whether the answer addresses the question.
• Context precision: Whether retrieved information is relevant.
• Context recall: Whether necessary information was retrieved.

Evaluation helps identify weaknesses, but it does not replace checks against authoritative product data.

12. What Happens When Something Fails?

Production AI requires failure handling, validation, monitoring, and evaluation—not just a successful demo.

13. Key Takeaways

1. A production AI assistant is more than an LLM connected to a chat window.
2. Caching can reduce repeated work, but answers must remain appropriate and current.
3. Hybrid retrieval combines keyword and semantic search.
4. RRF combines rankings; reranking prioritizes candidates.
5. Conversation memory helps interpret follow-up questions.
6. RAG provides evidence but does not guarantee correctness.
7. Prices and inventory require authoritative data.
8. Evaluation helps teams identify and improve system weaknesses.

The big lesson: The real engineering challenge is coordinating retrieval, memory, generation, validation, and evaluation to build an AI system that works reliably beyond a demo.