Lesson 14: The AI Answered in 400 ms. Could It Have Answered in 10? Inside Varastra’s Three-Layer Cache

Subtitle: Learn how Varastra uses exact-match caching, semantic caching, and explicit cache invalidation to make its AI shopping assistant faster, reduce repeated processing, and prevent outdated product answers from sticking around.

1. The Problem: Why Rebuild an Answer You Already Know?

Imagine customers ask Varastra, an Indian ethnic-wear store, these questions:

• “What is the price of this cotton kurta?”
• “What is the price of this cotton kurta?” — asked again.
• “How much does this kurta cost?”

If every request runs through retrieval, reranking, session memory, and an LLM, the system repeats expensive work—even when a suitable answer is already available.

Caching means temporarily storing a result so the system can reuse it instead of calculating it again.

But caching creates a risk: what if the product price changes after the answer is stored?

Varastra's design addresses both problems: reuse answers when appropriate and invalidate stale entries when catalog data changes.

2. Understand the Architecture

The diagram contains two workflows.

Answer workflow:

Customer query → L1 exact-match cache → L2 semantic cache → Full pipeline → Fresh answer and cache filling.

• An L1 hit returns a cached answer in approximately 0 ms.
• An L2 hit returns a semantically matching cached answer in approximately 10 ms.
• If both layers miss, the full pipeline runs in approximately 400 ms.

These are the timings shown in the diagram, not guaranteed production latency.

Cache invalidation workflow:

Catalog update → Invalidation worker → Remove stale entries from L1 and L2.

The second workflow is essential because a fast answer is not useful if it contains outdated information.

3. Step 1: L1 — Exact-Match Cache

The first layer is labelled:

L1 · exact-match cache

The diagram specifies:

SHA-256(query) → Redis · TTL 1h

Here's what that means:

• SHA-256: A function that converts the query into a fixed-length hash.
• Redis: A fast data store used here to hold cached answers.
• TTL: Time to live—the period after which a cached entry expires automatically.
• 1h: The diagram specifies a one-hour TTL.

Example

A customer asks:

“What is the price of the mint-green cotton kurta?”

The system creates a hash from the query and checks Redis for a matching cache entry.

If the exact query has a cached answer, the system returns it immediately.

Why use L1? Exact-match lookups are fast and avoid repeating the full AI workflow for identical queries.

Important: An exact query match does not guarantee that the stored answer is still factually current. That's why invalidation matters.

4. Step 2: L2 — Semantic Cache

If L1 misses, the request moves to:

L2 · semantic cache

The diagram specifies:

cosine ≥ 0.97 · Qdrant

Unlike an exact-match cache, a semantic cache can identify queries that are worded differently but have similar meanings.

For example:

• “What is the price of this cotton kurta?”
• “How much does this cotton kurta cost?”

The wording differs, but the underlying intent is similar.

What does cosine ≥ 0.97 mean?

Cosine similarity compares the direction of two vectors. In semantic caching, vectors represent queries or their meanings.

The diagram sets a similarity threshold of 0.97. A query must meet or exceed this threshold to qualify for a semantic-cache match under the depicted rule.

This is a strict similarity threshold, but it does not mean there is a 97% chance that the cached answer is correct.

Two similar questions can still require different answers—for example, if they refer to different product variants.

Why use L2? It can reuse suitable answers even when customers phrase similar questions differently.

5. Step 3: If Both Caches Miss, Run the Full Pipeline

When neither cache provides a suitable result, the request reaches the full pipeline.

The diagram lists:

Retrieve · rerank · memory · LLM · ~400 ms

This means the system performs the more expensive work needed to answer the question.

In Varastra's architecture, that can involve:

1. Retrieving relevant product information.
2. Reranking candidate results.
3. Incorporating relevant session memory.
4. Supplying the selected context to the LLM.
5. Generating a fresh answer.

The exact implementation details of every pipeline stage are outside this particular diagram.

Once the fresh answer is produced, the diagram shows it being used to fill the caches.

That way, future suitable requests may avoid repeating the full process.

6. Step 4: Why Explicit Cache Invalidation Matters

Now imagine Varastra changes the price of a kurta from ₹1,799 to ₹1,999.

A cached answer may still say ₹1,799.

Waiting for the cache's one-hour TTL to expire could leave the old price available for too long.

Varastra therefore uses a separate invalidation workflow.

The diagram shows:

Catalog update → Invalidation worker → L1 + L2 caches

Let's follow it.

A. The catalog changes

A product is updated, and the system publishes an event:

catalog.updated

The diagram identifies this as a pub-sub event—a notification that other components can react to.

B. The invalidation worker receives the event

The worker processes the catalog update and determines which cached answers may be affected.

C. Remove affected cache entries

The diagram specifies:

delete by retrieved_skus filter

This indicates that the worker uses retrieved SKU information to identify cache entries associated with updated products.

For example, if a cached answer was generated using SKU K-1299, an update to that product can trigger removal of affected entries.

The diagram shows invalidation targeting both L1 and L2.

Why remove entries instead of relying only on TTL? Because a catalog change can make a cached answer stale before its normal expiry time.

7. How the Two Workflows Work Together

Consider this sequence:

1. A customer asks about a kurta's price.
2. Both cache layers miss, so the full pipeline generates an answer.
3. The answer is stored in the caches.
4. Another suitable request can reuse that answer.
5. Varastra updates the kurta's price.
6. The catalog update triggers the invalidation worker.
7. Affected cached entries are removed.
8. A later request can run through the pipeline again and obtain updated information.

This is the central design principle: reuse answers when safe, but invalidate them when their supporting data changes.

The diagram's caption says stale answers should have a window of seconds, not days. That is the intended goal, not a guaranteed timing: actual freshness depends on event delivery, worker processing, and correct cache-key and SKU tracking.

8. What Can Go Wrong?

For critical facts such as prices, stock, sizes, and delivery promises, the system should use appropriate validation or current authoritative data. Cache invalidation reduces stale-answer risk, but it does not guarantee correctness on its own.

9. Key Takeaways

1. L1 uses an exact query hash in Redis and has a one-hour TTL in the diagram.
2. L2 uses semantic similarity with a cosine threshold of at least 0.97 in Qdrant.
3. If both layers miss, the full retrieval, reranking, memory, and LLM pipeline runs.
4. A fresh answer can fill the caches for future requests.
5. Catalog updates publish an event named catalog.updated.
6. An invalidation worker uses a retrieved_skus filter to remove affected entries from both cache layers.
7. Semantic similarity is not a guarantee that an answer is correct.
8. Fast caching and reliable invalidation must work together.

The big lesson: A production AI assistant needs more than a fast way to remember answers. It also needs a reliable way to stop reusing them when the facts change.