Semantic Caching in LLM Apps: How It Works and When to Use It

An LLM call costs money and takes time. When a customer asks "What fabrics does the cotton kurta come in?" and 400 other customers ask functionally the same question that day, paying for 401 LLM calls is wasteful. Semantic caching returns the stored answer when a new query is close enough to a previous one.

The Valid Failure

ShopBot's first caching attempt used exact string matching. Cache key: the full query string. Cache hit rate: 3%. Users phrase the same question differently. "Does the kurta have cotton?" and "Is this kurta made of cotton?" are different cache keys. Same answer. Two LLM calls.

The second attempt: vector similarity cache. Store every query as an embedding. On new query, check cosine similarity against all cached queries. If above 0.95, return the cached answer. Cache hit rate: 31%. LLM calls down by a third on high-traffic days.

Then the catalog was updated. The voile kurta was marked out of stock. The cache held the old availability answer for 24 hours. Customers were told it was available when it was not. The semantic cache was returning stale answers without knowing they were stale.

Caching without invalidation is worse than no cache for time-sensitive data.

The Three-Layer Cache Architecture

Layer 1: Exact match (Redis string)
           ↓ miss
Layer 2: Semantic match (vector similarity ≥ 0.97)
           ↓ miss
Layer 3: Full pipeline (retrieve + generate)
           → store result in both Layer 1 and Layer 2

Each layer has a different purpose:

Implementing the Two Cache Layers

import redis
import json
import hashlib
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, PointStruct
import uuid

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
embed_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
cache_client = QdrantClient(url="http://localhost:6333")

CACHE_COLLECTION = "query_cache"
SEMANTIC_THRESHOLD = 0.97
TTL_SECONDS = 3600

# Ensure cache collection exists
cache_client.recreate_collection(
    collection_name=CACHE_COLLECTION,
    vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)

def cache_key(query: str) -> str:
    return f"exact:{hashlib.sha256(query.encode()).hexdigest()}"

def check_exact_cache(query: str) -> str | None:
    return r.get(cache_key(query))

def check_semantic_cache(query: str) -> str | None:
    vector = embed_model.encode(query).tolist()
    results = cache_client.search(
        collection_name=CACHE_COLLECTION,
        query_vector=vector,
        limit=1,
        score_threshold=SEMANTIC_THRESHOLD,
    )
    if results:
        return results[0].payload.get("answer")
    return None

def store_in_cache(query: str, answer: str):
    # Exact cache
    r.setex(cache_key(query), TTL_SECONDS, answer)
    # Semantic cache
    vector = embed_model.encode(query).tolist()
    cache_client.upsert(
        collection_name=CACHE_COLLECTION,
        points=[PointStruct(
            id=str(uuid.uuid4()),
            vector=vector,
            payload={"query": query, "answer": answer},
        )],
    )

def cached_chat(query: str) -> tuple[str, str]:
    # Layer 1
    answer = check_exact_cache(query)
    if answer:
        return answer, "exact_cache"

    # Layer 2
    answer = check_semantic_cache(query)
    if answer:
        return answer, "semantic_cache"

    # Layer 3
    chunks = retrieve(query)
    answer = generate(query, chunks)
    store_in_cache(query, answer)
    return answer, "pipeline"

Cache Invalidation: The Hard Part

The semantic cache makes the invalidation problem harder. When a product's availability changes, you cannot invalidate by key — there is no single key for "all queries about this product's availability."

Two invalidation strategies:

Strategy 1 — TTL only (simple, imperfect) Set a short TTL. A 1-hour TTL means stale answers persist for at most 1 hour. Acceptable for slow-changing data (fabric, care instructions). Not acceptable for fast-changing data (availability, price).

Strategy 2 — Invalidation on ingestion event (correct) When a new product batch is ingested or a catalog update runs, flush the semantic cache:

def invalidate_cache_on_ingestion():
    # Clear exact cache keys
    for key in r.scan_iter("exact:*"):
        r.delete(key)
    # Clear semantic cache collection
    cache_client.delete_collection(CACHE_COLLECTION)
    cache_client.recreate_collection(
        collection_name=CACHE_COLLECTION,
        vectors_config=VectorParams(size=384, distance=Distance.COSINE),
    )

This ties cache lifetime to data freshness — the cache is valid exactly as long as the underlying data is valid.

When Semantic Caching Pays Off

ShopBot's measurement after adding the three-layer cache: 31% cache hit rate, average latency on cached requests 12ms vs 380ms for full pipeline. At 7,000 queries on campaign Friday, 2,170 were served from cache. LLM API cost: reduced by 31%.

The Valid Knowledge

• Exact string match hit rate is ~3%: users never ask exactly the same question twice. A string-match cache is nearly useless for conversational RAG.
• 0.97 is the semantic threshold, not 0.90: at 0.90, different questions about the same product get the same answer. "Is this kurta available in blue?" and "Is this kurta available in red?" score 0.92 similarity — same answer, wrong color. Threshold must be high enough to prevent cross-query contamination.
• Cache invalidation must be linked to data updates: a semantic cache with only TTL-based invalidation will serve stale availability and price data to customers. Tie invalidation to your ingestion pipeline.
• Cache the answer, not the chunks: chunk retrieval changes as the catalog updates. The answer at the time of the original query is what was semantically verified. Cache the final answer string.