Shadow Deployment and the Comparison Log in RAG Migration

ShopBot ran on ChromaDB. ChromaDB was a correct Book 1 choice — a single Python import, an embedded database inside the FastAPI process, no credentials required. It had carried five thousand catalog chunks and seventy queries a day without complaint.

For the next chapter, it was finished. Hybrid search needed BM25 alongside dense vectors. ChromaDB did not support that combination natively. The new architecture needed Qdrant.

The question was how to move the database under a working product without the product noticing.

The Valid Failure: Three Ways to Break It

Arjun identified three ways to do the migration badly before settling on the correct one.

Hot-swap. Stand up Qdrant, point ShopBot at it, deploy. If any retrieval regression lands on real customers, there is no way to prove the regression came from the migration rather than anything else — because there is no comparison. The damage is real-time and invisible until a customer complains.

Clean rebuild. Re-embed the catalog from scratch into Qdrant, run test queries, declare it good. This sounded responsible until he thought about it for ten minutes. ChromaDB and Qdrant do not produce identical embeddings even with the same model and the same text — they use different normalisations and different distance metric defaults. A clean rebuild is a new index, not a verified migration. The four test queries from Book 1 would not catch the subtle differences.

Parallel without traffic mirroring. Both stores live, ShopBot reads from ChromaDB, Qdrant fills in the background, occasional manual checks. This was the least dangerous of the three and the least informative. Without real traffic mirroring, the only queries hitting Qdrant are the ones Arjun chose to send — the ones he expected to work. The 22% that silently failed never made it into the test set.

None of the three could answer the question that mattered: does the new store return the same answers as the old store on the queries that real customers actually send?

The Valid Source: Dual-Write and Shadow Reads

The approach from Book 2 Chapter 2 had four components.

Dual-write. Every chunk indexed into ChromaDB was simultaneously indexed into Qdrant with the same id. The catalog had not changed, so the dual-write ran once against the existing five thousand chunks and produced a Qdrant collection that was, on the dense side, a faithful mirror.

Shadow reads. ShopBot's read path continued to hit ChromaDB and return the result to the customer. Asynchronously, after the response had already gone out, the same query was sent to Qdrant. Both result sets were written to a comparison log along with their chunk ids and scores. Customer-facing path unchanged. Customer latency unchanged.

async def retrieve_with_shadow(query: str, k: int = 3) -> list[Chunk]:
    query_vec = embed(query)
    chroma_results = chroma_collection.query(
        query_embeddings=[query_vec], n_results=k
    )
    asyncio.create_task(_shadow_compare(query, query_vec, chroma_results, k))
    return chroma_results

Comparison log. After one day of shadow reads, 931 queries had been mirrored. The log measured agreement_at_k — the proportion of top-three chunks that both stores returned for the same query.

• 86% of queries had identical top-three sets.
• 11% had two of three matching.
• 2% had only one chunk in common.
• 1% (eight queries) had completely disjoint top-three sets.

He read all eight. Six were edge-case queries where near-identical similarity scores broke differently because of distance-metric defaults — fixed by switching Qdrant to cosine. One was a query where Qdrant was returning the better answer (a ChromaDB stale chunk). Two were data-faithfulness errors in the dual-write itself — a typo in the sku field — found and fixed by the comparison log before they could harm a customer.

After the fixes, second-day logs showed 94% identical top-three. Every remaining disagreement was either ranking jitter or a Qdrant improvement. He ran the Book 1 RAGAS evaluation set — 50 questions — against Qdrant standalone: faithfulness 0.71, context precision 0.82. Identical to the locked Book 1 baseline.

No regression. No celebration.

The Valid Knowledge: What Makes a Migration Safe

A migration that produces different answers — even better answers — without a verified comparison breaks the warranty on every prior claim the system has made. The Grounding Layer is a contract. The contract has to survive infrastructure changes, or it is not a contract.

The comparison log is what made the migration provable, not merely plausible. It surfaced two ingestion bugs that would have been invisible under hot-swap or a clean rebuild. It proved equivalence against the same evaluation set the system had been measured against from the start.

The discipline has four requirements:

1. Dual-write strategy — every write goes to both stores under the same id.
2. Shadow read strategy — mirror real traffic to the new store asynchronously, without affecting the customer path.
3. Comparison metric — agreement_at_k or equivalent; decide in advance what threshold declares equivalence.
4. Rollback trigger — if the comparison metric breaches the threshold, return to the old store in one feature-flag flip.

If any of the four is hand-wavy, the migration is not yet safe to attempt.

The floor moved. The customers did not notice. The 22% was still 22% — because the migration was not a fix, it was a foundation. The improvements began the next day.

This article covers concepts from Book 2, Chapter 2 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.