Lesson 09: What If You Could Replace Your Vector Database Without Risking Customer Experience?

Subtitle: Learn how Varastra uses dual-write and shadow reads to migrate from ChromaDB to Qdrant Cloud, compare real search results, and validate the new system before switching customers to it.

1. The Problem: How Do You Replace a Database Safely?

Imagine Varastra uses ChromaDB to power semantic product search. Customers ask questions such as:

“Find me a comfortable cotton kurta for a summer wedding.”

Now, the engineering team wants to migrate to Qdrant Cloud.

Switching databases immediately is risky. Search results might change, relevant products might disappear, or the new system might behave differently under real queries.

The solution: Run both databases side by side, keep ChromaDB as the primary source for customer searches, and compare Qdrant's results in the background.

This architecture uses two techniques:

• Dual-write: Write catalog data to both databases.
• Shadow read: Query the new database in the background without using its results to answer customers.

2. Understand the Architecture

The diagram contains five main components:

JSONL means JSON Lines: a file format where each line contains a separate JSON record.

3. Step 1: Write Catalog Data to Both Databases

The diagram's note says catalog ingestion performs a dual-write to both stores under the same ID.

For example, imagine Varastra has a product:

SKU: VK102

During ingestion, the product's searchable representation is written to:

• ChromaDB, the existing database.
• Qdrant Cloud, the new database.

Both stores use the same ID for the corresponding record.

Why use the same ID?

It helps engineers identify corresponding records and compare search results across the two databases.

If one database returns a product ID that the other does not, the team can investigate whether the difference comes from ingestion, indexing, filtering, or retrieval behavior.

Important: Dual-writing does not automatically guarantee that both databases contain identical data. Write failures, retries, or indexing delays can cause differences. The ingestion process needs appropriate error handling and consistency checks.

4. Step 2: A Customer Asks a Question

A customer asks:

“Show me cotton kurtas for summer.”

The request enters the Varastra read path, the component responsible for handling the search.

The read path sends the query along two paths:

• Primary path: ChromaDB.
• Shadow path: Qdrant Cloud.

These paths have different responsibilities.

5. Step 3: ChromaDB Answers the Customer

ChromaDB is labelled “vector store · local” in the diagram.

It is the primary database during this migration stage.

ChromaDB searches for relevant products and returns its results to the Varastra read path. The response is then sent to the customer.

For example, ChromaDB might return three relevant cotton kurtas.

The customer receives results from ChromaDB—not Qdrant.

Why keep ChromaDB primary?

Because it is the existing search system, the team can continue serving customers while evaluating the replacement.

The key principle: The migration should not change the customer-facing search behavior merely because the new database is being tested.

6. Step 4: Qdrant Cloud Searches in Shadow Mode

At the same time, the read path sends a shadow query to Qdrant Cloud.

The diagram labels this connection “shadow” and identifies Qdrant's region as asia-south1.

Qdrant performs its search, but its results are not used to generate the customer's response.

Imagine the two databases return these results:

Both databases return the same three products, but their rankings differ.

This is exactly the kind of difference engineers need to discover before migration.

Why shadow reads matter: They let the team observe how the new database behaves with real queries without making its results customer-facing.

The diagram does not specify the exact timeout, retry, or error-handling configuration. In a production implementation, shadow failures should not disrupt the primary response.

7. Step 5: Compare Both Results in the JSONL Log

The diagram shows dashed arrows from both vector stores to the Comparison log (JSONL).

The log records four fields:

• query — the customer query being evaluated.
• chroma_ids — product or document IDs returned by ChromaDB.
• qdrant_ids — IDs returned by Qdrant Cloud.
• agreement_at_k and scores — information used to compare the results.

What does agreement at K mean?

Agreement at K measures how much the top results overlap between two systems.

For example, suppose both systems return five results:

• ChromaDB: A, B, C, D, E
• Qdrant: A, B, C, F, G

Three of the five results overlap.

Agreement at 5 = 3/5 = 0.60, or 60%, if agreement is calculated as the fraction of shared IDs in the two top-five lists.

The diagram does not specify the exact formula used by its implementation, so this is an illustrative calculation rather than a claim about the code.

Why log scores?

Two databases might return the same products but assign different similarity scores or rank them differently.

Recording scores helps engineers investigate those differences.

However, scores from different retrieval implementations may not be directly comparable. Their meaning depends on the distance metric, configuration, and scoring method.

The goal is to collect evidence, not assume that matching scores mean matching quality.

8. Step 6: Decide Whether the New Database Is Ready

After collecting enough comparisons, engineers can evaluate whether Qdrant is ready to replace ChromaDB.

They should examine:

1. Result agreement: Do both systems retrieve the same relevant products?
2. Ranking quality: Are the most useful products near the top?
3. Missing records: Are products absent from one database?
4. Score behavior: Are there unexpected differences in retrieval scores?
5. Operational reliability: Does the new search path behave reliably under realistic workloads?

High agreement is useful, but it does not prove that the results are relevant. Both databases could return the same poor results.

A migration decision should combine comparison logs with relevance evaluation, data consistency checks, and operational testing.

Once the evidence supports the migration, the team can plan a controlled switch to Qdrant. The diagram illustrates the validation stage; it does not show the final cutover.

9. What Can Go Wrong?

The primary search path should remain available even when shadow queries fail.

10. Key Takeaways

• Dual-write keeps both databases supplied with catalog data.
• Using the same IDs helps engineers compare corresponding records.
• ChromaDB remains the primary search source during the illustrated migration stage.
• Qdrant Cloud runs in shadow mode, so its results do not reach customers.
• JSONL logs capture query results, IDs, agreement information, and scores.
• Engineers use comparisons and relevance tests to evaluate the replacement.
• Shadow testing reduces migration risk, but it does not eliminate the need for validation and controlled rollout.

Remember: You don't have to trust a new database blindly. Let it work alongside the existing system, compare its behavior against real queries, and use the evidence to decide when it is ready.