Bottleneck Identification and Cost Per Query at Scale

Week 6, per-route RAGAS showed Route D at F 0.79. Week 8, 16 lines across two files brought it to F 0.86. Week 9, the RAGAS Monday blended number read 0.89.

The architecture was in the best shape it had been in across all three books. Every route had a name, a target, and a score. The per-query cost was ₹0.0148 — 26% below the Book 2 architecture running at peak.

Priya — zUdyog Fashion's marketing lead — sent a calendar invite: "Online Campaign Launch / Week 10 / Monday–Friday / Target: 35K queries/day".

The performance and cost claims were measured at a few hundred queries a minute. The campaign would require a thousand.

The Valid Failure: What 600 Queries Per Minute Revealed

A load test at 600 qpm — 60% of the campaign target — exposed three independent bottlenecks:

Bottleneck one: the cross-encoder. The reranking service ran one thread per request by default. At 600 qpm, CPU utilisation hit 65%. The p95 rerank latency was 78ms — acceptable individually, but occupying most of the total 280ms budget for every query. Any traffic spike above 700 qpm would push reranking over its SLA before the LLM call even started.

Bottleneck two: Qdrant. The vector database handled hybrid retrieval — dense + sparse + RRF — for each query. At 600 qpm, Qdrant's CPU hit 70%. The p95 retrieval latency was 14ms, but the headroom at 1,000 qpm was not there. Qdrant's current tier had a fixed CPU ceiling.

Bottleneck three: LLM rate limit. The generation step called GPT-4o-mini for the main response and — when CRAG found a PARTIAL verdict — an additional LLM call for the corrective sub-query. At 600 qpm, the pipeline was placing 1,300 LLM calls per minute: above the tier's rate limit of 1,200 calls/minute. Requests were being throttled. The p95 generation latency had a long right tail from retries.

None of these would have been visible from the labelled dataset. None would have been visible from per-route RAGAS. They were operational constraints, not algorithmic ones, and they appeared only under sustained production-level load.

The Valid Source: Three Fixes, Different Shapes

Fix one: cross-encoder replicas. The reranking service was stateless — it embedded and scored, it did not write or synchronise. Three replicas behind a round-robin load balancer: 40-second deploy. Per-query rerank latency dropped from p95 78ms to p95 26ms. CPU per replica at 600 qpm: 22%. Cost: ₹1,200/month.

Fix two: Qdrant tier upgrade. Qdrant's managed service offered a higher tier with more CPU headroom. zUdyog's Qdrant cost went from ₹6,800/month (Book 2's doubled instance) to ₹11,400/month. The retrieval p95 at 1,000 qpm dropped from a projected 40ms to 12ms. No code change — only infrastructure.

Fix three: second API key, round-robin rotation. OpenAI rate limits are per API key, not per application. A second API key and a Python itertools.cycle — alternating keys on each LLM call — doubled the effective throughput ceiling from 1,200 to 2,400 calls/minute.

from itertools import cycle

api_keys = cycle([os.getenv("OPENAI_KEY_1"), os.getenv("OPENAI_KEY_2")])

def get_client():
    return openai.OpenAI(api_key=next(api_keys))

No retry spike. The long right tail from throttling disappeared from the latency distribution.

After all three fixes, at 600 qpm: p50 285ms, p95 380ms. At 1,000 qpm: p50 340ms, p95 580ms — within SLA.

The Valid Knowledge: The Campaign and Its Regressions

Day-by-day load:

Friday peaked at 38,000 queries — 3,000 above the target. p50 held at 280ms. p95 held at 410ms.

Route C regression caught mid-campaign (Wednesday, week 10):

Per-route RAGAS ran daily during the campaign, not just Mondays. Wednesday morning's check showed Route C — HyDE queries with a RELEVANT CRAG verdict — drop from F 0.92 to F 0.88, CP from 0.93 to 0.88. Nine of nineteen Route C queries in that day's sample were underperforming.

The root cause was pattern-specific: campaign customers used more generic language than the baseline sample. Generic queries produced generic HyDE hypotheticals. Generic hypotheticals retrieved generic chunks. CRAG still ruled RELEVANT (the chunks were about fashion), but the chunks were too broad to be precise.

Fix: one line added to the HyDE generation prompt — "name a specific product category from the catalog, such as cotton kurta, silk saree, or linen co-ord set." — anchoring the hypothetical to something concrete. Route C the following week: F 0.93, CP 0.94 — above the pre-regression baseline.

The regression was caught in 90 minutes because the per-route view had a Route C row, had a week-over-week history, and showed an anomaly rather than burying it in the blended number.

Cost comparison, Friday peak (38,000 queries):

The Book 2 architecture would have cost ₹975 on the same Friday. The Book 3 architecture cost ₹402. The embedding API alone would have cost ₹570 more. INT8 in-house embeddings — ₹240 one-time training run, ₹0 ongoing — was the single largest contributor to the cost reduction.

Blended F trajectory:

The Friday peak was the largest load zUdyog Fashion's system had ever handled. The pipeline held.

This article covers concepts from Book 3, Chapter 8 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.