Redis Sliding-Window Rate Limiting for Multi-Tenant AI APIs

At 40× normal traffic, the shared queue filled. The national retailer's Black Friday promotion overwhelmed the container. Every tenant's p99 degraded — including the six tenants who had not launched a campaign. One tenant's burst consumed capacity that belonged to others. Redis sliding-window rate limiting solves this.

The Valid Failure

ShopBot's six tenants shared a single FastAPI process. One tenant — Vastralaya, the national retailer — ran a Diwali sale. Traffic: 40× normal volume for 6 hours.

Vastralaya's queries filled the container's worker pool. Other tenants' queries queued behind them. Jaipur Threads' customers waited 2.4 seconds for responses during a period when Jaipur Threads had normal traffic.

Vastralaya paid the same flat subscription as Jaipur Threads. There was no mechanism to prevent one tenant from consuming all shared capacity.

At peak: Vastralaya p99 2,470ms, error rate 6.1%. Jaipur Threads p99 1,890ms, error rate 4.3%. Both unacceptable. Neither was caused by a bug — it was caused by architecture that treated all tenants as a single traffic pool.

What a Sliding-Window Rate Limiter Does

A sliding-window rate limiter tracks the number of requests a tenant has made in the last N seconds. If the count exceeds the limit, the request is rejected with HTTP 429 immediately — before it consumes any retrieval or generation resources.

Unlike a fixed-window limiter (which resets at the top of each minute and allows burst at the boundary), a sliding window is continuous: no tenant can burst at the boundary.

Implementation with Redis

import redis
import time
from fastapi import HTTPException

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

# Per-tenant rate limits (requests per 60-second window)
RATE_LIMITS = {
    "vastralaya":       200,   # national retailer, high tier
    "jaipur_threads":    60,
    "mumbai_silk":       60,
    "chennai_weaves":    60,
    "delhi_couture":     60,
    "pune_fashion":      60,
}

def check_rate_limit(tenant_id: str) -> None:
    """Raises HTTP 429 if tenant has exceeded their rate limit."""
    limit = RATE_LIMITS.get(tenant_id, 30)  # default 30/min for unknown tenants
    window = 60  # seconds
    now = time.time()
    key = f"rate:{tenant_id}"

    pipe = r.pipeline()
    pipe.zremrangebyscore(key, 0, now - window)       # remove old entries
    pipe.zadd(key, {str(now): now})                    # add current request
    pipe.zcard(key)                                    # count in window
    pipe.expire(key, window)                           # TTL cleanup
    results = pipe.execute()

    count = results[2]
    if count > limit:
        raise HTTPException(
            status_code=429,
            detail=f"Rate limit exceeded. Limit: {limit} requests/minute.",
            headers={"Retry-After": "60"},
        )

The pipeline is atomic: all four Redis commands execute as a single transaction. No race conditions between checking the count and adding the new request.

Wiring into FastAPI

from fastapi import FastAPI, Request
from pydantic import BaseModel

app = FastAPI()

class Query(BaseModel):
    question: str
    tenant_id: str

@app.post("/chat")
async def chat(query: Query, request: Request):
    # Rate limit check — before any expensive operation
    check_rate_limit(query.tenant_id)

    chunks = await retrieve_for_tenant(query.question, query.tenant_id)
    answer = await generate(query.question, chunks)
    return {"answer": answer}

The rate limit check runs first — before embedding the query, before touching Qdrant, before calling the LLM. A rejected request costs one Redis pipeline call (~0.3ms). An accepted request that is rejected after LLM generation would cost ~380ms and the full token cost.

Per-Tenant SQS Queues for Burst Absorption

Rate limiting rejects burst traffic. SQS queuing absorbs it — requests are accepted and queued, processed at a controlled rate.

The two mechanisms serve different purposes:

import boto3

sqs = boto3.client("sqs", region_name="ap-south-1")

# One queue per tenant
TENANT_QUEUES = {
    "vastralaya": "https://sqs.ap-south-1.amazonaws.com/123456789012/shopbot-vastralaya",
    "jaipur_threads": "https://sqs.ap-south-1.amazonaws.com/123456789012/shopbot-jaipur_threads",
}

def enqueue_query(query: str, tenant_id: str, session_id: str) -> str:
    """Send query to tenant's SQS queue. Returns message ID."""
    response = sqs.send_message(
        QueueUrl=TENANT_QUEUES[tenant_id],
        MessageBody=json.dumps({
            "query": query,
            "tenant_id": tenant_id,
            "session_id": session_id,
            "enqueued_at": datetime.utcnow().isoformat(),
        }),
        MessageGroupId=tenant_id,          # FIFO queue — order within tenant
    )
    return response["MessageId"]

What Happened After the Fix

With a 200 requests/minute sliding-window limiter on Vastralaya and per-tenant SQS queues:

Vastralaya's excess requests (above 200/minute) were queued in SQS and processed within 4 seconds of the campaign peak. No requests were lost. Jaipur Threads' p99 dropped from 1,890ms to 115ms — isolated from Vastralaya's traffic entirely.

The Valid Knowledge

• Rate limiting must be tenant-scoped, not API-scoped: a global rate limit applied to all tenants does not prevent one tenant from consuming all capacity. Per-tenant limits are the correct unit.
• Sliding window is superior to fixed window: a fixed-window limiter allows 200 requests in the last second of minute 1 and 200 more in the first second of minute 2 — 400 requests in 2 seconds. A sliding window prevents this burst.
• Redis pipeline makes the check atomic: without pipelining, a race condition between ZCARD (count) and ZADD (add) allows more than the limit during concurrent requests.
• Rate limiting and queuing are complementary: the rate limiter protects the real-time path. The SQS queue absorbs burst beyond the real-time limit. Together they prevent capacity exhaustion and data loss.
• Check rate limit before any expensive operation: the rate limit check must be the first thing a request hits — before embedding, retrieval, or LLM calls. A check that runs after the expensive operations defeats its purpose.