Per-Tenant Embedding Models: LRU ModelCache, MODEL_REGISTRY, and Thread Safety

Krishna asked the question on a Wednesday morning, without looking up from his laptop.

"Which model is serving Vastralaya right now?"

Arjun opened a terminal, checked the container logs, found the environment variable that was supposed to hold the active model path. It said models/bge-small-zudyog-v2.

bge-small-zudyog-v2 was the model fine-tuned on zUdyog Fashion's catalog. It had been serving every tenant since Book 3 Chapter 6.

The Valid Failure: A Domain-Specific Model Serving the Wrong Domain

The bge-small fine-tune from Book 3 was not a general model. It was a zUdyog model — trained on zUdyog's five hundred products, paraphrased against zUdyog's vocabulary (kurtas, co-ords, Anarkalis, dupatta weights), evaluated against zUdyog's query distribution. Its 384-dimensional embedding space had been shaped by what zUdyog's customers asked about.

A Jaipur customer typed: do you have a mirror-work lehenga for a daytime function in Rajasthan?

Mirror-work embroidery — shisha, in Rajasthani — involves small pieces of reflective glass stitched into fabric with chain and buttonhole stitches. The craft vocabulary (shisha work, gota patti, tie-dye bandhani, block-printed mul cotton) appears rarely in general fashion training data and almost never in a catalog of five hundred kurtas and co-ord sets.

With the zUdyog model serving the Jaipur query, the embedding landed near silk sarees and festive Anarkalis — the closest thing in zUdyog's space to embellishment and occasion wear. The retriever returned chunks about the silk saree with zari border and the teal Anarkali suit. The answer described products not in the Jaipur boutique's catalog.

RAGAS for the Jaipur boutique with the wrong model:

The faithfulness drop was expected: the model was retrieving chunks about the wrong products, and the LLM, constrained by the grounding prompt, was faithfully describing the wrong things. The fix was not a better prompt. The fix was a model that knew Jaipur.

First approach: six containers running six models. Six tenants meant six fine-tuned models, each in its own container. Real isolation. The problem: six containers running six INT8 bge-small models consumed six times the memory of one, most of it idle most of the time. The Surat saree house sent perhaps three hundred queries on a busy day. The Jaipur boutique sent a hundred and fifty. A container provisioned for peak load at Surat's volume was sized for three hundred queries while sitting empty for most of its runtime. The utilisation justifying the cost did not exist.

Second approach: adapter layers. bge-small supports adapter layers in its base form — lightweight parameter sets that modify model behaviour for a specific task without retraining full weights. Swapping adapters at inference time would allow a single base model to serve multiple tenants.

The problem: the Book 3 fine-tuning pipeline had fine-tuned the full model weights and quantised to INT8 using ONNX. A full-weight INT8 ONNX model has no adapter architecture to hook into. Retrofitting would require retraining all six models from scratch with a different training objective — three weeks the schedule did not have, and RAGAS baselines that could not be directly compared.

The Valid Source: MODEL_REGISTRY + LRU ModelCache

The correct approach was simpler: the models did not all need to be in memory at the same time. They needed to be available — loaded when a query arrived for the relevant tenant and unloaded under memory pressure. A loader, a cache, an eviction policy.

MODEL_REGISTRY — the catalogue of what existed and where:

MODEL_REGISTRY: dict[str, str] = {
    "zudyog_fashion":    "s3://shopbot-models/bge-small-zudyog-v2.onnx",
    "surat_saree_house": "s3://shopbot-models/bge-small-surat-v1.onnx",
    "jaipur_boutique":   "s3://shopbot-models/bge-small-jaipur-v1.onnx",
    "national_retailer": "s3://shopbot-models/bge-small-national-v1.onnx",
    "vastralaya":        "s3://shopbot-models/bge-small-vastralaya-v1.onnx",
    "kolkata_silks":     "s3://shopbot-models/bge-small-kolkata-v1.onnx",
}

LOCAL_CACHE_DIR = os.environ.get("MODEL_CACHE_DIR", "/tmp/shopbot_models")

S3 loader — pulls the model on first request, returns local path on subsequent requests:

def ensure_local(s3_uri: str) -> str:
    bucket, key = s3_uri.replace("s3://", "").split("/", 1)
    local_path = os.path.join(LOCAL_CACHE_DIR, key.replace("/", "_"))
    if not os.path.exists(local_path):
        os.makedirs(LOCAL_CACHE_DIR, exist_ok=True)
        s3.download_file(bucket, key, local_path)
    return local_path

LRU ModelCache with threading.Lock on the miss path — the OrderedDict maintains insertion order; move_to_end() marks the most recently used entry; eviction removes the first (least recently used) key when at capacity. The threading.Lock on the miss path prevents two concurrent requests for the same cold model from triggering two simultaneous S3 downloads:

class ModelCache:
    def __init__(self, max_size: int = 3):
        self.max_size  = max_size
        self._cache: OrderedDict[str, InferenceSession] = OrderedDict()
        self._lock     = threading.Lock()

    def get(self, tenant_id: str) -> InferenceSession:
        with self._lock:
            if tenant_id in self._cache:
                self._cache.move_to_end(tenant_id)
                return self._cache[tenant_id]

            # Miss path — load from registry
            s3_uri     = MODEL_REGISTRY[tenant_id]
            local_path = ensure_local(s3_uri)
            session    = InferenceSession(local_path)

            if len(self._cache) >= self.max_size:
                evicted = next(iter(self._cache))
                del self._cache[evicted]

            self._cache[tenant_id] = session
            self._cache.move_to_end(tenant_id)
            return session

model_cache = ModelCache(max_size=3)

max_size=3 held the three most active tenants' models warm. In practice the national retailer's model was almost never evicted — its query volume was high enough that it was always the most recently used. The boutique tenants with lower traffic shared the remaining two cache slots, loaded on demand and evicted when a busier tenant's query arrived.

embed() — a single call that did not need to know which model it was using:

def embed(text: str, tenant_id: str) -> np.ndarray:
    session = model_cache.get(tenant_id)
    inputs  = tokenizer(text, return_tensors="np",
                        padding=True, truncation=True, max_length=512)
    outputs = session.run(output_names=["last_hidden_state"],
                          input_feed=dict(inputs))
    embedding = outputs[0].mean(axis=1)
    embedding = embedding / np.linalg.norm(embedding, axis=1, keepdims=True)
    return embedding[0]

The query path passed tenant_id through to the embedder as it already passed it to the retriever. No other change was required in the pipeline. The audit log gained one field — model_version, derived from the registry entry — so that every query record stated which model had produced the embedding.

The Valid Knowledge: The Jaipur Model Knew Jaipur

Fine-tuning the Jaipur boutique's model followed the same pipeline as Book 3 Chapter 5 — 100 labeled queries, paraphrase expansion using AWS Bedrock Claude 3.5 Sonnet, triplet loss, ONNX INT8 export. Output: bge-small-jaipur-v1.onnx, uploaded to the registry path.

With the Jaipur model in the registry, the mirror-work lehenga query ran again. The embedding landed in a neighbourhood the Jaipur boutique's catalog had been trained to occupy — mirror-work embroidery, Rajasthani occasion wear, shisha-embellished festive pieces. The retriever returned correct chunks.

RAGAS before and after:

The improvement matched the pattern from Book 3 Chapter 5: faithfulness recovered because the model was retrieving chunks that actually answered the question; context precision exceeded the Book 3 baseline because the Jaipur model's embedding space was tighter around Jaipur's specific vocabulary than the zUdyog model had been even for zUdyog's own queries.

Four months after the fix, Krishna asked the question again:

"Which model served the last Vastralaya query?"

$ grep "vastralaya" logs/audit.jsonl | tail -1 | jq .model_version
"bge-small-vastralaya-v1"

That was the right kind of silence.

This article covers concepts from Book 4, Chapter 4 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to a fleet of six isolated, compliant, auditable tenants.