Multimodal RAG: CLIP, Image Queries, and the Modality Router

Draupadi set her phone on the table face-up.

The photograph on the screen was from a family wedding. The lehenga the woman wore was heavy silk in deep copper-red, the skirt embroidered from hem to waist in a dense geometric pattern, the dupatta sheer and gold-bordered, the blouse cut close. A customer would look at this photograph and spend twenty minutes trying to find it in a catalog by describing it in words.

This customer had not spent twenty minutes describing it. She had sent the photograph.

Vastralaya's chatbot had received the message, extracted the text component — do you have anything like this? — and searched the catalog for those five words. Five words that, without the photograph, described nothing at all. The system had returned the three highest-traffic lehengas in Vastralaya's inventory: a rose-gold tissue lehenga, an ivory-and-gold bridal set, and a navy embroidered piece that had nothing in common with what the customer had shown.

The customer called the support number. She described the piece to a human agent for six minutes. The agent found the match in forty seconds. The sale happened through a phone call the chatbot had been built to replace.

The Valid Failure: Translation Is Narrower Than the Original

Approach one: describe the image, search the description. Claude 3.5 Sonnet on Bedrock could describe a photograph in detail — garment type, colour, embroidery style, silhouette, occasion register. The description was accurate. Run through bge-small as a text query, it retrieved relevant products.

The problem: the description was the model's interpretation of the image, filtered through the vocabulary it knew. A piece with unusual regional embroidery — mirror-work, kantha, chikankari, zardozi — would be described in the terms the model reached for: embellished, embroidered, decorative. The specific craft vocabulary, the visual density of the pattern, the regional identity — all of it passed through a bottleneck of generic words before reaching the retriever. The customer's image was rich with specific signal. The description was a compressed, lossy summary.

There was also the latency. Calling Bedrock to describe the image before retrieval added 800 to 1,200 milliseconds to every image query. The retrieval itself ran in under thirty milliseconds. The caption step cost forty times as much as the step it was enabling.

Approach two: extract structured attributes. A structured extraction — garment type, dominant colour, embroidery presence, occasion tier — was indexed as metadata; the retrieval used metadata filters to narrow the candidate set.

The attributes were correct for Vastralaya's catalog photographs, which were clean studio shots against white backgrounds. They were wrong for Draupadi's photograph: a family wedding scene with marigold garlands, soft indoor lighting, the lehenga half-obscured by foreground hands. The attribute extractor saw a red garment with some gold. It did not see the geometric embroidery from hem to waist because the hem was not fully visible. The filter returned sixteen red occasion lehengas. None of them were the right one.

Both approaches failed for the same reason: they translated the image into text before retrieval, and translation is always narrower than what it represents. What the customer meant to show was in the image. The retriever needed to compare against the image directly.

The Valid Source: CLIP Dual-Encoder and Dual-Collection Architecture

CLIP — Contrastive Language-Image Pre-training — was trained on four hundred million image-text pairs to produce a shared embedding space where images and their corresponding descriptions land near each other. The image of a copper-red embroidered lehenga and the catalog text heavy silk lehenga in copper-red with geometric zardozi embroidery produce embeddings that are close neighbours in CLIP space — not because either was translated into the other's modality, but because both were projected into a common representation during pretraining.

Dual-collection architecture — CLIP's output is 512-dimensional, not 384. The two models' embedding spaces are not comparable. Vastralaya received a second Qdrant collection at provisioning:

def provision_vastralaya_clip_collection() -> str:
    collection_name = "shopbot_vastralaya_clip"
    client.recreate_collection(
        collection_name=collection_name,
        vectors_config=VectorParams(size=512, distance=Distance.COSINE)
    )
    client.create_payload_index(
        collection_name=collection_name,
        field_name="product_id",
        field_schema="keyword"
    )
    return collection_name

CLIP model — openai/clip-vit-base-patch32, loaded once at container start, shared across all image queries:

clip_model     = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
clip_processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
clip_model.eval()

def embed_image_clip(image: Image.Image) -> list[float]:
    inputs = clip_processor(images=image, return_tensors="pt")
    with torch.no_grad():
        features = clip_model.get_image_features(**inputs)
        features = features / features.norm(dim=-1, keepdim=True)
    return features[0].tolist()

def embed_text_clip(text: str) -> list[float]:
    inputs = clip_processor(text=text, return_tensors="pt",
                            padding=True, truncation=True, max_length=77)
    with torch.no_grad():
        features = clip_model.get_text_features(**inputs)
        features = features / features.norm(dim=-1, keepdim=True)
    return features[0].tolist()

Ingest pipeline — every Vastralaya product was embedded twice at ingest: once with bge-small for text queries, once with CLIP's image encoder for image queries:

def ingest_vastralaya_product(product: dict, image: Image.Image):
    pid = product["product_id"]
    text_vec  = embed(product["description"], tenant_id="vastralaya")
    client.upsert(collection_name="shopbot_vastralaya",
                  points=[PointStruct(id=pid, vector=text_vec.tolist(),
                                      payload={**product, "tenant_id": "vastralaya"})])
    image_vec = embed_image_clip(image)
    client.upsert(collection_name="shopbot_vastralaya_clip",
                  points=[PointStruct(id=pid, vector=image_vec,
                                      payload={**product, "tenant_id": "vastralaya"})])

The ingest ran against Vastralaya's catalog — two thousand products, each with a studio photograph and a text description. The embedding job took 41 minutes on CPU.

Modality router — query modality was detected before any embedding call, before the collection name was determined:

def retrieve_vastralaya(query: str | None, image: Image.Image | None,
                        top_k: int = 5) -> list:
    if image is not None:
        collection = "shopbot_vastralaya_clip"
        vector     = embed_image_clip(image)
    elif query:
        collection = "shopbot_vastralaya"
        vector     = embed(query, tenant_id="vastralaya").tolist()
    else:
        raise ValueError("retrieve_vastralaya requires either query or image")

    return client.search(collection_name=collection,
                         query_vector=vector, limit=top_k)

The Valid Knowledge: What the Copper-Red Lehenga Retrieved

Before — text query "do you have anything like this", bge-small:

Rank 1: Rose-gold tissue lehenga (score 0.71)
Rank 2: Ivory and gold bridal set (score 0.68)
Rank 3: Navy embroidered lehenga (score 0.65)

None of these were the piece in the photograph.

After — image query, CLIP:

Rank 1: Copper-red geometric zardozi lehenga, SKU VAS-2847 (score 0.89)
Rank 2: Deep red kantha-embroidered lehenga, SKU VAS-1923 (score 0.81)
Rank 3: Burgundy mirror-work occasion lehenga, SKU VAS-3301 (score 0.77)

VAS-2847 was the piece in the photograph.

RAGAS for image queries — a new labeled set of 120 image-query pairs was annotated by Vastralaya's catalog team over three days:

Context Precision for image queries was lower than for fine-tuned text queries (which had reached 0.92 for Jaipur in Chapter 4) because CLIP's shared embedding space was less tightly calibrated to Vastralaya's specific catalog than a bge-small model fine-tuned on that catalog would be. That gap was noted as a backlog item: a Vastralaya-specific CLIP fine-tune, using Vastralaya's own image-description pairs as training data, was the logical extension of the per-tenant model strategy from Chapter 4. For now, 0.88 was the honest starting point.

Draupadi tested the system two weeks after the whiteboard meeting. She sent the photograph of the copper-red lehenga. The response came back in four seconds. The first result was VAS-2847 — in stock at the Bhopal and Nagpur boutiques. She looked at the screen, then at Arjun.

"The Bhopal boutique manager is going to ask me why it took this long."

This article covers concepts from Book 4, Chapter 5 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to a fleet of six isolated, compliant, auditable tenants.