The Prompt Is the Product

There is a moment in every technical project where the engineering is done and the design begins.

The retriever works. The chunks are structured. The vector database stores and returns the right documents. The mechanical parts of ShopBot are assembled and tested.

Now Arjun must answer a different kind of question. What kind of assistant is ShopBot?

Not what it can do. What it is allowed to do. What it must never do. How it speaks when it knows something, and what it says when it does not. Where the boundary sits between helpful and harmful.

These questions are not answered in code. They are answered in the system prompt — the set of instructions the model receives before every customer interaction. The system prompt is ShopBot's character, its values, its constraints, all compressed into a paragraph or two of text.

Writing it badly produces an assistant that sounds helpful and is not. Writing it well produces one that can be trusted.

He stared at the blank prompt file for eleven minutes, and the long pause was not procrastination but the small uncomfortable recognition that this single paragraph would matter more than any of the code that came before it. The chai on his desk has gone cold. He does not notice.

First Draft — The Charitable Reading

# src/prompt.py — first draft
from langchain_openai import ChatOpenAI
from langchain.schema import SystemMessage, HumanMessage

SYSTEM_V1 = """You are ShopBot, a helpful product assistant
for zUdyog Fashion, an Indian fashion e-commerce store.

Answer the customer's question using the product information
provided below.

Product Information:
{context}
"""

Plain. Polite. Asks the model to use the retrieved context. Twenty minutes of writing. He connects it to the chain, runs three test queries, reads the outputs.

The answers are good. Conversational, accurate, helpful. He shows them to Krishna.

Krishna reads them in silence. Reads the system prompt. Reads the answers again.

"What happens," Krishna says, "when the product information does not contain the answer?"

Arjun runs a test. "Does the silk saree need dry cleaning?"

The retrieved chunk says, exactly: "Silk saree care instructions: dry clean only. Do not machine wash, tumble dry, or wring."

ShopBot's output:

"Yes, the silk saree requires dry cleaning. It should not be machine washed, tumble dried, or wrung. Generally speaking, silk garments are delicate and benefit from professional dry cleaning after each wear to preserve the fabric's sheen and prevent damage to the weave. Zari embroidery in particular can tarnish if exposed to moisture or harsh detergents, so storing the saree in a muslin cloth away from direct sunlight is also recommended."

The first two sentences are grounded. They come directly from the retrieved chunk. The rest — the advice about professional cleaning after each wear, the muslin-cloth storage, the zari tarnishing — comes from the model's training data. None of it is in zUdyog's catalog. None of it has been verified for this specific product.

It sounds authoritative. It reads naturally. A customer would not know where the retrieved evidence ended and the model's extrapolation began.

This is Confident Drift in its most dangerous form — not the obvious fabrication of Chapter 1, but the subtle extension of correct information into unverified territory. The retrieved chunk was right. The model used it correctly. Then it kept going, because nothing told it to stop.

A Grounding Layer that delivers accurate evidence to a model with no constraint is not a Grounding Layer. It is a starting point for drift.

The Argument

Arjun reads the muslin-cloth answer aloud to Krishna.

"That is a hallucination," Krishna says.

"Yes."

"The prompt does not prevent it."

"No."

"Then the prompt is not finished."

Arjun knows Krishna is right. He also knows that fixing this means writing a prompt that feels restrictive — that will sometimes make ShopBot say I don't know when a more confident answer would feel more helpful to the customer in the moment.

He says this out loud. Krishna's response is not what he expects.

"A customer who trusts a wrong answer and acts on it is worse than a customer who receives no answer and contacts support," Krishna says. "The first customer files a return. The second customer gets a phone call. One costs more."

He does not say it as a business argument. He says it as a matter of principle.

Arjun goes back to the prompt file.

Second Draft — Closing the Door

# src/prompt.py — second draft
SYSTEM_V2 = """You are ShopBot, a product assistant for zUdyog Fashion.

You have been given specific product information. Your answers
must come ONLY from this information.

If the answer is not in the product information below, say:
"I don't have that information. Please contact our support team
at support@zudyog.com."

Do not guess. Do not use general knowledge about fashion.
Do not estimate or infer details not present in the context.

Product Information:
{context}
"""

He tests the same dry-cleaning question. ShopBot now reproduces the retrieved chunk almost word for word. The drift is gone.

Then he tests "Is the saree good for a wedding?" The retrieved chunk says: "Silk saree with zari border — festive wear, ideal for weddings, receptions, and Diwali celebrations."

ShopBot's output: "Silk saree with zari border. Festive wear. Ideal for weddings, receptions, and Diwali celebrations."

The evidence is correct. Every word is sourced. The answer is technically accurate.

The customer reads a list, not an answer. The voice is absent. The model was told to copy and it copied — efficiently, completely, uselessly. No customer reads a product chunk and feels served. They read a database entry and feel like they have been handed a file.

A model that cannot speak is not a voice. It is a display.

This is a Grounding Failure of the opposite shape. Not drift from insufficient constraint, but uselessness from excessive constraint. Both fail the customer. Both fail ShopBot. The correct prompt lives between them.

Third Draft — Character Without Freedom

The problem with the second draft is that it constrains what ShopBot says without giving it a voice for how to say it. The instructions are all negative — do not guess, do not estimate, do not infer. There is nothing positive about how to respond when the information is present.

The right prompt gives the model three instructions. Answer only from the provided context — if the context does not contain the answer, say so explicitly without inferring or extending. Answer in natural language — the customer asked a question, give them an answer, not a transcription. If context is absent or insufficient, route the customer to the fallback. Each instruction closes a specific failure mode. The first closes drift. The second closes robotic copying. The third handles the case the retriever has already withheld evidence from.

# src/prompt.py — final
from langchain_openai import ChatOpenAI
from langchain.schema import SystemMessage, HumanMessage
from dotenv import load_dotenv
from src.retriever import retrieve

load_dotenv()

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

SHOPBOT_SYSTEM = """You are ShopBot, a warm and knowledgeable product
assistant for zUdyog Fashion — an Indian fashion e-commerce store
specialising in kurtas, sarees, anarkali suits, and contemporary Indian
fashion.

YOUR ONLY SOURCE OF TRUTH is the Product Information provided below.
You have no other knowledge about zUdyog's products.

When you have the information:
- Answer directly and specifically. Include relevant details: size,
  colour, price, occasion suitability, care instructions.
- Be warm and conversational. You are helping someone choose clothing
  — that is a personal decision that deserves care.
- If multiple products are relevant, mention all of them briefly.

When you do NOT have the information:
- Say exactly: "I don't have that specific information. Please reach
  our support team at support@zudyog.com for help."
- Do not guess. Do not approximate. Do not use fashion industry
  knowledge to fill gaps.
- A customer who receives an honest "I don't know" can get the right
  answer from support. A customer who receives a wrong answer with
  confidence cannot.

Product Information:
{context}
"""


def answer(query: str) -> str:
    """
    Retrieve evidence for a query and generate a grounded response.
    Called by evaluation/evaluate.py to produce answers for RAGAS scoring.
    """
    evidence = retrieve(query)

    if evidence is None:
        return (
            "I don't have a product that matches that specifically. "
            "Could you describe what you're looking for differently, "
            "or contact our team at support@zudyog.com?"
        )

    context = "\n\n---\n\n".join(
        f"[{e['metadata']['chunk_type'].upper()}]\n{e['text']}"
        for e in evidence
    )

    messages = [
        SystemMessage(content=SHOPBOT_SYSTEM.format(context=context)),
        HumanMessage(content=query),
    ]
    response = llm.invoke(messages)
    return response.content

The third draft has a positive voice for successful answers and a principled, specific instruction for failures. The reason for honesty is written into the prompt — not as a rule, but as a consequence. A customer who receives a wrong answer with confidence cannot — cannot what? Cannot recover. The model that reads this understands not just what to do but why.

Why Temperature Must Be Zero

Before connecting the prompt to the chain, one parameter decision deserves explicit justification.

Language models have a setting called temperature — a number between 0 and 1 that controls how much randomness enters the text-generation process. At temperature 1.0 the model samples freely from its probability distribution. It might use a different word each time it answers the same question. Responses feel varied, creative, sometimes surprising. At temperature 0 the model always picks the most probable next token. Same question, same context, same answer every time. Fully deterministic.

For a creative writing assistant or a conversational AI meant to feel spontaneous, temperature 0.7 or 0.8 is appropriate. Variation makes the responses feel more natural.

For ShopBot, temperature 0 is not a preference — it is a requirement.

At temperature 0.7, ShopBot might answer "Does the anarkali come in XL?" with "Yes, the anarkali is available in XL" on one query and "The anarkali suit offers sizes ranging from S through XXL, so XL is absolutely available" on the next — both correct. But the same temperature that produces variation in phrasing also occasionally produces variation in facts. At 0.7, there is a non-zero probability that the model phrases something in a way that implies a detail not present in the context. Small probability. Real consequence.

At temperature 0, the same input always produces the same output. There is no random variation to accidentally produce a fabricated detail. The model is constrained to its highest-probability path — which, for a model following a tight factual prompt, is the faithful rendering of the retrieved information.

Building the Chain

# src/chain.py
from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain.prompts import ChatPromptTemplate
from src.retriever import get_retriever
from src.prompt import SHOPBOT_SYSTEM
from dotenv import load_dotenv

load_dotenv()


def format_docs(docs: list) -> str:
    """
    Format retrieved chunks into a single context string for the prompt.
    Numbered and labelled — the model can distinguish multiple chunks.
    """
    if not docs:
        return "No product information available."
    formatted = []
    for i, doc in enumerate(docs, 1):
        chunk_type = doc.metadata.get("chunk_type", "general")
        formatted.append(f"[Item {i} — {chunk_type}]\n{doc.page_content}")
    return "\n\n---\n\n".join(formatted)


def build_chain():
    retriever = get_retriever(k=3)
    llm = ChatOpenAI(
        model="gpt-4o-mini",
        temperature=0,  # Deterministic. Always.
    )
    prompt = ChatPromptTemplate.from_messages([
        ("system", SHOPBOT_SYSTEM),
        ("human", "{question}"),
    ])
    chain = (
        {
            "context":  retriever | format_docs,
            "question": RunnablePassthrough(),
        }
        | prompt
        | llm
        | StrOutputParser()
    )
    return chain

The format_docs function deserves a note. It labels each chunk with its type — [Item 1 — care], [Item 2 — policy] — before passing context to the model. This labelling helps the model navigate multiple chunks. Without labels, the model receives a wall of text and must infer structure. With labels, it knows exactly what kind of evidence each chunk represents. For queries that return chunks from different products, the labels also help the model distinguish which information belongs to which product. Small addition. Measurable improvement in answer coherence.

The Complete Test

shopbot = build_chain()

test_cases = [
    "Does the embroidered anarkali suit come with a dupatta?",
    "What is the return policy for the silk saree?",
    "Does the cotton kurta shrink after washing?",
    "Do you sell men's sherwanis?",
    "Does the silk saree come with a blouse?",
]

for question in test_cases:
    print(f"\nCustomer: {question}")
    print(f"ShopBot:  {shopbot.invoke(question)}")
    print("-" * 60)

Output:

Customer: Does the embroidered anarkali suit come with a dupatta?
ShopBot:  Yes, the embroidered anarkali suit includes a dupatta. It comes
          in teal, available in sizes S to XXL, priced at ₹3,499.

Customer: What is the return policy for the silk saree?
ShopBot:  The silk saree can be returned within 7 days of delivery. The
          saree must be unstitched, unused, and in its original packaging.
          Colour exchanges are available within 5 days.

Customer: Does the cotton kurta shrink after washing?
ShopBot:  Minimal shrinkage of 2–3% may occur after the first wash. To
          maintain the fit, wash in cold water and air dry rather than
          tumble drying.

Customer: Do you sell men's sherwanis?
ShopBot:  I don't have that specific information. Please reach our support
          team at support@zudyog.com for help.

Customer: Does the silk saree come with a blouse?
ShopBot:  I don't have that specific information. Please reach our support
          team at support@zudyog.com for help.

The first three answers are accurate, specific, and warm. The fourth — "Do you sell men's sherwanis?" — deflects correctly: zUdyog carries only women's clothing. The fifth — the blouse question that produced a hallucination under the first prompt — now produces an honest admission.

That last answer is the one that matters. Not because it is impressive. Because it is what the system should say when it does not know. The customer asking about blouses receives no wrong information. They receive a path to the right answer.

Before and After

Same query. Five chapters of infrastructure delivered the same evidence in both cases. Only the prompt changed.

Before — unconstrained prompt:

"Yes, the silk saree requires dry cleaning. It should not be machine washed, tumble dried, or wrung. Generally speaking, silk garments are delicate and benefit from professional dry cleaning after each wear to preserve the fabric's sheen and prevent damage to the weave. Zari embroidery in particular can tarnish if exposed to moisture or harsh detergents..."

Correct start. Unverified extension. No signal to the customer that the last three sentences came from training, not from zUdyog's catalog.

After — constrained prompt:

"Yes, the silk saree needs to be dry cleaned. Please avoid machine washing, tumble drying, or wringing it."

Two sentences. Both sourced from the retrieved chunk. Nothing added. The customer has the answer. The Grounding Layer held.

The difference between these two outputs is one paragraph in the system prompt.

The Prompt Is Policy

Arjun saves the prompt file. Adds a comment at the top.

# SHOPBOT SYSTEM PROMPT
# Version: 1.0
# Last revised: [date]
# Change log: V1 (no honesty constraint) → V2 (too restrictive,
#             no warmth) → V3 (honest and warm). Tested against
#             5 cases including the dry-cleaning subtle-drift case
#             and the blouse hallucination case.

The comment is not for the machine. It is for whoever reads this file next — including Arjun himself, six months from now, when he has forgotten why it was written this way.

Drona's principle applies here too — structure decided once, correctly, so you do not decide it again. The prompt is not a configuration knob to be tweaked whenever an answer seems off. It is a policy document — the considered position of what this assistant is and is not, written carefully once, revised only when there is a principled reason to revise it. Every change to the prompt is a policy change. Policy changes have consequences. They should be deliberate, tested, and documented.

The Final Lock

The retriever decides what evidence reaches the model. The prompt decides what the model does with that evidence.

Both decisions are part of the Grounding Layer. The retriever is the gatekeeper. The prompt is the final lock. Without the retriever, irrelevant evidence enters. Without the prompt constraint, relevant evidence is used as a starting point rather than a boundary.

The prompt is the final clause of the Pramana contract — the instruction that tells the model it may only speak from what has been verified. The retrieved chunks are the valid sources. The prompt enforces that validity. Between the two, Confident Drift has no entry point.

A Reader Exercise

Take the system prompt and deliberately weaken one constraint. Remove the line "Do not guess. Do not approximate." Re-run the dry-cleaning test and the blouse test.

Observe what changes. Does ShopBot stay within the retrieved context? Does it start extending into training data on the dry-cleaning question? Does it invent details about the blouse?

Then restore the original constraint and re-run. Observe that the behaviour returns. The exercise makes the prompt's constraints concrete and tangible. You will see exactly what each line prevents by removing it. A system prompt whose every sentence can be removed and its absence felt is a tight prompt. A prompt that contains sentences whose removal changes nothing is a padded prompt. Tight prompts are difficult to write. They are also the only kind worth deploying. [Effort: 25 minutes.]

What Just Happened

Three prompt drafts. The first allowed Confident Drift by omission — correct beginning, unverified extension. The second prevented drift but removed the assistant's voice — accurate, robotic, useless to a customer. The third gave the model a positive instruction for success and a principled instruction for failure, with the reason for honesty written into the prompt. Temperature zero removed random variation in facts. Format labels helped the model distinguish chunk types when multiple were retrieved. The same evidence, delivered through five chapters of infrastructure, produced a confident hallucination under the first prompt and a grounded answer under the third.

The retriever decides what evidence reaches the model. The prompt decides what the model does with that evidence. Together they are the Grounding Layer. Apart, drift returns.

Five chapters of code — embeddings, ChromaDB, chunking, retrieval, thresholds. One paragraph of English.

The paragraph governs everything. It tells the model who it is, what it is permitted to say, and what to do when the evidence is not enough. It is the only part of ShopBot that Arjun writes in the language he speaks rather than the language a machine runs. And it is the part that determines whether the machine behaves.

That contrast — five chapters of infrastructure, one paragraph of instruction — is worth sitting with. The engineering created the conditions for honesty. The English enforced it.

ShopBot now understands meaning, stores evidence, retrieves with precision, and speaks from what it receives. Every component works. The problem is that only Arjun can run it. Chapter 8 wraps the whole pipeline behind a FastAPI endpoint — the door that lets anyone with a URL send a question and receive an answer. The Grounding Layer does not change. The access does.