System Prompt and the Grounding Constraint in RAG

The retriever works. The chunks are structured. The vector database stores and returns the right documents. Five chapters of infrastructure assembled correctly.

Then the first unconstrained prompt produces this:

"Yes, the silk saree requires dry cleaning. It should not be machine washed, tumble dried, or wrung. Generally speaking, silk garments are delicate and benefit from professional dry cleaning after each wear to preserve the fabric's sheen and prevent damage to the weave. Zari embroidery in particular can tarnish if exposed to moisture or harsh detergents, so storing the saree in a muslin cloth away from direct sunlight is also recommended."

The first two sentences are grounded. They come directly from the retrieved chunk. The rest — the advice about professional cleaning after each wear, the muslin-cloth storage, the zari tarnishing — comes from the model's training data. None of it is in the catalog. None of it has been verified for this specific product.

The Grounding Layer delivered accurate evidence to a model with no constraint. The model used it correctly, then kept going — because nothing told it to stop. A Grounding Layer that delivers accurate evidence to an unconstrained model is not a Grounding Layer. It is a starting point for drift.

Three Drafts, Three Failure Modes

Draft one — no constraint on the model's output. Plain, polite, asks the model to use the retrieved context. The dry-cleaning answer drifts into training data the moment the retrieved chunk runs out.

Draft two — maximum constraint:

You have been given specific product information. Your answers
must come ONLY from this information. Do not guess. Do not use
general knowledge about fashion. Do not estimate or infer details
not present in the context.

Testing the same dry-cleaning question: ShopBot reproduces the retrieved chunk almost word for word. The drift is gone.

Testing "Is the saree good for a wedding?": ShopBot responds: "Silk saree with zari border. Festive wear. Ideal for weddings, receptions, and Diwali celebrations."

Accurate. Every word sourced. The customer reads a product listing, not an answer. The voice is absent. The model was told to copy and it copied — efficiently, completely, uselessly. This is a Grounding Failure of the opposite shape. Not drift from insufficient constraint, but uselessness from excessive constraint.

Draft three — the correct prompt — gives the model three instructions:

1. Answer only from the provided context. If the context does not contain the answer, say so explicitly without inferring or extending.
2. Answer in natural language. The customer asked a question — give them an answer, not a transcription.
3. If context is absent or insufficient, route the customer to the fallback.

Each instruction closes a specific failure mode. The first closes drift. The second closes robotic copying. The third handles the case where the retriever has already withheld evidence.

The reason for honesty is written into the prompt — not as a rule, but as a consequence: "A customer who receives a wrong answer with confidence cannot" — cannot recover. Cannot be corrected. Cannot get help. The model that reads this understands not just what to do but why.

The Constraint Is Probabilistic, Not Absolute

A tightly constrained system prompt reduces Confident Drift significantly. It does not eliminate it.

RAGAS evaluation of ShopBot's Book 1 baseline returns a Faithfulness score of 0.71 — seventy-one percent of the content in ShopBot's answers is directly supported by retrieved context. Twenty-nine percent is not. The prompt constraint held most of the time. On multi-product queries and price-comparison questions, the model added context the retrieved chunks did not contain, because the prompt instruction is a strong prior, not an enforcement mechanism.

The model has no internal lock that prevents it from extending into training data. The system prompt is read as very strong context. The model follows it reliably — and not perfectly. Improving retrieval quality raises Faithfulness more reliably than tightening the prompt: a model cannot add from training data what it was never given, and high-quality retrieved chunks leave fewer gaps to fill.

Temperature Zero

Language models have a temperature setting that controls randomness in generation. At temperature 0.7, the model samples from its probability distribution — phrasing varies across identical queries. At temperature 0, the model always picks the most probable next token. Same input, same output, every time.

For a product assistant, temperature 0 is not a preference. It is a requirement. The variation that makes creative text feel spontaneous at 0.7 also occasionally produces variation in facts. At 0.7, there is a non-zero probability that the model phrases something in a way that implies a detail not present in the retrieved context. At temperature 0, the highest-probability path for a model following a tight factual prompt is the faithful rendering of the retrieved information.

The Retriever and the Prompt Together

The retriever decides what evidence reaches the model. The prompt decides what the model does with that evidence.

Neither alone is sufficient. A retriever without a prompt constraint delivers accurate evidence that the model uses as a launching pad. A prompt constraint without a retriever gives the model no real evidence to constrain it to — it falls back on training data because there is nothing else.

The retriever is the gatekeeper. The prompt is the final lock. Between the two, Confident Drift has no entry point.

The engineering created the conditions for honesty. The prompt enforces it.

The Prompt Is Policy

The system prompt is not a configuration knob to be tweaked whenever an answer seems off. It is a policy document — the considered position of what this assistant is and is not, written carefully once, revised only when there is a principled reason to revise it.

Every change to the prompt is a policy change. Policy changes have consequences. They should be deliberate, tested, and documented. A tight prompt — one whose every sentence closes a specific failure mode, and whose removal would change the behaviour demonstrably — is difficult to write and worth deploying. A padded prompt whose sentences can be removed without effect is not a grounding constraint. It is decoration.

This article covers concepts from Book 1, Chapter 7 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.