What Is a System Prompt and Why It Matters in RAG

A system prompt is the instruction the LLM reads before it reads anything the user sent. In RAG, it is the final lock between correct retrieval and a grounded answer. Without it, a model that retrieved the right evidence will still hallucinate.

The Valid Failure

ShopBot had correct retrieval. The linen co-ord's care instruction was retrieved at score 0.89. The context was passed to the model. The system prompt said: "You are a helpful fashion assistant. Be friendly and informative."

The model read the care instruction, then elaborated. It confirmed the hand-wash instruction from the retrieved chunk — and added: "You may also dry-clean this garment, which is gentler on the linen fibres."

The care instruction said nothing about dry-cleaning. The product was not dry-clean safe. The addition came from the model's training data — from patterns it had absorbed about linen care generally.

The retrieval layer worked. The system prompt did not constrain generation. The model drifted from retrieved evidence into training memory. This is the exact failure RAG was designed to prevent, occurring inside a RAG system with a correct retrieval score.

What the System Prompt Does

The system prompt has two jobs in a RAG system:

1. Define the persona and scope — what the assistant is and what questions it handles
2. Install the grounding constraint — the explicit instruction that the model may only speak from the retrieved context, not from training memory

Without the grounding constraint, the model treats retrieved evidence as a starting point. With it, the model treats retrieved evidence as the only permitted source.

A Minimal Grounding Constraint

SYSTEM_PROMPT = """You are ShopBot, a customer service assistant for an Indian fashion e-commerce store.

IMPORTANT GROUNDING RULE:
Answer ONLY from the context provided below each question.
Do not add information from your training, general knowledge, or assumptions.
If the context does not contain enough information to answer the question fully and accurately, respond:
"I don't have enough information about that in our catalog. Please contact our support team."

Never:
- Confirm features not mentioned in the context
- Suggest alternatives not present in the context
- Add care, sizing, or material details not stated in the context
"""

How It Works at Runtime

def generate(query: str, chunks: list[dict]) -> str:
    if not chunks:
        return "I don't have enough information about that in our catalog."

    context = "\n\n".join(
        f"[Source {i+1}]\n{c['text']}" for i, c in enumerate(chunks)
    )

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
        ],
    )
    return response.choices[0].message.content

The user message contains both the retrieved context and the question. The system prompt has already told the model what it is permitted to do with that context.

temperature=0 and Why It Matters

temperature=0 minimises sampling randomness. At temperature 1.0, the model samples from a probability distribution over all possible next tokens. At temperature 0.0, it always takes the highest-probability token.

This does not eliminate hallucination — a model at temperature 0 will still generate from training memory if the grounding constraint is absent or weak. But it reduces the variance that causes models to add synonyms, elaborations, and "helpful" extras that were not in the retrieved context.

Think of temperature as the width of the generation path and the grounding constraint as the walls of a channel. You need both.

Testing the Grounding Constraint

Three test queries to verify your system prompt is working:

# 1. Query with strong evidence — should answer precisely from context
"What fabric is the silk saree made of?"

# 2. Query with partial evidence — should answer what it knows and flag what it doesn't
"Is the linen co-ord available in XL and is it dry-clean safe?"

# 3. Query with no evidence — must refuse, not invent
"Do you stock traditional Rajasthani jewellery?"

If the model answers query 3 with product suggestions it was not given in context, the grounding constraint is not working. Tighten the constraint language: "You MUST NOT suggest, infer, or speculate about products or features not explicitly present in the provided context."

The Difference Between a Soft and Hard Constraint

Soft constraints reduce hallucination. Hard constraints eliminate it for well-retrieved queries. Production systems use hard constraints.

The Valid Knowledge

• The system prompt is not a style guide: it is a safety boundary. In RAG, the most important line in the system prompt is the grounding constraint, not the persona description.
• Retrieval quality and prompt constraints are independent failure modes: correct retrieval with a weak prompt still produces hallucination. The system prompt is the last line of defence.
• temperature=0 reduces variance, not hallucination: it shrinks the surface area where the model can drift, but only the grounding constraint closes the door.
• Test against no-evidence queries: the model's response to a question with zero retrieved context is the clearest test of whether your system prompt is actually constraining generation.
• RAGAS Faithfulness measures grounding constraint effectiveness: it asks "what fraction of claims in the answer are supported by the retrieved context?" A Faithfulness score below 0.85 often indicates a weak system prompt, not poor retrieval.