How to Build a RAG Chatbot with Python from Scratch

Most LLM tutorials end with a chatbot that makes things up. This one ends with a chatbot that only answers from retrieved evidence — and refuses gracefully when it cannot find any.

The Valid Failure

The first version of ShopBot — the fashion e-commerce chatbot in the RAG Mastery Series — had no retrieval layer. A customer asked: "Does this kurta have a cotton lining?" The model answered: "Yes, this kurta features a soft cotton lining for added comfort."

The kurta had no lining. The customer ordered it, received it, and filed a return.

The model did not malfunction. It did what LLMs do: it predicted the most plausible continuation of the conversation. That prediction was wrong, stated with full confidence, with no signal to the customer or developer that anything had failed. This failure mode is called Confident Drift.

The fix is not a better prompt. The fix is retrieval: place the actual product specification in front of the model before it generates a single word.

What You Will Build

A RAG chatbot has three components:

1. Data layer — product descriptions chunked into retrievable units and stored as vector embeddings
2. Retrieval layer — a search that finds the most relevant chunk for each query
3. Generation layer — an LLM that reads the retrieved chunk and answers from it, not from memory

When all three work, the chatbot answers from evidence. When retrieval fails, the chatbot refuses — it does not invent.

Step 1: Project Structure

shopbot/
├── .env                  # API keys — never committed
├── .gitignore
├── requirements.txt      # pinned versions
├── ingest.py             # load, chunk, embed, store
├── retrieve.py           # query → nearest chunks
├── generate.py           # chunks + query → answer
└── main.py               # FastAPI endpoint

Pin your dependencies in requirements.txt:

openai==1.30.1
qdrant-client==1.9.1
sentence-transformers==3.0.1
fastapi==0.111.0
uvicorn==0.30.1
python-dotenv==1.0.1

Step 2: Chunk and Embed Your Data

Each product gets chunked by concern — one chunk per attribute cluster, not one chunk per product and not one sentence per chunk.

# ingest.py
import json
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, PointStruct
import uuid

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
client = QdrantClient(url="http://localhost:6333")

COLLECTION = "shopbot"
client.recreate_collection(
    collection_name=COLLECTION,
    vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)

with open("products.json") as f:
    products = json.load(f)

points = []
for product in products:
    for chunk in product["chunks"]:
        vector = model.encode(chunk["text"]).tolist()
        points.append(PointStruct(
            id=str(uuid.uuid4()),
            vector=vector,
            payload={"text": chunk["text"], "product_id": product["id"]},
        ))

client.upsert(collection_name=COLLECTION, points=points)
print(f"Ingested {len(points)} chunks")

Step 3: Retrieve

# retrieve.py
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
client = QdrantClient(url="http://localhost:6333")
COLLECTION = "shopbot"
THRESHOLD = 0.72

def retrieve(query: str, top_k: int = 3) -> list[dict]:
    vector = model.encode(query).tolist()
    results = client.search(
        collection_name=COLLECTION,
        query_vector=vector,
        limit=top_k,
        score_threshold=THRESHOLD,
    )
    return [{"text": r.payload["text"], "score": r.score} for r in results]

The score_threshold=0.72 is critical. Without it, the retriever always returns something — even a 0.51 similarity score that has nothing to do with the query. That retrieved noise becomes the context the model generates from.

Step 4: Generate with a Grounding Constraint

# generate.py
import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

SYSTEM_PROMPT = """You are ShopBot, a fashion assistant.
Answer ONLY from the context provided below.
If the context does not contain enough information to answer, respond:
"I don't have enough information about that in our catalog."
Never invent product details, prices, or availability."""

def generate(query: str, chunks: list[dict]) -> str:
    if not chunks:
        return "I don't have enough information about that in our catalog."

    context = "\n\n".join(c["text"] for c in chunks)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
        ],
    )
    return response.choices[0].message.content

temperature=0 is not a style choice — it minimises the sampling randomness that produces variation in factual claims. The grounding constraint in the system prompt is the instruction that tells the model it may only speak from what it was given.

Step 5: Wire It Together as a FastAPI Endpoint

# main.py
from fastapi import FastAPI
from pydantic import BaseModel
from retrieve import retrieve
from generate import generate

app = FastAPI()

class Query(BaseModel):
    question: str

@app.post("/chat")
def chat(query: Query):
    chunks = retrieve(query.question)
    answer = generate(query.question, chunks)
    return {"answer": answer, "chunks_used": len(chunks)}

Run it:

uvicorn main:app --reload
curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"question": "Does the silk kurta have a lining?"}'

What You Have Now

A chatbot that answers from retrieved evidence and refuses when evidence is absent. No hallucination on the questions your product catalog can answer. Clean refusals on questions it cannot.

The RAGAS Faithfulness score for this architecture at the end of Book 1 was 0.71 — meaning 71% of claims in every answer were directly supported by the retrieved context. That is the baseline. Books 2–4 show how to get it to 0.91.

The Valid Knowledge

• Chunking strategy matters: one embedding per product dilutes meaning across all attributes (the blob problem); one sentence per chunk loses the product anchor (context loss). Chunk by concern.
• The threshold is a calibration decision: set it by testing against real queries, not by intuition. 0.72 is a starting point, not a rule.
• The grounding constraint is not optional: a system prompt without it lets the model use retrieved evidence as a starting point for drift — not as the only permitted source.
• temperature=0 reduces variance: it does not eliminate hallucination. The grounding constraint does.

The full production version of this system — with hybrid search, reranking, conversation memory, and multi-tenant isolation — is built step by step in the RAG Mastery Series.