How to Deploy a FastAPI LLM App to Production

A RAG chain that only runs on your laptop is not a product. This article covers the steps from a working local script to a live, persistent, publicly accessible API — using FastAPI, Railway, and Qdrant Cloud.

The Valid Failure

ShopBot's first deployment attempt used uvicorn main:app inside a Railway container. It started successfully. Then it tried to connect to http://localhost:6333 — the local Qdrant instance that existed only on the developer's machine. The retrieval layer threw a connection error on every request.

The local development setup assumed services that did not exist in the deployment environment. This is the most common first-deployment failure for LLM applications: the code is correct, the infrastructure assumptions are wrong.

The Three Infrastructure Components

A production RAG API has three services, each needing a persistent address:

1. The FastAPI application — handles HTTP requests, orchestrates retrieval and generation
2. The vector database — stores and searches embeddings (Qdrant Cloud, not localhost)
3. The LLM — OpenAI API (external) or a self-hosted model

In development, all three can run locally. In production, each must be independently addressable via environment variables.

Step 1: Externalise Every Hardcoded Address

Never hardcode a URL, key, or host. Use environment variables for everything that changes between environments.

# config.py
import os
from dotenv import load_dotenv

load_dotenv()

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
QDRANT_URL = os.getenv("QDRANT_URL")          # http://localhost:6333 in dev
QDRANT_API_KEY = os.getenv("QDRANT_API_KEY")  # empty in dev, required in prod
COLLECTION_NAME = os.getenv("COLLECTION_NAME", "shopbot")

# retrieve.py
from config import QDRANT_URL, QDRANT_API_KEY, COLLECTION_NAME
from qdrant_client import QdrantClient

client = QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY or None)

Step 2: The FastAPI App

# main.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, field_validator
from retrieve import retrieve
from generate import generate
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

app = FastAPI(title="ShopBot API", version="1.0.0")

class Query(BaseModel):
    question: str

    @field_validator("question")
    @classmethod
    def question_not_empty(cls, v: str) -> str:
        v = v.strip()
        if not v:
            raise ValueError("question cannot be empty")
        if len(v) > 500:
            raise ValueError("question must be under 500 characters")
        return v

@app.get("/health")
def health():
    return {"status": "ok"}

@app.post("/chat")
async def chat(query: Query):
    try:
        chunks = retrieve(query.question)
        answer = generate(query.question, chunks)
        logger.info(f"Query: {query.question[:60]} | Chunks: {len(chunks)}")
        return {"answer": answer, "chunks_used": len(chunks)}
    except Exception as e:
        logger.error(f"Error: {e}")
        raise HTTPException(status_code=500, detail="Internal error")

The /health endpoint is required by Railway (and most cloud platforms) for uptime monitoring. Without it, the platform cannot determine whether the container is alive.

Step 3: The Dockerfile

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

--host 0.0.0.0 is required. Without it, uvicorn binds to 127.0.0.1 and the container accepts no external traffic.

Step 4: Deploy Qdrant Cloud

The local Qdrant instance disappears when your laptop closes. Use Qdrant Cloud for a persistent endpoint.

1. Create a free cluster at cloud.qdrant.io
2. Copy the cluster URL and API key
3. Run your ingest script against the cloud instance before deployment:

QDRANT_URL="https://your-cluster.qdrant.io" \
QDRANT_API_KEY="your-key" \
python ingest.py

The vectors now live in Qdrant Cloud. The deployed app can reach them.

Step 5: Deploy to Railway

Railway deploys from a GitHub repository. Push your code, connect the repo, and set environment variables in the Railway dashboard:

OPENAI_API_KEY=sk-...
QDRANT_URL=https://your-cluster.qdrant.io
QDRANT_API_KEY=your-qdrant-key
COLLECTION_NAME=shopbot

Railway auto-detects the Dockerfile and deploys. When deployment completes, Railway provides a public URL: https://shopbot-production.up.railway.app.

Step 6: Test the Live API

curl -X POST https://shopbot-production.up.railway.app/chat \
  -H "Content-Type: application/json" \
  -d '{"question": "Does the silk saree have a cotton lining?"}'

Expected response:

{
  "answer": "The silk saree does not have a cotton lining. It is made from pure Banarasi silk with a zari border.",
  "chunks_used": 2
}

Common First-Deployment Failures

The Valid Knowledge

• Environment variables, not hardcoded values: every address, key, and collection name that differs between environments must be an env var. This is not a best practice — it is a deployment requirement.
• The health endpoint is infrastructure: it tells the platform the app is alive. Without it, the platform restarts healthy containers.
• Ingest before you deploy: the vector database must be populated before the app starts. An empty collection returns zero results for every query.
• --host 0.0.0.0 is required: uvicorn's default 127.0.0.1 means "only accept connections from inside this container" — which means no external traffic.
• Pydantic validation at the boundary: validate all user input at the API layer. An unconstrained question field that accepts unlimited text can cause timeout failures at the embedding and LLM steps.