How to Choose the Right Chunk Size for RAG (With Numbers)

Chunk size is not a hyperparameter to tune. It is a decision about what unit of information your retriever will search across. Get it wrong and retrieval fails in ways that look like model failures.

The Valid Failure

ShopBot's first ingestion used one embedding per product. A product description contained fabric, care instructions, size guide, price, and availability — all in one chunk, all in one vector.

A customer asked: "How do I wash this silk saree?" The silk saree's care instruction was in the database. But the embedding for that product had averaged across all five attribute clusters. The care instruction was diluted. The retrieval score for the relevant product was 0.64 — below the 0.72 threshold.

The retriever returned nothing. The chatbot said it had no information. The care instruction was there. The chunking strategy made it unretrievable. This is called the blob problem.

The second attempt used one sentence per chunk. "Hand wash only in cold water" became a chunk. It retrieved correctly for washing queries. But there was no product name in the chunk. The model received: "Hand wash only in cold water" and generated: "This product should be hand washed in cold water." Which product? Unknown.

This is called context loss.

The Two Failure Modes

Both failure modes produce wrong retrieval scores or wrong answers. Neither is a model problem.

The Fix: Chunk by Concern

Chunk by concern means each chunk contains one coherent unit of information, with enough surrounding context to be answered from without the rest of the document.

For a fashion e-commerce product:

def chunk_product(product: dict) -> list[dict]:
    base = f"{product['name']} ({product['sku']})"
    return [
        {
            "text": f"{base} — Fabric and materials: {product['fabric']}",
            "type": "fabric",
        },
        {
            "text": f"{base} — Care instructions: {product['care']}",
            "type": "care",
        },
        {
            "text": f"{base} — Available sizes: {', '.join(product['sizes'])}. {product['fit_notes']}",
            "type": "sizing",
        },
        {
            "text": f"{base} — Price: ₹{product['price']}. {product['availability']}",
            "type": "pricing",
        },
    ]

Each chunk has:

1. A product anchor (base) — so the model always knows which product it is reading about
2. A concern label — so the chunk is semantically cohesive and retrieves precisely
3. Enough text to answer a full question — not a fragment

What the Numbers Look Like

After switching from one-chunk-per-product to chunk-by-concern on ShopBot's 850-product catalog:

Context Precision (the fraction of retrieved chunks that were actually relevant) moved from 0.74 to 0.88 after re-ingesting with the chunk-by-concern strategy.

Character Count vs Token Count

Chunk size is usually discussed in tokens (for LLM context budgets) or characters (for practical splitting). The two rules that matter:

1. Minimum: a chunk must contain enough text to be answerable from — usually 50–200 tokens. A single sentence is often too short unless it is a complete, self-contained fact.
2. Maximum: a chunk must not contain so much text that the vector averages across too many concepts — usually 200–500 tokens. Beyond 500 tokens, retrieval scores for specific sub-topics drop.

For product descriptions, 100–250 tokens per concern chunk works well. For long documents (policies, contracts, technical manuals), 250–500 tokens with 50-token overlaps between adjacent chunks prevents answers from falling on chunk boundaries.

Overlapping Chunks

When documents are long and answers might span a boundary, use overlapping chunks:

def sliding_chunks(text: str, size: int = 400, overlap: int = 50) -> list[str]:
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = min(start + size, len(words))
        chunks.append(" ".join(words[start:end]))
        start += size - overlap
    return chunks

Overlap ensures that a sentence split across a boundary appears in full in at least one chunk.

The Valid Knowledge

• Chunking strategy is a retrieval decision, not a text processing decision: the right chunk size is the one that makes the information your users ask about retrievable at high similarity scores.
• Chunk by concern for structured data: product catalogs, FAQs, documentation with clear sections. One coherent attribute cluster per chunk, with a product anchor.
• Sliding window with overlap for long documents: policies, contracts, manuals. 300–500 tokens, 50-token overlap.
• Measure with RAGAS Context Precision after re-ingesting: the metric tells you what fraction of retrieved chunks were relevant. If it is below 0.80, your chunking strategy is the first thing to investigate.
• Re-ingestion is cheap compared to wrong answers: re-chunking and re-embedding an 850-product catalog takes under two minutes. The retrieval quality improvement from getting chunking right is permanent.