Semantic Chunking vs Fixed-Size Chunking in RAG

How you split documents before embedding them determines what your retriever can find. Fixed-size and semantic chunking make different trade-offs — and the wrong choice causes silent retrieval failures.

Fixed-Size Chunking

Split every document at a fixed token boundary — 256 tokens, 512 tokens, or whatever fits your context window budget. Optionally add a sliding overlap (e.g., 50 tokens) so that answers near a boundary are not lost.

What it gets right: simple to implement, deterministic, no additional model calls.

What it gets wrong: it splits at arbitrary boundaries. A sentence that begins in chunk 7 ends in chunk 8. A product's care instructions may be cut off from the product's name. The chunk is retrievable, but its meaning is incomplete.

Semantic Chunking

Split at natural language boundaries — sentence endings, paragraph breaks, or section headings — rather than at a fixed token count. Some implementations use a small model to detect topic shifts and split there.

What it gets right: each chunk is a complete thought. The embedding accurately represents a coherent piece of information. Retrieval is more precise.

What it gets wrong: variable chunk sizes make index management more complex. Embedding costs are harder to predict. Very long semantic units may still overflow the context window.

The Blob Problem and the Context Loss Problem

Both strategies fail in specific ways that the RAG Mastery Series calls the Blob Problem (one embedding per document dilutes meaning across all attributes) and the Context Loss Problem (one sentence per chunk loses its anchor to the document it came from).

The recommended strategy: chunk by concern. Group attributes that would be retrieved together — product name + dimensions + materials — into a single chunk. Do not let the chunking boundary cut across a logically unified fact.

Chunking strategy is covered in depth in Book 1, Chapter 5 of the RAG Mastery Series. Book 1 is free — no sign-in needed.