Lesson 04: Why Your AI Search Fails — The Secret Is How You Split Your Data

Subtitle: Learn How RAG Chunking Helps AI Find the Right Information and Answer Customer Questions.

Imagine This: A Customer Wants to Return a Saree

Varastra is an Indian ethnic wear e-commerce platform selling sarees, kurtas, linen co-ord sets, and festive clothing.

A customer asks:

Can I return the saree?

Varastra's product information contains the answer:

Returns are accepted within 7 days, provided the original tags remain attached.

But what if the AI cannot find this information, even though it is already stored in the database?

One possible reason is poor chunking.

Let's understand how chunking works and why it matters for AI search.

1. What Is Chunking?

Chunking means breaking large amounts of text into smaller, meaningful sections.

Imagine Varastra stores this complete product description as one chunk:

Silk Saree — Full Description

• Occasion: Festive wear, weddings, Diwali
• Colors: Red, gold, emerald
• Length: 5.5 metres
• Care: Dry clean only; do not machine wash
• Returns: Eligible within 7 days; original tags must remain attached
• Price: ₹4,299
• Included: Blouse piece

This single description answers many different questions:

• Is this saree suitable for a wedding?
• How should I wash it?
• Can I return it?
• What colors are available?
• How much does it cost?

When a customer asks about returns, the search system needs to find the return policy inside a description containing many other details.

This can make retrieval less effective.

2. The Left Side: One Large Chunk per Product

In the diagram, the entire saree description is stored as one large chunk.

The customer asks:

Can I return the saree?

The embedding model converts the question into a vector. The search system compares it with the vector representing the full product description.

Because the description contains several topics, its embedding represents the text as a whole rather than isolating the return policy.

The diagram shows a similarity score of 0.71, representing a weaker match in this example.

The return policy exists, but the search system may not rank the large chunk highly enough to retrieve it.

The lesson: Having the right information in your database does not guarantee that your AI will find it.

Note that large chunks do not always perform worse. Retrieval quality depends on the text, embedding model, and search method.

3. The Right Side: Attribute-Level Chunks

Instead of storing everything together, Varastra divides the description into focused chunks.

Chunk 1 — Occasion

Festive wear, weddings, Diwali.

Chunk 2 — Care

Dry clean only. Do not machine wash.

Chunk 3 — Returns

Eligible within 7 days, provided the original tags remain attached.

Chunk 4 — Details

Red, gold, emerald. Price: ₹4,299.

Each chunk focuses on a particular topic.

When the customer asks, "Can I return the saree?", the search system can retrieve the Returns chunk directly.

The diagram shows a similarity score of 0.93, indicating a stronger match in this example.

These scores are illustrative, not guaranteed results.

The key idea: Focused chunks can help AI find the exact information needed to answer a question.

4. Why Does Focused Chunking Help?

Different customer questions require different information.

When chunks focus on specific topics, the search system has a better chance of finding relevant information.

This approach is called attribute-level chunking when product information is divided according to individual attributes or fields.

It is useful for product catalogs, policies, specifications, and frequently asked questions.

5. What Does "Meaning Diluted" Mean?

Imagine asking someone, "Can I return this saree?"

Instead of showing you the return policy, they hand you a long document containing the product's price, colors, measurements, washing instructions, and return rules.

You must search through all that information to find one answer.

Now imagine receiving only the return policy.

Finding the answer becomes easier.

That is the basic difference between a large, mixed chunk and a focused chunk.

However, splitting text too aggressively can also cause problems.

For example, these two statements belong together:

• Returns are accepted within 7 days.
• Original tags must remain attached.

If the AI retrieves only the first statement, it may give an incomplete answer.

Good chunking separates unrelated topics while keeping related information together.

6. How Does the Grounding Layer Use These Chunks?

After retrieval, the Grounding Layer supplies relevant evidence to the language model.

Customer question:

"Can I return the saree?"

Retrieved evidence:

"Returns are accepted within 7 days, provided the original tags remain attached."

The language model can now respond:

Yes, you can return the saree within 7 days, provided the original tags remain attached.

The answer is supported by the retrieved policy, assuming the policy is current and applies to the customer's purchase.

Chunking helps find the evidence. The Grounding Layer supplies that evidence to the AI.

7. How Should Varastra Organize Product Information?

A practical starting point is to create chunks around meaningful topics.

Varastra can also store metadata, such as product IDs and categories, with each chunk.

This helps the application connect retrieved information to the correct product.

Changing information, such as prices and stock, should be verified against the current commerce database.

Not every attribute needs its own chunk. The best structure depends on the questions customers ask and the relationships between the information.

8. How Does Chunking Fit into RAG?

Chunking is an important part of Retrieval-Augmented Generation (RAG).

The process works like this:

1. Collect data: Gather product descriptions, policies, and attributes.
2. Create chunks: Divide the information into meaningful sections.
3. Generate embeddings: Convert each chunk into a vector.
4. Store the data: Save vectors and associated information in a searchable database.
5. Receive a question: The customer asks the AI something.
6. Retrieve evidence: Search for relevant chunks.
7. Build the prompt: Provide the question and retrieved evidence to the language model.
8. Generate the answer: The AI responds using the available evidence.

Complete workflow:

Product Data → Chunking → Embeddings → Vector Database → Retrieval → Grounding Layer → Language Model → Answer

Poor chunking can make relevant information harder to retrieve. Well-designed chunks can improve retrieval, but they do not guarantee a correct answer.

Key Takeaways

1. Chunking breaks large text into smaller, meaningful sections.
2. Large chunks can mix unrelated information.
3. Focused chunks help the search system find specific details.
4. Good chunks preserve context, including important conditions.
5. Better chunking can improve retrieval and help AI answer using relevant evidence.

The big lesson: Don't just store information for AI. Structure it so AI can find the right answer when it matters.