Subtitle: Learn how to evaluate AI answers, measure retrieval quality, and identify problems before your customers do.
1. Why Does AI Evaluation Matter?
Imagine Varastra, an Indian ethnic wear e-commerce platform, has launched an AI shopping assistant called ShopBot.
A customer asks:
"Can I return this silk saree after 5 days?"
ShopBot searches Varastra's product information and generates an answer.
But how do we know whether the answer is correct? How can we check whether ShopBot retrieved the right information or added details that were never provided?
Simply testing a few questions and deciding that the answers look good is not enough.
We need a way to measure how well the AI performs.
This is where RAGAS (Retrieval Augmented Generation Assessment) comes in.
RAGAS is a framework for evaluating RAG applications. It provides metrics that help developers assess retrieval quality and the quality of generated answers.
The diagram shows ShopBot's baseline evaluation: its initial performance before improvements are made.
2. Understanding the Evaluation Table
The diagram shows four evaluation metrics:
| Metric | Score | What It Measures |
|---|---|---|
| Faithfulness | 0.71 | Whether the answer is supported by the retrieved context |
| Context Precision | 0.82 | Whether the retrieved information is relevant |
| Answer Relevance | Pending | Whether the answer addresses the customer's question |
| Context Recall | Pending | Whether the retrieval process finds the relevant information needed to answer |
The first two metrics have baseline scores. The other two are marked as pending, meaning their scores have not yet been established in this evaluation.
Important: These scores are evaluation results, not percentages of all customer questions answered correctly.
3. Faithfulness — Does the AI Stay Within the Evidence?
The diagram shows a faithfulness score of 0.71.
Faithfulness measures whether the claims in the generated answer are supported by the retrieved context.
Let's use Varastra as an example.
Product information retrieved from the database:
- Returns are allowed within 7 days.
- Product tags must remain intact.
Customer question:
"Can I return this silk saree after 5 days?"
Example A: Faithful Answer
"Yes, the saree is eligible for return within 7 days, provided its tags remain intact."
The answer follows the retrieved information.
Example B: Unfaithful Answer
"Yes, you can return it within 7 days, and Varastra will refund your money within 24 hours."
The retrieved context does not mention a 24-hour refund policy.
The extra claim is unsupported by the available evidence.
This is the type of problem faithfulness evaluation helps identify.
A score of 0.71 indicates room for improvement in the evaluation being shown. It does not mean that exactly 29% of answers are wrong.
The goal: Make ShopBot answer using evidence instead of inventing product details.
4. Context Precision — Did the AI Retrieve the Right Information?
The diagram shows a context precision score of 0.82.
Context precision measures how relevant the retrieved information is to the question. Depending on the implementation, it also considers the ranking of relevant and irrelevant retrieved chunks.
Imagine a customer asks:
"Can I return this silk saree?"
ShopBot retrieves these four chunks:
- Return policy: Returns allowed within 7 days.
- Product tags: Tags must remain intact.
- Saree colour: Red and gold.
- Washing instructions: Dry clean only.
The first two chunks are directly relevant to the return question.
The colour and washing instructions may be valid product information, but they are not needed to answer this particular question.
A retriever that prioritizes the return policy and tag requirements provides more useful context.
The goal: Retrieve relevant evidence instead of overwhelming the model with unrelated product information.
A context precision score of 0.82 suggests that retrieval relevance can still be improved. It does not mean that exactly 82% of retrieved information is always correct.
5. Answer Relevance — Did ShopBot Answer the Actual Question?
The diagram marks answer relevance as pending.
Answer relevance evaluates whether the generated answer addresses the customer's question.
For example:
Customer:
"Can I return this saree after 5 days?"
Relevant answer:
"Yes, returns are allowed within 7 days if the product tags remain intact."
Irrelevant answer:
"This saree comes in red and gold and is suitable for festive occasions."
The second answer may describe the product accurately, but it does not answer the customer's question.
Answer relevance helps identify responses that are off-topic or fail to address the user's intent.
Since the score is pending, we should not assume ShopBot performs well or poorly on this metric yet.
6. Context Recall — Did the Retriever Miss Important Information?
Context recall is also marked as pending.
It evaluates whether the retrieved context contains the relevant information needed to answer the question, compared with the available reference information.
Suppose Varastra's return policy says:
- Returns are allowed within 7 days.
- Product tags must remain intact.
- Products must be unused.
A customer asks:
"Can I return this saree after 5 days if I have removed the tags?"
The retriever returns only:
"Returns are allowed within 7 days."
It misses the requirement that the tags must remain intact.
The retrieved information is relevant, but it is incomplete.
This can lead ShopBot to give an incomplete or misleading answer.
Context recall helps reveal whether important evidence is being missed during retrieval.
The goal: Retrieve enough relevant information to answer the customer's question properly.
7. Why Are There Two Baseline Scores?
The diagram shows:
- Faithfulness: 0.71
- Context Precision: 0.82
These are the starting measurements for the evaluation.
They help the development team identify where improvements may be needed.
For example:
- A lower faithfulness score may indicate that answers contain unsupported claims.
- A lower context precision score may indicate that retrieval includes too much irrelevant information.
The scores alone do not prove the exact cause of a problem. Developers should examine individual questions, retrieved chunks, and generated answers to understand why a metric is low.
Once the team improves the system, it can run the same evaluation again and compare the results.
For example:
| Metric | Baseline | After Improvement |
|---|---|---|
| Faithfulness | 0.71 | To be measured |
| Context Precision | 0.82 | To be measured |
| Answer Relevance | Pending | To be measured |
| Context Recall | Pending | To be measured |
The second column should contain actual evaluation results, not guessed or promised scores.
The goal is to measure real improvement rather than assume the system has become better.
8. How Can We Improve ShopBot's Scores?
Different metrics may require different improvements.
To Improve Faithfulness
- Strengthen the system prompt.
- Instruct the model not to invent unsupported product details.
- Ensure the retrieved context contains the necessary evidence.
- Test answers for unsupported claims.
To Improve Context Precision
- Improve product descriptions and metadata.
- Use attribute-level chunks for details such as price, returns, and care instructions.
- Tune retrieval settings and ranking.
- Filter out irrelevant chunks.
To Improve Answer Relevance
- Make sure the prompt clearly identifies the customer's question.
- Avoid unnecessary information in responses.
- Test whether answers directly address the user's intent.
To Improve Context Recall
- Check whether product information is missing from the knowledge base.
- Review chunking strategies.
- Tune retrieval settings.
- Ensure all relevant policy conditions can be retrieved together when needed.
These improvements should be tested rather than assumed to work.
9. How Does RAGAS Fit into the ShopBot Pipeline?
The ShopBot RAG pipeline works like this:
- A customer asks a question.
- The embedding model converts the question into a vector.
- The retriever searches ChromaDB.
- Relevant product information is retrieved.
- The prompt builder combines the context and question with the system instructions.
- The language model generates an answer.
- The evaluation process measures the quality of the retrieval and answer.
RAGAS helps evaluate the system using suitable metrics and evaluation data.
The evaluation results help developers understand where the system needs improvement.
A good evaluation process should use representative test questions, reliable reference information where required, and consistent evaluation settings.
10. The Most Important Lesson
An AI shopping assistant can sound confident and still provide an unsupported or irrelevant answer.
That is why building a RAG application is not enough. We must also measure its performance.
For Varastra, this means checking whether ShopBot retrieves relevant product information, stays within the available evidence, and answers customer questions directly.
You cannot reliably improve what you never measure.
RAGAS provides a way to turn AI evaluation into measurable evidence rather than relying only on how convincing an answer sounds.