Subtitle: Learn how Varastra evaluates its AI shopping assistant using four RAGAS dimensions to detect unsupported answers, irrelevant search results, unanswered questions, and missing information.
1. Why Does RAG Need Evaluation?
Imagine a customer asks Varastra, an Indian ethnic-wear store:
“Is this cotton kurta available in size L, and how much does it cost?”
The AI retrieves product information and generates an answer. But what if it invents a price, retrieves irrelevant products, ignores the size question, or misses the document containing the price?
A fluent answer isn't necessarily a reliable answer.
RAGAS helps evaluate different parts of a Retrieval-Augmented Generation (RAG) system.
This diagram focuses on four dimensions.
2. The Four RAGAS Dimensions
| Dimension | What it checks |
|---|---|
| Faithfulness (F) | Is the answer grounded in retrieved context? |
| Context Precision (CP) | Are the retrieved chunks relevant to the query? |
| Answer Relevance (AR) | Does the answer address the question? |
| Context Recall (CR) | Did retrieval miss information the answer needed? |
Each metric detects a different type of failure.
3. Faithfulness: Is the Answer Grounded?
Question: Is the answer supported by the retrieved context?
Suppose the retrieved product data says:
- Product: Mint cotton kurta
- Sizes: S, M, L
- Price: ₹1,799
A faithful answer says: “The kurta is listed at ₹1,799 and is available in size L.”
If the AI adds “Free delivery is guaranteed tomorrow” without supporting evidence, that claim is ungrounded.
Why it matters: Faithfulness helps detect unsupported claims.
A faithful answer isn't automatically factually correct. If the source data is outdated, the answer may faithfully repeat outdated information.
4. Context Precision: Did Retrieval Find Relevant Information?
Question: Are the retrieved chunks relevant to the query?
For “Find a mint cotton kurta under ₹2,000,” useful results might include matching products and their prices. An unrelated silk saree or beige linen outfit adds noise.
Context Precision evaluates how effectively relevant information appears in the retrieved ranking, giving importance to relevant results near the top.
Why it matters: Irrelevant chunks waste context space and can distract the LLM.
5. Answer Relevance: Did the AI Answer the Question?
Question: Does the response address what the customer asked?
Customer: “Is this kurta available in size L?”
Relevant answer: “The product data lists size L as available.”
Irrelevant answer: “This kurta is made from comfortable cotton.”
The second answer may be true, but it doesn't answer the question.
Why it matters: A response can be fluent and factual while still failing to address the customer's actual need.
6. Context Recall: Did Retrieval Miss Important Information?
Question: Did retrieval find the information needed to answer the question?
Suppose the reference data contains the price and size availability, but the retriever returns only a description of the kurta's fabric.
The LLM may not have the evidence needed to answer the complete question.
Context Recall evaluates whether retrieval found the relevant information represented in the reference answer.
Why it matters: Sometimes the problem isn't answer generation—the retriever never supplied the required evidence.
7. Why Do We Need All Four?
| Failure | Metric to investigate |
|---|---|
| AI invents a delivery promise | Faithfulness |
| Search results contain unrelated products | Context Precision |
| Answer ignores the size question | Answer Relevance |
| Retrieved context omits the price | Context Recall |
A system can score well on one dimension and poorly on another. For example, it may retrieve highly relevant chunks but still miss one crucial piece of information.
No single metric tells the whole story.
8. Understanding the Baseline Scores
The diagram shows these scores for the first labelled evaluation run:
| Metric | Score |
|---|---|
| Faithfulness (F) | 0.83 |
| Answer Relevance (AR) | 0.81 |
| Context Precision (CP) | 0.88 |
| Context Recall (CR) | 0.79 |
- Context Precision: 0.88 — The highest reported score.
- Context Recall: 0.79 — The lowest reported score and a useful area to investigate.
- Faithfulness: 0.83 — The reported grounding score.
- Answer Relevance: 0.81 — The reported relevance score.
These values are the baseline scores shown in the diagram. They don't, by themselves, reveal why a metric scored as it did.
Important: These scores aren't automatically percentages of correct answers. Interpret them according to the metric implementation and evaluation dataset.
9. How Varastra Can Use the Results
- Collect representative customer questions.
- Run them through the RAG pipeline.
- Save the retrieved chunks and generated answers.
- Evaluate the four dimensions.
- Inspect low-scoring examples.
- Improve retrieval, ranking, or answer generation.
- Run the evaluation again and compare results.
The metrics help identify where to investigate; they don't automatically prescribe the fix. Human review and checks against current product data remain important.
Technical note: The diagram marks Answer Relevance as requiring ground-truth labels. In standard RAGAS implementations, classic Answer Relevancy generally does not require a reference answer, while Context Recall does. Requirements depend on the specific metric and evaluation setup.
10. Key Takeaways
- Faithfulness: Is the answer supported by retrieved context?
- Context Precision: Are the retrieved results relevant and well ranked?
- Answer Relevance: Does the answer address the question?
- Context Recall: Did retrieval find the information needed?
- The diagram's baseline is F 0.83, AR 0.81, CP 0.88, and CR 0.79.
- The four metrics complement one another and help diagnose different RAG failures.
- Evaluation scores guide improvement but do not guarantee correctness.
The big lesson: Reliable RAG requires the right evidence, enough evidence, and an answer that stays grounded while addressing the customer's actual question.