Back to Glossary
RAG
RAG Evaluation
Definition
RAG evaluation measures the quality of RAG systems across multiple dimensions: retrieval accuracy, answer faithfulness to sources, relevance to the query, and overall response quality.
Why It Matters
Without proper evaluation, you can’t improve your RAG system systematically. Different components can fail in different ways: retrieval might miss relevant documents, the generator might hallucinate, or answers might be correct but not address the query. Comprehensive evaluation identifies where to focus optimization efforts.
Key Metrics
Retrieval Metrics:
- Precision@K: Proportion of retrieved docs that are relevant
- Recall@K: Proportion of relevant docs that are retrieved
- MRR: Mean Reciprocal Rank of first relevant result
Generation Metrics:
- Faithfulness: Is the answer supported by retrieved context?
- Answer Relevancy: Does the answer address the question?
- Contextual Relevancy: Was the right context retrieved?
Tools
Frameworks like RAGAS, DeepEval, and TruLens provide automated evaluation using LLM-as-judge approaches. These can scale evaluation beyond manual review while maintaining reasonable accuracy.