💡 The Align AI Research Review is a weekly dive into the most relevant research papers, AI industry news and updates
Introduced an analysis framework to systematically assess RAG systems by considering retrieval accuracy, generation quality, and other additional factors.
Retrieval Augmented Generation (RAG) is an important breakthrough in NLP to tackle two main concerns: optimal retrieval methods and generating better responses.
In the paper (Evaluation of Retrieval-Augmented Generation: A Survey), authors conducted a comprehensive analysis and comparison of several measurable metrics linked with the Retrieval and Generation component of the existing RAG evaluation methods.
It provides relevance, accuracy, and faithfulness. The analysis encompasses both potential output and ground truth pairings.
Figure 1 depicts the overall structure of the RAG systems as mentioned.
RGAR: Analysis Framework for Evaluation
The research presented a better understanding of RAG benchmarks by introducing an analysis framework known as RGAR (see figure 2). It considers the Target, Dataset, and Metric comprehensively.
💡 RGAR (Retrieval, Generation, and Additional Requirement)
Let’s discuss all three modules in this research:
Target Module: aims to figure out the direction of the evaluation.
Dataset Module: assists the comparison of diverse data models in RAG benchmarks.
Metrics Module: describes the metrics linked with the specific targets and datasets incorporated in the evaluation process.
The workflow of RGAR framework as outlined in figure below, categorizes the methodologies into retrieval, generation, and ground truth phase.
Let’s discuss the evaluation framework through three key questions:
What should the Evaluation Target be?
💡 Evaluable Outputs (EOs)
💡 Ground Truths (GTs)
Once identified, these targets can be used to analyze all aspects of existing RAG benchmarks. This is achieved by defining them based on a specific pair of EOs or an EO with a GT (see figure 2).
Retrieval: The retrieval process EOs are the relevant papers used in evaluating the retrieval component, dependent on the query.
For the retrieval component, two pairwise relationships can be established:
Relevant Documents ↔ Query and Relevant Documents → Document Candidates.
Table 1 Evaluation Criteria for Retrieval Components
Generation: The EOs consist of the specified structured content and generated text. These EOs must then be compared to the GTs and provided labels (see table 2).
Table 2 Evaluation Criteria for Generation Components
The emergence of the targets for the Retrieval and Generation components is observed. Relevant work on enhancing and assessing RAG and RAG benchmarks is detailed in Table 3. Each evaluation criterion in the table is denoted by a distinct color.
How should the Evaluation Dataset be assessed?
Powerful LLMs have significantly transformed dataset construction. By designing queries and ground truths for specific evaluation objectives, these frameworks enable researchers to easily generate datasets in the desired format.
Benchmarks like RGB, MultiHop-RAG, CRUD-RAG, and CDQA have advanced this approach by building their own datasets using online news articles.
This tests RAG systems' ability to handle real-world information beyond the training data of language model frameworks.
Table 4 presents various dataset construction strategies utilized by different benchmarks. These strategies range from using pre-existing resources to generating entirely new data specifically designed for evaluation purposes.
How should the Evaluation Metric be quantified?
To effectively assess RAG systems, one must possess a comprehensive understanding of the metrics that accurately quantify the evaluation objectives.
However, developing evaluative criteria that align with human preferences and consider practical aspects is challenging.
Retrieval Metrics
The retrieval evaluation focuses on metrics that possess the capability to precisely measure the relevance, precision, diversity, and reliability of the information obtained in reaction to queries.
The metrics should not solely assess the accuracy with which the system retrieves relevant information, but also its ability to handle the challenges of navigating the ever-changing, extensive, and at times misleading world of accessible data.
These metrics incorporate not only the accuracy and recall of retrieval systems but also the relevance and variety of documents retrieved. This approach aligns with the dynamic and complex nature of information requirements in RAG systems.
Generation Metrics
Within the domain of generation, evaluation extends beyond the simple precision of generated responses. It encompasses the overall quality of the text in terms of coherence, relevance, fluency, and alignment with human perception.
This necessitates metrics capable of evaluating the complex aspects of language production, such as factual accuracy, readability of the generated content, and user satisfaction.
Whereas, human evaluation remains a crucial standard for comparing the performance of generation models with each other or with the ground truth.
Additional Requirements
RAG systems are designed to be practical in real-world scenarios and align with human preferences by incorporating additional requirements such as latency, diversity, noise robustness, negative rejection, and counterfactual robustness.
Final Thoughts
From Align AI's perspective, evaluating RAG systems involves a range of complex challenges, as these systems must simultaneously create coherent responses that meet user expectations and retrieve reliable and relevant data.
This research analyzes several important factors that need attention, covering the extensive range of evaluations necessary to advance RAG technologies.
For Retrieval, key evaluation aspects include relevance, accuracy, diversity, and robustness. Metrics such as Mean Average Precision (MAP), Mean Reciprocal Rank (MRR), and misleading rate are examined.
For Generation, key evaluation aspects are relevance, faithfulness, and correctness. Metrics such as ROUGE, BLEU, F1, and assessments by LLM judges are discussed.
Additional evaluated requirements include latency, diversity, noise robustness, negative rejection, and counterfactual robustness.
Learn More!
To explore deeper into the materials of this blog post, we share a few resources below:
