EncouRAGe

the all-in one solution for evaluate RAG methods.

This repository provides a flexible library for running Retrieval-Augmented Generation (RAG) methods and evaluate them. It is designed to be modular and extensible, allowing users to easily integrate their own data and test them on RAG methods and calculate metrics.

The following libraries are used to provide the core functionality:

For Inference Runners:

• 🏃 vllm
  • A fast and flexible framework for LLM inference.

For Templates:

• ⚙️ jinja2
  • Offers a template engine for dynamic prompt generation.

For Evaluation Metrics:

• 📊 evaluate
  • A library for easily accessing and computing a wide range of evaluation metrics.

For Vector Databases:

• 🔄 chroma
  • Strong in-memory vector database for efficient data retrieval.
• 🧭 qdrant
  • Supports robust vector search for efficient data retrieval.

pip install encourage-rag

To initialize the environment using uv, run the following command:

uv sync

For understanding how to use the inference runners, refer to the following tutorials:

• BatchInferenceRunner

Encourage provides several RAG (Retrieval-Augmented Generation) methods to enhance your LLM responses with relevant context:

• Overview of RAG Methods

Encourage offers a comprehensive set of metrics for evaluating LLM and RAG performance:

• Metrics Overview - Table of all available metrics
• Metrics Explanation - Detailed explanations and formulas
• Metrics Tutorial - Step-by-step guide to using metrics

To use a custom template for the inference, follow the steps below:

• Create a custom template

For tracking the model performance, use the following commands:

• Track the model

We welcome and value any contributions and collaborations. Please check out Contributing to encourage for how to get involved.

This project is developed as cooperation project by the HCDS at the University of Hamburg and dida GmbH.

The research and development project is funded by the Federal Ministry of Research, Technology and Space (BMFTR) and supervised by the German Aerospace Center (DLR).

Funding code of the University of Hamburg: 16IS24044B