RAG AutoML tool for automatically finds an optimal RAG pipeline for your data.

Explore our 📖 Document!!

Plus, join our 📞 Discord Community.

• Step 1: Basic of AutoRAG | Optimizing your RAG pipeline
• Step 2: Create evaluation dataset

Muted by default, enable sound for voice-over

You can see on YouTube

🛣️ Support plans & Roadmap

💻 Hardware Specs

⭐ Running AutoRAG

🍯 Tips/Tricks

☎️ TroubleShooting

There are many RAG pipelines and modules out there, but you don’t know what pipeline is great for “your own data” and "your own use-case." Making and evaluating all RAG modules is very time-consuming and hard to do. But without it, you will never know which RAG pipeline is the best for your own use-case.

AutoRAG is a tool for finding optimal RAG pipeline for “your data.” You can evaluate various RAG modules automatically with your own evaluation data, and find the best RAG pipeline for your own use-case.

AutoRAG supports a simple way to evaluate many RAG module combinations. Try now and find the best RAG pipeline for your own use-case.

You can check our all supporting Nodes & modules at here

You can check our all supporting Evaluation Metrics at here

• Retrieval Metrics
• Retrieval Token Metrics
• Generation Metrics

We recommend using Python version 3.9 or higher for AutoRAG.

pip install AutoRAG
git clone https://github.com/vllm-project/vllm.git
pip install -r requirements-build.txt --extra-index-url https://download.pytorch.org/whl/cpu
PIP_PRE=1 PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu https://storage.openvinotoolkit.org/simple/wheels/nightly/" VLLM_TARGET_DEVICE=openvino python -m pip install -v .

• I use

python ./sample_dataset/triviaqa/load_triviaqa_dataset.py

For evaluation, you need to prepare just three files.

1. QA dataset file (qa.parquet)
2. Corpus dataset file (corpus.parquet)
3. Config yaml file (config.yaml)

There is a template for your evaluation data for using AutoRAG.

• Check out how to make evaluation data at here.
• Check out the evaluation data rule at here.
• Plus, you can get example datasets for testing AutoRAG at here.

• I use

autorag evaluate --config ./sample_config/simple_local.yaml  --qa_data_path ./data/qa.parquet --corpus_data_path ./data/corpus.parquet

You can get various config yaml files at here. We highly recommend using pre-made config yaml files for starter.

If you want to make your own config yaml files, check out the Config yaml file section.

You can evaluate your RAG pipeline with just a few lines of code.

from autorag.evaluator import Evaluator

evaluator = Evaluator(qa_data_path='your/path/to/qa.parquet', corpus_data_path='your/path/to/corpus.parquet')
evaluator.start_trial('your/path/to/config.yaml')

or you can use command line interface

autorag evaluate --config your/path/to/default_config.yaml --qa_data_path your/path/to/qa.parquet --corpus_data_path your/path/to/corpus.parquet

Once it is done, you can see several files and folders created at your current directory. At the trial folder named to numbers (like 0), you can check summary.csv file that summarizes the evaluation results and the best RAG pipeline for your data.

For more details, you can check out how the folder structure looks like at here.

You can use a found optimal RAG pipeline right away. It needs just a few lines of code, and you are ready to use!

First, you need to build pipeline yaml file from your evaluated trial folder. You can find the trial folder in your current directory. Just looking folder like '0' or other numbers.

from autorag.deploy import Runner

runner = Runner.from_trial_folder('your/path/to/trial_folder')
runner.run('your question')

Or, you can run this pipeline as api server. You can use python code or CLI command. Check out API endpoint at here.

from autorag.deploy import Runner

runner = Runner.from_trial_folder('your/path/to/trial_folder')
runner.run_api_server()

You can run api server with CLI command.

autorag run_api --config_path your/path/to/pipeline.yaml --host 0.0.0.0 --port 8000

You can run dashboard to easily see the result.

autorag dashboard --trial_dir /your/path/to/trial_dir

• sample dashboard:

You can use your RAG pipeline from extracted pipeline yaml file. This extracted pipeline is great for sharing your RAG pipeline to others.

You must run this at project folder, which contains datas in data folder, and ingested corpus for retrieval at resources folder.

from autorag.deploy import extract_best_config

pipeline_dict = extract_best_config(trial_path='your/path/to/trial_folder', output_path='your/path/to/pipeline.yaml')

You can use a found optimal RAG pipeline right away with extracted yaml file.

from autorag.deploy import Runner

runner = Runner.from_yaml('your/path/to/pipeline.yaml')
runner.run('your question')

You can run this pipeline as an API server.

Check out API endpoint at here.

from autorag.deploy import Runner

runner = Runner.from_yaml('your/path/to/pipeline.yaml')
runner.run_api_server()

autorag run_api --config_path your/path/to/pipeline.yaml --host 0.0.0.0 --port 8000

you can run this pipeline as a web interface.

Check out web interface at here.

autorag run_web --trial_path your/path/to/trial_path

• sample web interface:

Thanks go to these wonderful people:

We are developing AutoRAG as open-source.

So this project welcomes contributions and suggestions. Feel free to contribute to this project.

Plus, check out our detailed documentation at here.