English | 简体中文

If you like AGI-Eval, please star the project to get the latest updates～⭐️

This project is an open-source Large Language Model (LLM) evaluation framework from the AGI-Eval platform, aiming to provide a flexible and extensible evaluation platform. The framework adopts a plugin-based architecture design, supporting multiple evaluation scenarios, model access methods, and evaluation metrics.

The following provides the environment setup process for AGI-Eval, requirements for custom datasets, and evaluation model requirements.

We strongly recommend using conda to manage your Python environment.

# Python version: Requires Python 3.11 or higher
conda create --name agieval python=3.11 -y
conda activate agieval

• Clone source code

git clone https://github.com/AGI-Eval-Official/agi-eval.git

• Install dependencies

cd agi-eval

# Install Native backend
pip install -e .

• Optional dependencies installation

The above installs the necessary dependencies for framework operation and execution commands. The core plugin system of the framework allows users to implement custom plugins. Their necessary dependencies can be placed as optional dependencies of the framework in the plugin dependencies file requirements.json, and will be automatically installed when the corresponding plugin is loaded and used for the first time. The format requirements are as follows. The file content is in json format, the key is the module where the plugin is located, and the value is the dependency list, which can specify versions and installation sources.

{
  "agieval.plugin.metrics.drop_f1_metrics": [
      "scipy>=1.16.0 -i https://mirrors.aliyun.com/pypi/simple/"
  ]
}

AGI-Eval has adapted some public datasets that can be used directly. Use the agieval benchmarks command to view the list of supported datasets. The first evaluation of these datasets will automatically download data files to the local datasets directory. For detailed instructions, see Public Datasets.

AGI-Eval supports using local datasets for evaluation. You can download datasets to local through HuggingFace or ModelScope, then adapt them according to Dataset Adaptation before use.

Currently only supports evaluation through litellm API calls, so you need to provide a model service that supports the OpenAI API protocol. If the model to be evaluated has been deployed with a model service that supports the OpenAI API protocol, you can use it directly. Execute the following commands to configure model parameters:

# Model to be evaluated
export API_BASE_URL=http://your-api-endpoint
export MODEL_NAME=your-model-name
export API_KEY=your-api-key

# Scoring model
export SCORE_API_BASE_URL=http://your-api-endpoint
export SCORE_MODEL_NAME=your-model-name
export SCORE_API_KEY=your-api-key

If you want to use weight models for evaluation, please deploy the model service yourself.

If the deployed model service cannot support the OpenAI protocol, please adapt load_model type plugins according to the Plugin Development Guide.

Ensure all the above preparations are completed, now you can start using AGI-Eval for evaluation! This project supports two startup methods: Command Line Interface (CLI) and Python scripts.

For public datasets that have been adapted and supported by the framework or datasets that have been adapted according to framework requirements, it is recommended to use the Command Line Interface (CLI) for evaluation.

Execute the following command to ensure AGI-Eval is installed, refer to Source Installation.

agieval -v # Output AGI-Eval 1.0.0
agieval -help  # View help documentation

agieval start

• Dataset name, required
• --runner, running method, overwrites the runner field in dataset configuration
• --benchmark_config, dataset configuration file, overwrites the benchmark_config field in dataset configuration

For detailed instructions, see Parameter Description

# Specify evaluation model information through environment variables
export API_BASE_URL=http://your-api-endpoint
export MODEL_NAME=your-model-name
export API_KEY=your-api-key
agieval start test

# Or specify evaluation model information through runtime parameters
agieval start test --plugin_param base_url=http://your-api-endpoint model=your-model-name api_key=your-api-key

# If the task starts successfully, it will output process id and log file address
Task process started successfully pid: 8059, log address: /home/user/result/test/logs/info.log

The script execution entry is run.py. For new datasets that require secondary development of the framework for adaptation, it is recommended to use Python scripts for evaluation.

python3 run.py \
  --runner data_parallel \
  --benchmark_config_template  \
  --dataset_files datasets/test \
  --benchmark_config example/dataset/test/benchmark_config.json \
  --flow_config_file example/flow_config/default_flow.json \
  --work_dir result/test \
  --data_parallel 2 \
  --global_param k1=v1 k2=v2 \
  --plugin_param base_url=http://your-api-endpoint model=your-model-name api_key=your-api-key

For detailed descriptions of each parameter, see Detailed Parameter Description.

• --runner: Optional, scheduler type, optional values dummy, local, data_parallel, default is dummy.
• --benchmark_config_template: Optional, whether the dataset configuration file is used as a template, bool value, default is False.
• --dataset_files: Optional, parent directory of dataset files, required when --benchmark_config_template is True.
• --benchmark_config: Required, dataset configuration file, must be in json format, evaluation tasks run according to this configuration.
• --flow_config_file: Optional, evaluation flow configuration, must be in json format, as the fallback configuration for evaluation task configuration.
• --work_dir: Required, specify the output directory of evaluation results.
• --data_parallel: Optional, specify the concurrency degree when --runner is data_parallel, default is 1.
• --global_param: Optional, set global parameters, default is empty.
• --plugin_param: Optional, override runtime parameters of each plugin, default is empty.

Evaluation results support simple visualization display. Run the agieval appstart command to start the visualization service.

agieval appstart --result_dir=result/test

# Output example
Visit the following URL to view the evaluation report: http://localhost:38410/agieval/visualization/reportor.html?path=result/test

Evaluation Metrics	Evaluation Details
Evaluation Parameters	Evaluation Flow

@misc{2025agi-eval,
    title={AGI-Eval: An Evaluation Platform for AGI},
    author={AGI-Eval Team},
    howpublished = {\url{https://github.com/AGI-Eval-Official/agi-eval}},
    year={2025}
}