Reinforcement learning streamlined.
Easier and faster reinforcement learning with RLOps. Visit our website. View documentation.
Join the Discord Server for questions, help and collaboration.

AgileRL is a Deep Reinforcement Learning library focused on improving development by introducing RLOps - MLOps for reinforcement learning.

This library is initially focused on reducing the time taken for training models and hyperparameter optimization (HPO) by pioneering evolutionary HPO techniques for reinforcement learning.
Evolutionary HPO has been shown to drastically reduce overall training times by automatically converging on optimal hyperparameters, without requiring numerous training runs.
We are constantly adding more algorithms and features. AgileRL already includes state-of-the-art evolvable LLM fine-tuning, on-policy, off-policy, offline, multi-agent and contextual multi-armed bandit reinforcement learning algorithms with distributed training.

AgileRL offers 10x faster hyperparameter optimization than SOTA.

• Benchmarks
• Get Started
• Training
• Arena
• Tutorials
• Algorithms
• Citing AgileRL

AgileRL's multi-turn LLM training enables state-of-the-art performance on long-horizon tasks with small models. In the following example, AgileRL's CISPO was benchmarked against ART and TRL on the GEM Sudoku Hard task. This is a difficult multi-turn problem, which requires a context length of 32k tokens and up to 50 turns per rollout. The sync AgileRL run is a single agent using the AgileRL framework. The async and HPO runs were performed on Arena, AgileRL's RLOps platform. All runs used the same starting hyperparameters. AgileRL runs were run on A100 40GB nodes, whereas ART and TRL required A100 80GB nodes due to a lack of optimizations. AgileRL runs significantly outperformed those using the ART and TRL frameworks.

Reinforcement learning algorithms and libraries are usually benchmarked once the optimal hyperparameters for training are known, but it often takes hundreds or thousands of experiments to discover these. This is unrealistic and does not reflect the true, total time taken for training. What if we could remove the need to conduct all these prior experiments?

In the charts below, a single AgileRL run, which automatically tunes hyperparameters, is benchmarked against Optuna's multiple training runs traditionally required for hyperparameter optimization, demonstrating the real time savings possible. Global steps is the sum of every step taken by any agent in the environment, including across an entire population.

AgileRL offers an order of magnitude speed up in hyperparameter optimization vs popular reinforcement learning training frameworks combined with Optuna. Remove the need for multiple training runs and save yourself hours.

AgileRL also supports multi-agent reinforcement learning using the Petting Zoo-style (parallel API). The charts below highlight the performance of our MADDPG and MATD3 algorithms with evolutionary hyper-parameter optimisation (HPO), benchmarked against epymarl's MADDPG algorithm with grid-search HPO for the simple speaker listener and simple spread environments.

To see the full AgileRL documentation, including tutorials, visit our documentation site. To ask questions and get help, collaborate, or discuss anything related to reinforcement learning, join the AgileRL Discord Server.

Install as a package with pip:

pip install agilerl

Or install in development mode:

git clone https://github.com/AgileRL/AgileRL.git && cd AgileRL
pip install -e .

AgileRL ships optional dependency groups that you can install as needed:

In development mode, quote the extras:

pip install -e ".[arena]"

To install the nightly version of AgileRL with the latest features, use:

pip install git+https://github.com/AgileRL/AgileRL.git@nightly

AgileRL provides the tools to train RL algorithms in a variety of ways, focusing on flexibility and modularity as a stepping stone for efficiently training arbitrarily large populations of agents in a distributed manner on Arena.

The simplest way to train an RL agent with AgileRL is through the LocalTrainer. Here is an example of training a DQN agent on the LunarLander-v3 environment:

from agilerl import LocalTrainer

trainer = LocalTrainer(algorithm="DQN", environment="LunarLander-v3")
population, fitnesses = trainer.train()

With no other arguments provided, LocalTrainer defaults to 1,000,000 steps with a single agent and the algorithm's default hyperparameters — no evolutionary HPO is applied.

To unlock AgileRL's evolutionary hyperparameter optimization, train a population of agents whose hyperparameters will evolve and mutate towards their optimal values:

from agilerl import LocalTrainer
from agilerl.models import TrainingSpec

trainer = LocalTrainer(
    algorithm="DQN",
    environment="LunarLander-v3",
    training=TrainingSpec(pop_size=4), # Train four agents simultaneously
    hpo=True, # Enable evolutionary HPO using default settings
)
population, fitnesses = trainer.train()

This trains a population of four DQN agents that share experiences but learn individually. Every 10,000 steps (default value for evo_steps in TrainingSpec), tournament selection identifies the best performers and mutations are applied to explore the hyperparameter space. See Evolutionary Hyperparameter Optimization for details on how evolutionary HPO works in AgileRL.

Or via a YAML manifest:

---
algorithm:
    name: DQN
    batch_size: 128
    lr: 6.3e-4
    learn_step: 4
    gamma: 0.99
    tau: 0.001
    double: false
    cudagraphs: false

environment:
    name: LunarLander-v3
    num_envs: 16

mutation:
    probabilities:
        no_mut: 0.4
        arch_mut: 0.2
        new_layer: 0.2
        params_mut: 0.2
        act_mut: 0.2
        rl_hp_mut: 0.2
    rl_hp_selection:
        lr:
            min: 0.0000625
            max: 0.01
        batch_size:
            min: 8
            max: 512
        learn_step:
            min: 1
            max: 10
    mutation_sd: 0.1
    rand_seed: 42

network:
    latent_dim: 128
    arch: mlp
    encoder_config:
        hidden_size:
            - 128
    head_config:
        hidden_size:
            - 128

replay_buffer:
    max_size: 100_000

tournament_selection:
    tournament_size: 2
    elitism: true

training:
    max_steps: 1_000_000
    target_score: 200.0
    pop_size: 4
    evo_steps: 10_000
    eval_steps:
    eval_loop: 1
    learning_delay: 0
    eps_start: 1.0
    eps_end: 0.1
    eps_decay: 0.99

Python

from agilerl import LocalTrainer

trainer = LocalTrainer.from_manifest("configs/training/dqn/dqn.yaml")
population, fitnesses = trainer.train()

CLI

python -m agilerl.train configs/training/dqn/dqn.yaml

Every aspect of the training pipeline is customisable — from modifying hyperparameters and mutation strategies in our off-the-shelf tools, to implementing your own evolvable algorithms, network architectures, and training loops.

For full control over training, you can build each component individually:

import torch

from agilerl.algorithms import DQN
from agilerl.utils.utils import make_vect_envs
from agilerl.components.replay_buffer import ReplayBuffer
from agilerl.hpo.tournament import TournamentSelection
from agilerl.hpo.mutation import Mutations
from agilerl.training.train_off_policy import train_off_policy

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Initialize environment
env = make_vect_envs(env_name="LunarLander-v3", num_envs=16)

# Network configuration
net_config = {
    "latent_dim": 64,
    "encoder_config": {"hidden_size": [64]},
    "head_config": {"hidden_size": [64]}
}

# Algorithm hyperparameters
init_hp = {
    "double": True,
    "batch_size": 256,
    "lr": 1e-3,
    "gamma": 0.99,
    "learn_step": 1,
    "tau": 1e-3
}

# Create a population of DQN agents
population_size=6
agent_pop = DQN.population(
    size=population_size,
    observation_space=env.single_observation_space,
    action_space=env.single_action_space,
    net_config=net_config,
    device=device,
    **init_hp
)

# Replay buffer
memory = ReplayBuffer(max_size=10_000, device=device)

# Evolutionary HPO
tournament = TournamentSelection(
    tournament_size=2,
    elitism=True,
    population_size=population_size
)
mutations = Mutations(
    no_mutation=0.4,
    architecture=0.2,
    new_layer_prob=0.2,
    parameters=0.2,
    activation=0.0,
    rl_hp=0.2,
    mutation_sd=0.1,
    rand_seed=42,
    device=device,
)

trained_pop, pop_fitnesses = train_off_policy(
    env=env,
    env_name="LunarLander-v3",
    algo="DQN",
    pop=agent_pop,
    memory=memory,
    max_steps=1_000_000,
    evo_steps=10_000,
    target=200.0,
    tournament=tournament,
    mutation=mutations,
)

This approach gives you the flexibility to swap in your own Gymnasium or PettingZoo environments, custom evolvable networks, or entirely custom training loops while still leveraging AgileRL's evolutionary HPO.

Arena is the RLOps platform from AgileRL. We provide tools to create and validate custom reinforcement learning environments on the platform and train RL agents on managed cloud infrastructure specifically tailored to RL workloads.

AgileRL ships a Python SDK and a CLI for interacting with the platform through the agilerl-arena package. It is a separate PyPI distribution that contributes the agilerl.arena namespace. Install it directly, or via the AgileRL extra:

pip install agilerl-arena
# or
pip install "agilerl[arena]"

Use the ArenaClient to interact with Arena programmatically from scripts or notebooks:

from agilerl.arena import ArenaClient

client = ArenaClient()
client.login() # OAuth2 device-flow authentication

# Upload and validate a custom environment
client.validate_environment(name="my-custom-env", source="path/to/my_env.py")

# Train on validated custom environment
client.submit_experiment(
    manifest="path/to/manifest.yaml",
    project="my-project",
)

The same operations are available from the command line:

# Authenticate with Arena
arena login

# Upload and validate
arena env validate my-custom-env --source path/to/my_env.py

# Train on validated custom environment
arena experiments submit path/to/manifest.yaml --project my-project

For the full CLI and Python SDK reference, including authentication, environment validation, experiments, and deployment, see the Arena Client documentation.

We are constantly updating our tutorials to showcase the latest features of AgileRL and how users can leverage our evolutionary HPO to achieve 10x faster hyperparameter optimization. Please see the available tutorials below.

If you use AgileRL in your work, please cite the repository:

@software{Ustaran-Anderegg_AgileRL,
author = {Ustaran-Anderegg, Nicholas and Pratt, Michael and Sabal-Bermudez, Jaime},
license = {Apache-2.0},
title = {{AgileRL}},
url = {https://github.com/AgileRL/AgileRL}
}