👋 Hi, everyone! AReno is a fast, effortless, and self-contained toolkit that scales RL post-training up locally, initiated by the inclusionAI ASystem team and maintained by the AReno community.

AReno is a local LLM post-training toolkit for RL, SFT/DPO-style training, serving, and agentic RL. It was originally developed by engineers from the ASystem Team at Ant Group.

Built on a self-contained, full-stack design, AReno is optimized to extract maximum performance from a single node, making it well-suited for fast, local post-training with no external training or inference backend in the loop.

AReno's mission is to make LLM RL accessible for a broad community of researchers and developers — so you can go from a base checkpoint to a trained, served model on a single node, without standing up a cluster or wiring together a training framework, an inference server, and a kernel library.

Small but complete, like its name — nano in footprint, full-stack in capability. We hope AReno makes scaling up your ideas locally both fast and delightful. Enjoy!

• ✨ Plug-and-play: various post-training methods are easily accessible via the --algo flag or the same Trainer class from Python, no cluster or launcher to set up.
• 🪶 Lightweight: single self-contained package, no external training/inference backend, just PyTorch, FlashAttention, and a handful of other libraries.
• 🧰 Agentic RL ready: run an agent function against AReno's local OpenAI-compatible proxy, return explicit trajectories, and train from tokens, logprobs, rewards, and loss masks derived by the trainer.
• 🧩 Extensible: easily register new algorithms, model adapters, reward functions, and hardware backends without changing the core.

AReno currently requires Linux (x86_64 or aarch64) with an NVIDIA GPU and CUDA-enabled PyTorch 2.6 or newer; Windows users can use WSL2.

git clone https://github.com/inclusionAI/AReno.git
cd AReno
bash scripts/install.sh

Use our official AReno image inclusionai/areno container. Run:

# Please make sure the host has an NVIDIA driver and NVIDIA Container Toolkit.
docker run --gpus all --rm -it \
  ghcr.io/inclusionai/areno:v0.0.6 \
  areno check

With the SDK, RL loop is a short cycle of Trainer calls. Each step below maps a concept to the SDK call that performs it:

flowchart LR
    A["Trainer<br/>init()"] -->
    B["rollout_batch<br/>on-policy samples"] -->
    C["reward fn<br/>score"] -->
    D["train<br/>optimizer step"] -->|repeat| B

1. Create the trainer — construct a Trainer on the AReno backend and init() it to load the tokenizer and start workers.
2. Roll out — inside rollout_session(...), rollout_batch(...) generates on-policy completions for each prompt.
3. Score — reward each completion and turn rewards into advantages (your reward function, not AReno's).
4. Train — pack the rollout into TrainSequence objects and call train(batch, loss_fn) to run one optimizer step.
5. Repeat — new weights produce new rollouts; loop until done, then close().

import asyncio
from functools import partial

from datasets import load_dataset

from areno.api import (
    Areno,
    ArenoConfig,
    SamplingParams,
    Trainer,
    TrainSequence,
    gspo_loss_fn,
)
from examples.math.math_verify_reward import reward_fn


def to_advantages(rewards):
    mean = sum(rewards) / len(rewards)
    var = sum((r - mean) ** 2 for r in rewards) / max(len(rewards), 1)
    std = max(var**0.5, 1e-6)
    return [(r - mean) / std for r in rewards]


async def main():
    # 1. Create the trainer
    trainer = Trainer(
        world_size=1,
        model_path="Qwen/Qwen3-0.6B",
        backend_type=Areno,
        custom_config=ArenoConfig(tp_size=1),
    )
    trainer.init()

    try:
        # 2. Roll out on-policy completions for one GSM8K prompt
        row = load_dataset("gsm8k", "main", split="train[0:1]")[0]
        prompt = (
            "Solve the problem and put the final answer in \\boxed{}.\n\n"
            f"Problem: {row['question']}\nSolution:"
        )
        prompt_tokens = trainer.get_tokenizer().encode(prompt)
        sampling = SamplingParams(max_new_tokens=512, temperature=1.0)

        async with trainer.rollout_session(sampling_params=sampling, proxy=False):
            rollout = trainer.rollout_batch(
                [prompt],
                n_samples=8,
                sampling_params=sampling,
            )[0]

        # 3. Score with the same reward function the CLI uses, then form advantages
        completions = [trainer.get_tokenizer().decode(seq.resp_tokens) for seq in rollout.sequences]
        rewards = reward_fn(row, completions)
        advantages = to_advantages(rewards)

        batch = []
        for seq, reward, advantage in zip(rollout.sequences, rewards, advantages, strict=True):
            response_len = len(seq.resp_tokens)
            batch.append(
                TrainSequence(
                    prompt_mask=[True] * len(prompt_tokens) + [False] * response_len,
                    tokens=prompt_tokens + seq.resp_tokens,
                    logprobs=[0.0] * len(prompt_tokens) + seq.resp_logprobs,
                    advantages=[0.0] * len(prompt_tokens) + [advantage] * response_len,
                    reward=reward,
                    eos_token_id=trainer.get_tokenizer().eos_token_id,
                )
            )

        # 4. Train one step
        stats = trainer.train(batch, partial(gspo_loss_fn, clip_eps=3.0e-4), mini_bs=4)

        # 5. Repeat the loop over more prompts
    finally:
        trainer.close()


asyncio.run(main())

See the documentation for the full Trainer API.

You can use the AReno Command Line Interface (CLI) to quickly get started with post-training without writing any Python.

areno agent is a local operations assistant for AReno train and serve tasks. It uses an OpenAI-compatible model to inspect the current checkout, read command help, run diagnostics, and produce or execute AReno commands for the current machine. Configure the agent once with your endpoint, model, and API key:

areno agent --set \
  --base-url http://127.0.0.1:8000/v1 \
  --model deepseek-v4-flash \
  --api-key "$OPENAI_API_KEY"

Then describe the job in natural language:

areno agent "Give me a complete command to run the math demo with n-samples=8, fitting the current GPU and using as much GPU memory as practical."

From a source checkout, you can run the same agent without installing AReno:

./agent.sh "Give me a complete command to run the math demo with n-samples=8, fitting the current GPU and using as much GPU memory as practical."

The agent can ask follow-up questions through the terminal when a required parameter is missing, and it streams command output while it works.

Check whether the current machine is ready to run AReno:

areno check

areno check prints OK, WARN, and FAIL statuses with concrete next steps for common setup issues such as missing CUDA, CPU-only PyTorch, missing CUDA_HOME, unavailable nvcc, missing optional runtime dependencies, or a missing areno_accel extension.

For issue reports, collect a descriptive environment report:

areno env --json

The report includes AReno, Python, platform, PyTorch/CUDA, GPU, nvcc, dependency import status, and relevant environment variables.

Use this command when you only want to check that a machine can run one small official training task end to end:

areno train \
  --ckpt Qwen/Qwen3-0.6B \
  --dataset-path gsm8k:main \
  --dataset-loader-fn examples/math/dataset_loader.py \
  --reward-fn-path examples/math/math_verify_reward.py \
  --algo gspo \
  --tp-size 1 \
  --world-size 1 \
  --batch-size 1

This is a smoke/sanity task for the CLI, dataset loader, reward function, rollout, and training-step wiring. It is not a quality benchmark. It requires a CUDA-capable NVIDIA GPU; CPU-only machines can install the package for docs and metadata checks, but cannot run the AReno training engine. A successful run should reach rollout logs and a train_stats=... line without raising an exception.

Run GSPO on a GSM8K-style dataset with a reward function:

areno train \
  --ckpt Qwen/Qwen3-0.6B \
  --dataset-path gsm8k:main \
  --dataset-loader-fn examples/math/dataset_loader.py \
  --reward-fn-path examples/math/math_verify_reward.py \
  --algo gspo \
  --tp-size 4

--ckpt and --dataset-path accept either local paths or remote repo IDs. By default, remote refs use ModelScope. Use --model-hub hf to pull non-local model and dataset refs from Hugging Face. Switch algorithms by changing --algo (e.g. --algo grpo, --algo sft).

For models whose tokenizer chat template supports a thinking-mode switch, add --disable-thinking to pass enable_thinking=False during training prompt rendering. Tokenizers that do not support this argument automatically use their normal chat-template path.

For rollout-based algorithms, add --tune-params when you want AReno to probe rollout and train memory before the real run and fill conservative values for --max-running-prompts, --batch-size, and --mini-bs:

areno train \
  --ckpt Qwen/Qwen3-0.6B \
  --dataset-path gsm8k:main \
  --dataset-loader-fn examples/math/dataset_loader.py \
  --reward-fn-path examples/math/math_verify_reward.py \
  --algo gspo \
  --tp-size 1 \
  --world-size 1 \
  --n-samples 8 \
  --tune-params \
  --mem-frac 0.9 \
  --tune-max-samples 256

The tuner uses dummy-loaded model weights and synthetic token rows, respects the configured sequence limits and tensor-parallel size, and enables --drop-rollout-state for the tuned run. See docs/cli/training.rst for the full tuning rules.

For Agentic RL, add --agent-fn to supply an agent function. The agent calls the local OpenAI-compatible endpoint, including tools and tool_choice when needed, and returns explicit AgentTrajectoryTurn objects. AReno converts those turns into trainable assistant outputs and masks tool results by default:

python examples/agentic/tictactoe/dataset_generator.py \
  --output /tmp/areno-tictactoe.jsonl \
  --count 2048 \
  --seed 2026

areno train \
  --ckpt Qwen/Qwen3-0.6B \
  --dataset-path /tmp/areno-tictactoe.jsonl \
  --dataset-loader-fn examples/agentic/tictactoe/dataset_loader.py \
  --reward-fn-path examples/agentic/tictactoe/reward.py \
  --agent-fn examples/agentic/tictactoe/run_agent.py \
  --algo gspo \
  --tp-size 1 \
  --world-size 1

For a more realistic multi-turn software-engineering loop, see examples/agentic/coding. It uses a 100-record SWE-bench-style local dataset that ranges from easy to hard with most records marked hard, constrained Codex-style tools (inspect_tree, read_file, rg, apply_patch, run_command, submit), and an explicit trajectory-returning run_agent.py. The records are self-contained local Python tasks and do not require sandbox services, package installation, or network access.

DuelGrid is a larger agentic demo with a browser game UI and multi-action turns. Before GSPO/RLVR post-training, Gemma-E2B-it performs poorly in this game and often moves back and forth without progress. After training, it learns to collect pickups, chase the user, attack when in range, and avoid trap tiles.

Train before	Reward	Train after

See examples/agentic/duelgrid for the rule engine, fixed-path dataset loader, reward function, OpenAI-compatible agent, and browser UI.

For the full list of training options, run areno train --help.

Serve a trained checkpoint as an OpenAI-compatible endpoint with continuous batching:

areno serve \
  --model-path /path/to/model \
  --tp-size 1 \
  --world-size 1 \
  --port 8000

Point any OpenAI client at http://localhost:8000/v1/chat/completions to start generating. For the full list of serving options, run areno serve --help.

Use --disable-thinking with areno serve to pass enable_thinking=False to compatible tokenizer chat templates for request rendering.

If you want to contribute to AReno or customize it for your own needs, read the contribution guide and make a development install:

git clone https://github.com/inclusionAI/AReno.git
cd AReno
bash scripts/install.sh

# Set up pre-commit hooks (formatting, linting, commit message checks)
pip install pre-commit
pre-commit install --install-hooks

New algorithms, model adapters, kernels, reward functions, and hardware backends all have first-class extension points, so most contributions land without forking the core.

If you find the project helpful, please cite:

@misc{areno2026,
  title        = {AReno: A Self-Contained, Full-Stack Toolkit for Single-Node LLM RL Post-Training},
  author       = {Zibo He and Le Su and Zongyu Li and Xiaowei Zhu and Cheng Wang and Zhenxuan Pan},
  year         = {2026},
  url          = {https://github.com/inclusionAI/AReno},
  license      = {Apache-2.0}
}

AReno's API design is inspired by Tinker from ThinkingMachines. We would like to express our gratitude for their pioneering work.

This repository's source code is available under the Apache 2.0 License.