Skip to content

Releases: huggingface/trl

Release list

v1.10.0

v1.10.0 Latest
Latest

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 13 Aug 01:36
v1.10.0
a7be897
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

🎓 DistillationTrainer is now a stable trainer

After a ~30-PR refactor that reshaped its data contract, loss path, generation stack, config surface, and tests, DistillationTrainer and DistillationConfig graduate from trl.experimental.distillation to the top-level trl package. Same import surface as SFT / DPO / GRPO / KTO. The old experimental path still works and emits a FutureWarning (removal in v2.0.0).

# Before
from trl.experimental.distillation import DistillationConfig, DistillationTrainer

# Now
from trl import DistillationConfig, DistillationTrainer

Also lands a trl distillation CLI and moves the tests to tests/. The full refactor arc: switched signature columns to prompt (deprecated messages-format), pinned the generation stack to GRPO's, wired the chunked JSD loss and deleted the full-logit path, cleaned the Liger path to share extraction with the chunked path, adopted GRPO's log() / training_step timing / _save_checkpoint, rebuilt the test suite to GRPO shape, and much more.

by @qgallouedec across ~30 PRs (#6479, #6480, #6481, #6482, #6484, #6487, #6497, #6508, #6509, #6510, #6511, #6512, #6513, #6521, #6522, #6523, #6524, #6525, #6526, #6530, #6537, #6604, #6605, #6606, #6607, #6609, #6610, #6611, #6612, #6613, #6614, #6629, #6630, #6632, #6633, #6634, #6639, #6640, #6641, #6642, #6643, #6644, #6645, #6647, #6653).

DistillationTrainer supports Vision Language Models

Alongside the graduation, VLMs work end-to-end in DistillationTrainer.

by @qgallouedec in #6650

AsyncGRPO: loop-owning agent training (OpenEnv harness)

New experimental loop-owning (black-box) path for AsyncGRPOTrainer — for training external agents like opencode that run their own tool loop, rather than TRL driving each turn as in the environment_factory white-box path.

The agent runs in an OpenEnv session in transparent_proxy mode; an in-sandbox proxy captures each turn's token ids and logprobs. On completion, TRL reads the trace, rebuilds per-turn training rows, and scores the workspace with the session's verify(). Ships with three caller-supplied policy hooks (rollout_reward_fn, train_turn_fn, agent_turn_fn) so you can drop framework aux calls (title generator, context summarizer) and reinforce only the turns you want.

Includes a self-contained examples/scripts/openenv/opencode.py (local subprocess sandbox + DeepCoder held-out stdin/stdout verifier), validated end-to-end on Qwen3-4B.

by @AmineDiro in #6420, plus HF-sandbox variant by @sergiopaniego in #6565

AsyncGRPO: observability + sampling controls

DiffusionGemma block-diffusion SFT example

A new SFT example for google/diffusiongemma-26B-A4B-it implementing the reference recipe: one response block per step, uniform random token corruption with t ~ U(eps, 1), two-pass self-conditioning, flat cross-entropy over the whole canvas plus an autoregressive co-loss on the encoder. LoRA targets attention + dense MLP linears. Requires transformers >= 5.12.0.

Ships with diffusion_gemma.jinja / diffusion_gemma_training.jinja chat templates (with {% generation %} markers) so assistant_only_loss=True works out of the box.

by @kashif in #6003

More model & template support

Default changes

Two default flips this cycle — pinning them explicitly is recommended if you want the old behavior.

  • SFTConfig.max_completion_length / GRPOConfig.max_completion_length: default bumped from 256 to 512. By @dhruvnigam93 in #6264
  • GRPOConfig.use_bias_correction_kl now defaults to True. By @gowtham-sai-yadav in #6503

Packing and padding-free on VLMs with text-only data

VLM configurations no longer reject packing / padding-free when the batch happens to be text-only.

by @cris96spa in #6547

Other

vLLM version sweep

Fixes

  • Fix DAPO / CISPO / VESPO loss normalization when steps_per_generation != gradient_accumulation_steps — gradients were silently mis-scaled by gradient_accumulation_steps / steps_per_generation (e.g. 0.5× or 4× the intended token-mean in common configs). Default-equivalent configs are unaffected. By @0xadvait in #6024
  • Fix Liger kernel crash on pre-Ampere GPUs by @albertvillanova in #6518
  • Fix vLLM weight synchronization when vllm_enable_sleep_mode=True by @muupan in #5313
  • Fix vLLM server-mode communicator initialization to use the current accelerator device by @sywangyi in #6417
  • Use accelerator device instead of hardcoded cuda in FSDP2 vLLM weight sync by @verma8076 in #6592
  • Fix prepare_deepspeed crash with CPU offload optimizer by @roycho96 in #5916
  • fix(ppo): exclude padding tokens from entropy calculation by @mukund1985 in #6121
  • fix: avoid CopySlices when scaling policy logits by @DaoyuanLi2816 in #6554
  • Fix multimodal field handling in GRPO tool-call loop by @HariomJangra in #6286
  • [GRPO] Apply the completion mask elementwise in the LuSPO loss aggregation by @YaseenBashaT in #6654
  • [GRPO] Fix entropy bonus normalization inconsistency across loss types by @YaseenBashaT in #6648
  • Fix incorrect example script path in Online DPO docs by @latent-9 in #6598
  • Point the opencode example at huggingface/OpenEnv by @sergiopaniego in #6529
  • Remove symlink support from skills installation by @albertvillanova in #6665

Documentation and Examples

CI

Read more

Contributors

kashif, muupan, and 22 other contributors
Assets 2
Loading

v1.9.2

v1.9.2

Choose a tag to compare

View all tags
@albertvillanova albertvillanova released this 28 Jul 10:39
v1.9.2
33f9e46
This commit was signed with the committer’s verified signature.
albertvillanova Albert Villanova del Moral
GPG key ID: 49A42B09B88A48F4
Verified
Learn about vigilant mode.

What's Changed

  • Revert xfail for NemotronH GRPO/RLOO tests now that transformers#47569 fixed the kernels bug by @albertvillanova in #6558
  • Fix test_peft_with_quantization tests for bitsandbytes 0.50.0 by @albertvillanova in #6546

Full Changelog: v1.9.1...v1.9.2

Contributors

albertvillanova
Loading

v1.9.1

v1.9.1

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 26 Jul 16:49
v1.9.1
7a0456e

What's Changed

Full Changelog: v1.9.0...v1.9.1

Contributors

albertvillanova, AmineDiro, and 4 other contributors
Loading

v1.9.0

v1.9.0

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 21 Jul 20:36
v1.9.0
35def3e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

Iterable / streaming datasets in GRPO and RLOO

Long-standing request finally landed. GRPO and RLOO rely on RepeatSampler to repeat each prompt num_generations times and group them across processes — but samplers can't attach to iterable datasets (no length, no indexing), and the old behavior silently bypassed the sampler, quietly corrupting advantage computation.

A new repeat_iterable_dataset generator reproduces RepeatSampler's exact ordering by transforming the stream itself instead of reordering indices. Trainers wrap iterable datasets via IterableDataset.from_generator, force dispatch_batches=False, and shard the stream per-process to preserve prompt groups after cross-process gathering.

from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer

dataset = load_dataset("trl-lib/DeepMath-103K", split="train", streaming=True)

trainer = GRPOTrainer(
    model="Qwen/Qwen3-4B",
    args=GRPOConfig(max_steps=1000),  # required for iterable datasets
    train_dataset=dataset,
    reward_funcs=accuracy_reward,
)

max_steps is required for iterable datasets (no length). A clear error also fires if dispatch_batches=True is combined with an iterable dataset.

by @albertvillanova in #6351

Environment-owned datasets

When an environment_factory is provided, the environment can now own the data. train_dataset becomes optional: reset() returns the prompt. No more fabricating a dummy dataset just to drive the loop.

class WordleEnv:
    def reset(self, **kwargs) -> str:      # returns the prompt
        self._target = sample(words)
        return f"Guess a {len(self._target)}-letter word."
    def get_reward(self) -> float:
        return 1.0 if self._solved else 0.0
    def guess(self, word: str) -> str:     # exposed tool
        self._solved = word == self._target
        return _feedback(word, self._target)

trainer = GRPOTrainer(
    model="Qwen/Qwen3-4B",
    args=GRPOConfig(max_steps=1000),
    environment_factory=WordleEnv,          # no train_dataset needed
)

Applies to both GRPOTrainer and AsyncGRPOTrainer. A provided train_dataset still works exactly as before; multi-environment routing datasets can now be environment-only (no prompt column required).

by @qgallouedec in #6349

Message-level rollouts in AsyncGRPO

An opt-in way to build training rows from a multi-turn conversation. Message mode keeps the conversation as messages and re-tokenizes the whole thing each turn, then checks whether the fresh tokens still start with the tokens held so far. If yes → append the new part (same as token mode). If no → a rewrite happened → close the row and open a new one that matches what the model actually read.

AsyncGRPOConfig(
    rollout_protocol="message",   # "token" (default) | "message"
    fork_threshold_tokens=1024,   # message-mode only
)

Per-turn drift is classified as CLEAN (append), REALIGN (last-answer tail wobble → overwrite same row), or FORK (real rewrite → new row). One advantage per conversation, stamped on every row it produced; under token-mean loss a fork is invisible. Sets up the plumbing for future tree-trajectory support — scoring is already branch-agnostic (groups by rollout_id).

by @AmineDiro in #6250

AsyncGRPO: VLLMClient and decoupled weight sync

A follow-up to v1.8's rollout worker split. A small VLLMClient becomes the single place that speaks HTTP to the vLLM server (named methods for wait_for_server_ready, get_max_model_len, pause/resume, weight-update endpoints — stateless, so picklable). Generation deliberately stays on its own async aiohttp session in the rollout child.

Weight sync is now built from config independent of rollout_worker (previously a custom rollout worker force-set it to None), and is injectable via a new WeightTransferProtocol. token_budget now waits for /health before defaulting to get_max_model_len(), so a slow-starting vLLM waits instead of failing the run.

by @AmineDiro in #6269

DistillationTrainer refactor

A ~13-PR sweep that reshapes the experimental DistillationTrainer for future stability:

  • Extract ServerDistillationTrainer for the teacher-server path (#6454)
  • Move IW-OPD to a dedicated experimental trainer (#6453)
  • Remove the local sparse top-1 loss path from the base trainer (#6455)
  • Remove top-k support from the base generalized JSD loss (#6456)
  • Validate the teacher by vocab size instead of tokenizer identity (#6457)
  • Deprecate & remove lmbda on/off-policy mixing and the off-policy training branch (#6458, #6460); GKD paper reproduction now points at GKDTrainer (#6459)
  • Accept a prompt column alongside messages (#6461); deprecate messages-format datasets (#6474)
  • ⚠️ Fix num_items_in_batch to count generated completion tokens — the loss denominator was wrong for on-policy training (counted dataset completions, not generated ones) and NaN'd on prompt-only datasets. This changes loss values for anyone on the old path (#6478)
  • Preparatory pins on the objective and on/off-policy behavior end to end (#6450, #6451, #6452)

All by @qgallouedec.

New IW-OPD distillation objective

Importance-Weighted On-Policy Distillation lands as an optional objective on the experimental DistillationTrainer — detached IW-OPD advantages built from sampled-token teacher and rollout logprobs (with cached vLLM rollout logprobs when use_vllm=True).

DistillationConfig(
    distillation_objective="iw_opd",
    iw_opd_gamma=1.0,
    iw_opd_epsilon=0.2,
    lmbda=1.0, reverse_kl_top_1_mode="sampled",  # required
)

Now split into its own experimental trainer as part of the refactor above.

by @kashif in #6191

log_multimodal for GRPO / RLOO

New config toggle to control whether images from multimodal completions are logged to trackers. Also applied to experimental DPPO and GRPO-with-replay-buffer.

by @apardyl in #5408 and by @albertvillanova in #6352

vLLM version sweep

Other

Deprecations and removals

  • Remove the experimental DPPO trainer by @qgallouedec in #6402
  • Deprecate messages-format datasets in DistillationTrainer — use prompt column instead — by @qgallouedec in #6474
  • Deprecate lmbda on/off-policy mixing in DistillationTrainer — off-policy path fully removed by @qgallouedec in #6458 and #6460

Fixes

  • Fix biased cross-rank aggregation of token-weighted metrics in GRPO / RLOO — mean-of-means was biased when per-rank token counts differ; now reduces numerator + count jointly for a true global mean. Also fixes clip_ratio/low_min / high_max to be per-completion instead of per-rank-batch-mean (DP-layout-independent). Logging-only, no loss change. By @qgallouedec in #6380
  • Fix DPO/KTO use_liger_kernel under DeepSpeed ZeRO-3 by @qgallouedec in #6372
  • Force use_reentrant=True for PEFT + ZeRO-3 + gradient checkpointing across all trainers by @qgallouedec in #6356
  • Fix precompute_ref_log_probs=True with DeepSpeed in DPO and KTO by @qgallouedec in #6403
  • Fix cast_lm_head_to_fp32 under FSDP2 / DeepSpeed ZeRO-3 by @qgallouedec in #6330
  • Fix corrupted reasoning blocks in Qwen3-family training chat templates — the both-tags guard broke replay of s...
Read more

Contributors

kashif, albertvillanova, and 11 other contributors
Loading

v1.8.0

v1.8.0

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 09 Jul 18:41
v1.8.0
95809b9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

🎓 KTO is now a stable trainer

After many cycles of KTOTrainer ↔ DPOTrainer alignment work, KTO graduates from trl.experimental.kto to the top-level trl package. Same API as DPO/GRPO/SFT — imports move from experimental, tests move to the main test tree, docs no longer flag it as experimental. The experimental path still works and emits a FutureWarning (removal in v2.0.0).

# Before
from trl.experimental.kto import KTOConfig, KTOTrainer

# Now
from trl import KTOConfig, KTOTrainer

Per our telemetry, KTO is the 4th most used trainer in TRL — this graduation was overdue.

by @albertvillanova in #6175, #6287 and #6345

Environment-owned rewards & multi-environment support

Three interrelated changes make agentic RL training with environments substantially more ergonomic.

Environment-owned reward. If your environment_factory env defines a reserved get_reward() method (no args → float), it's called once per completed rollout and treated as a reward source. reward_funcs becomes optional — no more leaking env state back out to trainer-owned reward funcs.

class WordleEnv:
    def reset(self, **kwargs):
        self._target = sample(words); self._solved = False

    def get_reward(self) -> float:       # optional, reserved (not a tool)
        return 1.0 if self._solved else 0.0

    def guess(self, word: str) -> str:   # exposed as a tool
        self._solved = word == self._target; ...

trainer = GRPOTrainer(
    model=model,
    train_dataset=dataset,
    environment_factory=WordleEnv,       # no reward_funcs needed
)

Multi-environment support. environment_factory now accepts dict[str, factory] in addition to a single callable. Each dataset row selects its environment via an environment column, and only that env's tools are exposed in that row's prompt — so a coding task and a game can train together in one run without leaking each other's tool schemas. Single-callable usage is unchanged.

Same wiring lands in GRPO, AsyncGRPO, DPPO, and GRPO-with-replay-buffer.

Env-owned reward by @qgallouedec in #6238; multi-env in #6001 and #6002

Entropy regularization for GRPO

GRPOConfig now supports both static and adaptive entropy regularization (Skywork-OR1). The bonus encourages exploration and helps prevent premature policy collapse.

GRPOConfig(
    entropy_coef=0.01,          # static
    # or:
    use_adaptive_entropy=True,  # adjust to target entropy
    entropy_target=1.5,
    entropy_coef_delta=0.01,
    entropy_coef_min=0.0,
    entropy_coef_max=0.1,
)

Adaptive mode adjusts the coefficient once per optimizer step from window-aggregated entropy (gradient accumulation-aware) and persists entropy_coef in the checkpoint for resume. Not compatible with the Liger kernel.

by @albertvillanova in #6140

quantization_config trainer argument (streamlined QLoRA)

QLoRA no longer requires reaching into model_init_kwargs or pre-loading the model manually.

SFTTrainer(
    model="meta-llama/Llama-2-7b-hf",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
    peft_config=LoraConfig(),
    train_dataset=dataset,
)

Added to SFTTrainer, DPOTrainer, GRPOTrainer, RLOOTrainer, RewardTrainer, and KTOTrainer. Sits next to peft_config (the other non-serializable QLoRA ingredient), flows into from_pretrained, and raises if also set in args.model_init_kwargs. Also drops the redundant get_kbit_device_map() line — QLoRA trains identically without it across all tested configurations.

by @qgallouedec in #6157 and #6276

MoE aux loss extends to DPO and KTO

v1.7 added the router load-balancing auxiliary loss to GRPOTrainer / RLOOTrainer / AsyncGRPOTrainer. It's now on DPOTrainer and KTOTrainer too, so post-training MoE models with preference data keeps experts balanced.

by @qgallouedec in #6208 and #6275

Neuron-friendly chunked_nll via static-shape token packing

The chunked NLL path had data-dependent indexing that broke XLA/Neuron compilation. This PR reworks it to pack valid tokens to the front via a stable argsort on the validity mask, iterate over ceil(n_valid / chunk_size) * chunk_size whole chunks (a tensor, no Python-int sync), and use ignore_index=-100 inside each chunk. GPU behavior is unchanged; Neuron now works too — same tiny memory footprint (~1.33 GiB vs the naive 5.94 GiB alternative) with no per-mask recompilation.

by @michaelbenayoun in #6314

Packing-aware dynamic batching in AsyncGRPO

AsyncGRPO micro-batching becomes packing-aware and token-bounded on top of the padding-free path from v1.7:

  • Σ Lᵢ² row balancing — a greedy longest-first assignment gives every DP row the same Σ Lᵢ² instead of the same token count (attention is O(L²), FFN is O(L), so equal token counts don't equalize wall-clock). Cross-rank stragglers vanish; +19% MFU at 4B in the benchmark.
  • Token-budget packing (opt-in) — cap each row at token_budget tokens with a variable sample count, decoupling peak memory from per_device_train_batch_size. Useful in memory-bound regimes (no gradient checkpointing, long context, very large models).

Both ride HF Trainer's existing gradient accumulation — no training-loop surgery, FSDP/EP collectives stay in lockstep.

by @AmineDiro in #6092

VLM support in GOLDTrainer

GOLDTrainer now supports vision-language models end-to-end.

by @Strongich in #5969

Support tool calling in KTO

by @qgallouedec in #6259

Support PEFT + Liger in DPO and KTO

by @albertvillanova in #6159 and #6277

Per-dataset fraction in dataset mixtures

You can now weight a DatasetMixtureConfig by fraction instead of only by explicit counts.

by @qgallouedec in #6199

SFT: truncate during dataset preparation

Continuing the label-refactor from v1.7 (#6037), truncation moves out of the collator and into dataset preparation. SFTTrainer and DPOTrainer now truncate consistently at the same phase.

by @qgallouedec in #6155

Simplify tokenization [1-5/N]

A big refactor pass merging DPO / SFT / Reward / KTO tokenization into one shared implementation:

  • Remove redundant is_vlm parameter — #6298
  • Make _tokenize a module-level function — #6301
  • Factor _tokenize into a single shared function — #6302
  • Bundle apply_chat_template kwargs — #6305
  • Pass chat_template as apply_chat_template_kwargs — #6315

All by @albertvillanova. Related: align collators across DPO / SFT / Reward / KTO by @qgallouedec in #6178.

Other

Deprecations and removals

  • Remove GFPOTrainer by @qgallouedec in #6309 — no known usage, and its behavior can be reproduced by adjusting reward_funcs.
  • Remove PAPOTrainer by @qgallouedec in #6235
  • Remove post-training-toolkit integration by @qgallouedec in...
Read more

Contributors

winglian, muupan, and 13 other contributors
Loading

v1.7.1

v1.7.1

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 04 Jul 04:05
v1.7.1
f279c23

What's Changed

Full Changelog: v1.7.0...v1.7.1

Contributors

winglian, albertvillanova, and qgallouedec
Loading

v1.7.0

v1.7.0

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 25 Jun 22:52
v1.7.0
06b42c7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

SFT default loss_type is now "chunked_nll"

The flip announced in v1.6 has landed. Setting loss_type is optional, and the default now resolves to "chunked_nll" — giving every SFTTrainer run ~30% less peak VRAM on average (up to ~50% on large-vocab models) with wall-clock time neutral or slightly faster. No action needed.

The auto-resolve falls back to "nll" when use_liger_kernel=True (the two paths are incompatible). If you want the old behavior — e.g. for custom heads — pin it explicitly:

SFTConfig(loss_type="nll")

by @qgallouedec in #5846

MoE auxiliary loss in GRPO / RLOO / AsyncGRPO

Post-training MoE models now correctly include the router load-balancing auxiliary loss, matching the model's own reference forward and SFTTrainer. Enable via model_init_kwargs:

GRPOConfig(
    ...,
    model_init_kwargs={"output_router_logits": True, "router_aux_loss_coef": 0.001},
)

Plumbed through _get_per_token_logps_and_entropies (now returns a 3-tuple including aux_loss), folded into the policy loss with grad-accum scaling matched per trainer, and logged as aux_loss. AsyncGRPO recomputes it via load_balancing_loss_func in the chunked LM-head path (same as SFT's chunked path).

by @AmineDiro in #6083, plus router_aux_loss_coef config wiring by @qgallouedec in #6085

New experimental GMPO trainer

Geometric-Mean Policy Optimization lands as an experimental trainer. Replaces GRPO's per-token arithmetic mean of importance ratios with a sequence-level geometric mean (mean of clipped log-ratios, then exp); clipping is one-sided by advantage sign and applied in log space. Default epsilon=0.4 per the paper.

from trl.experimental.gmpo import GMPOConfig, GMPOTrainer

trainer = GMPOTrainer(
    model="Qwen/Qwen3-4B",
    args=GMPOConfig(epsilon=0.4),
    reward_funcs=accuracy_reward,
    train_dataset=dataset,
)

by @raghulchandramouli in #6078

Transformers continuous batching in GRPO / RLOO

use_transformers_paged was deprecated in v1.4; it's now replaced with proper transformers continuous batching. The old branch silently bypassed importance-sampling correction (logprobs = None); the new path captures logprobs from output.logprobs and exposes a ContinuousBatchingConfig for KV-cache tuning.

GRPOConfig(
    ...,
    use_transformers_continuous_batching=True,
    transformers_continuous_batching_config={
        "use_cuda_graph": False,
        "max_memory_percent": 0.4,  # leave headroom for training
    },
)

Benchmark (Llama-3.2-1B-Instruct, A100 80GB, GSM8K): 1.25× faster at N=64 generations with -16 GB peak VRAM vs default generate(). Use when N ≥ 32 with variable completion lengths.

use_transformers_paged=True still works and forwards to the new flag with a FutureWarning. Requires transformers>=5.8.0.

by @sergiopaniego in #5765

AsyncGRPO: native weight sync with vLLM ≥ 0.22.0

WeightTransferClient now drives vLLM's native 4-phase RL weight-transfer API instead of the older 2-call flow: pause(mode="keep") → start_weight_update → threaded update_weights + NCCL broadcast → finish_weight_update → resume. Validated end-to-end on H100 across single-node, FSDP2×4 + TP=4, and 2-node FSDP2×4 + DP=2×TP=4 (weight-sync time ≈ 0.18-0.8 s).

by @AmineDiro in #5892

Padding-free training in AsyncGRPO

AsyncGRPO now supports the same padding-free path SFT already had. Flattens the batch and uses position_ids-based document boundaries instead of right-padding to the longest sequence — meaningful speedup and memory savings on heterogeneous-length workloads.

by @qgallouedec in #5854

Experimental Harbor integration

A new trl.experimental.harbor adapter plugs Harbor agentic task suites into GRPOTrainer via environment_factory. Same pattern as the OpenReward integration — one spec wires all three trainer slots:

from trl import GRPOConfig, GRPOTrainer
from trl.experimental.harbor import HarborSpec

spec = HarborSpec("AdithyaSK/data_agent_rl_environment_train", agent="bash", num_tasks=64)

trainer = GRPOTrainer(
    model="Qwen/Qwen3-4B",
    args=GRPOConfig(num_generations=8, max_steps=50, max_tool_calling_iterations=25),
    train_dataset=spec.train_dataset,
    environment_factory=spec.environment_factory,
    reward_funcs=spec.reward_funcs,
)

Built-in bash harness, plus jupyter and terminal_notes example harnesses. Gated by the new trl[harbor] extra.

by @adithya-s-k in #6018

trust_remote_code in trainer configs

A single trust_remote_code: bool = False field on the trainer configs now covers the whole load surface — model, processor / tokenizer, reference model, reward model, reward tokenizer, teacher — instead of forcing users to thread it through several independent kwarg dicts.

SFTConfig(trust_remote_code=True)

ModelConfig.trust_remote_code is removed to avoid duplicate --trust_remote_code when combining dataclasses; CLI behavior is unchanged.

by @qgallouedec in #5802

KTO ↔ DPO alignment: tests, evaluate, sync_ref_model

The last alignment cycle before graduation: KTO now has parity with DPO on pad_to_multiple_of, sync_ref_model, evaluate(), method order/signature, metric placement (all moved into _compute_loss), and a real text + VLM (including multi-image) test suite.

PRs all by @albertvillanova: #6029, #6030, #6033, #6034, #6035, #6080, #6093, #6148, #6149, #6150, #6152, #6160, #6163.

LFM2-VL multimodal inputs in GRPO / RLOO

GRPO and RLOO now support LFM2-VL multimodal inputs end-to-end.

by @zwischenraum in #6114

New built-in reward helpers

SFT refactor: build labels during dataset preparation

Label construction moves out of the collator and into dataset preparation, so "what's trainable" is defined in exactly one place. A single batched map produces a labels column where each token keeps its ID when every applicable mask is 1, else -100. Plain LM stays storage-neutral; pre-tokenized datasets with mask columns now go through the same path. Step 1 toward fixing #3927.

by @0xadvait in #6037

Idefics3 chat template

{% generation %}-marker training template for Idefics3, enabling assistant_only_loss=True.

by @aazizyan in #5871

vLLM version sweep

Other

Fixes

  • Share frozen layers with reference model instead of duplicating in memory — create_reference_model with num_shared_layers was double-allocating the "shared" frozen layers because the loop never assigned _ref_param back. Now it does, so shared layers are held once. By @behroozazarkhalili in #6053
  • Fix chunked_nll mixed Tensor/DTensor error under FSDP2 + PEFT by @albertvillanova in #6065
  • Fix per-chunk lm_head.weight all-gathers under FSDP2 + chunked_nll by @albertvillanova in #6077
  • **Fix ZeRO-3 + PEF...
Read more

Contributors

discobot, strickvl, and 21 other contributors
Loading

v1.6.0

v1.6.0

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 11 Jun 22:00
v1.6.0
0dac440
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

AsyncGRPO rollout worker now runs in a separate process

AsyncRolloutWorker is no longer a thread — it's a spawned child process with its own GIL. The trainer's autograd engine no longer competes with recursive_parse / accuracy_reward for the GIL, which was causing 1-5s stalls in real Qwen3-30B-A3B @ 16k runs and ultimately NCCL watchdog timeouts on other ranks.

Architectural changes:

  • AsyncRolloutWorker (parent) owns the child process + shared mp.Queue / mp.Value / mp.Event.
  • _AsyncRolloutLoop (child-only) handles tokenization, dataset iteration, reward funcs, and asyncio loops.
  • A new WeightTransferClient owns the NCCL group with vLLM (/pause, /resume, /init_weight_transfer_engine, /update_weights); the rollout child only talks to /v1/completions.

Two correctness fixes shipped alongside (they would have conflicted otherwise): broader aiohttp retry (now catches ClientPayloadError) with bounded exponential backoff, and all-NaN reward columns are now preserved — np.nansum was silently returning 0, giving unscorable completions a real advantage signal and pushing the policy away from correct answers (~30% of DeepMath / OpenR1-Math rows).

Note

reward_funcs / tools / environment_factory must now be picklable, and the child runs CPU-only (CUDA_VISIBLE_DEVICES="").

by @AmineDiro in #5749

New experimental A2PO trainer (Optimal Advantage Regression)

A new A2POTrainer implements A*-PO from "Accelerating RL for LLM Reasoning with Optimal Advantage Regression". Two stages: an offline V* estimation pass from reference policy samples (with optional filter_all_incorrect to drop prompts where every reference completion fails), then on-policy training with one generation per prompt and a plain least-squares loss on β₂·log(π/π_ref) vs r − V*. No group, no critic, no clipping, no reward normalization.

from trl.experimental.a2po import A2POConfig, A2POTrainer

trainer = A2POTrainer(
    model="Qwen/Qwen3-4B",
    args=A2POConfig(num_value_samples=8, filter_all_incorrect=True),
    train_dataset=dataset,
    reward_funcs=accuracy_reward,
)
trainer.train()

Designed for binary verifiable rewards (math/code), not open-ended problems.

by @raghulchandramouli in #5940

KTO now supports VLMs + big alignment push

The biggest KTO ↔ DPO alignment cycle yet — KTOTrainer now supports vision-language models, plus a deep restructuring of compute_loss, KL dataset generation, ref-logp precomputation, activation offloading, sampler strategy, metrics, and more. KTO graduation is very close.

from trl.experimental.kto import KTOConfig, KTOTrainer

trainer = KTOTrainer(
    model="Qwen/Qwen2.5-VL-3B-Instruct",
    args=KTOConfig(...),
    train_dataset=vision_kto_dataset,
)

VLM support: by @albertvillanova in #5939. Plus ~20 alignment PRs all by @albertvillanova: #5820, #5849, #5852, #5850, #5866, #5864, #5856, #5872, #5875, #5900, #5901, #5899, #5906, #5909, #5914, #5982, #5936, #5996, #5998, #5999.

Cross-tokenizer alignment in GOLD via byte offsets

The GOLD distillation trainer used to align student/teacher tokens by extending two decoded strings and flushing on equality. It silently broke on any byte-level disagreement — including the common case of one tokenizer prepending BOS while the other doesn't (Llama-3 ↔ Qwen-3). The X-Token paper called this out by name.

Each side now carries (start_byte, end_byte) spans derived once from the fast tokenizer's char offsets, and the walker syncs on cumulative byte boundaries. On the on-policy path, spans come from piece_byte_len over the sampled token ids (not from re-encoding the decoded completion — BPE makes that round-trip non-injective).

Two related fixes shipped: long rows no longer lose the completion (now keeping the last max_length tokens), and the vLLM on-policy original_prompt_text is now decoded from the truncated ids the student actually consumed.

by @kashif in #5885

SDFT / SDPO: live teacher logprobs from the vLLM server

When teacher_model_kind="live" and vllm_mode="server", the vLLM generation server already holds the current student weights (synced every step for rollouts). The new use_teacher_server=True flag scores the teacher's log-probs on that same server instead of running a separate local teacher forward — removing the teacher from the training step entirely.

Supported modes: sampled_token (reverse KL on the realized token) and topk_logits. When buffered batches reuse steps (num_iterations > 1), weights are re-synced before scoring so the teacher never scores stale.

by @kashif in #5989

Bidirectional masked importance sampling (MIS) for IcePop

vLLM importance sampling in GRPO now uses a two-sided band [C_min, C_max] instead of a single upper cap, aligning TIS/MIS with IcePop's bidirectional handling of train–inference ratio outliers.

from trl import GRPOConfig

config = GRPOConfig(
    vllm_importance_sampling_clip_min=0.5,
    vllm_importance_sampling_clip_max=2.0,
    vllm_importance_sampling_correction="mask",  # or "truncate"
)

The old vllm_importance_sampling_cap is deprecated and maps to clip_max.

by @casinca in #4732

NemotronH and Nemotron 3 Ultra support

Day-zero training support for NVIDIA's new model families.

Even more training chat templates

Three more model families with {% generation %} markers (assistant-only loss out of the box):

Distributed backend boilerplate, hidden

A new trl/distributed.py introduces a single DistributedBackend class that detects ZeRO stage and FSDP version once, then exposes two context managers (gather_params, summon_full_params) used everywhere. Replaces the scattered getattr(state, "fsdp_plugin", None) / gather_if_zero3 / summon_full_params if ... else nullcontext() boilerplate spread across vllm_generation.py, models/utils.py, and the main trainers. Future deprecations land in one place.

by @albertvillanova in #6000

Decoupled self-distillation trainers

A two-PR refactor that disentangles SDPO, SDFT, and other self-distillation trainers from their shared base, making each one self-contained and consistent with the rest of the codebase.

by @LeonEricsson in #5862 and #5883

Heads-up: SFT default loss_type will change in 1.7

Setting SFTConfig.loss_type is now optional, and leaving it unset emits a FutureWarning: in TRL 1.7 the default will switch from "nll" to "chunked_nll". No action needed — you'll just get the new default automatically on upgrade — unless you want to pin the current behavior (e.g. for custom models) with loss_type="nll".

by @qgallouedec in #5997

Other

Fixes

Read more

Contributors

kashif, aazizyan, and 15 other contributors
Loading

v1.5.1

v1.5.1

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 27 May 15:26
v1.5.1
32efb4b

What's Changed

  • 🔒 Gate trainer telemetry on an explicit class-name allowlist by @qgallouedec in #5851

Full Changelog: v1.5.0...v1.5.1

Contributors

qgallouedec
Loading

v1.5.0

v1.5.0

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 25 May 15:33
v1.5.0
bd1e73f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

Even more training chat templates

Three more model families gain training-compatible templates with {% generation %} markers (so assistant_only_loss=True just works):

Final logits softcapping for async GRPO

The chunked LM-head path used by AsyncGRPOTrainer now supports models that use final_logit_softcapping (notably Gemma 2). _ChunkedLogProbFunction applies logit_scale, optional tanh-based softcapping, and temperature consistently in both forward and backward — softcapped models are no longer rejected.

by @mlarnouhet in #5691

KTO ↔ DPO alignment continues

Two more cycles closer to KTO graduation:

Trainer telemetry (opt-out)

_BaseTrainer.__init__ now emits a single anonymous huggingface_hub.send_telemetry ping per trainer instantiation, so we can finally see which trainers / model families / distributed backends are actually being used in practice and prioritize accordingly.

The payload is intentionally minimal — TRL version, trainer class name, model architecture, PEFT yes/no, distributed backend (deepspeed/fsdp/ddp/none), bucketed world size, device type, GPU model when available. No user data, no dataset names, no model paths, no hyperparameter values, never sent in CI / offline / HF_HUB_DISABLE_TELEMETRY mode.

See usage_stats.md for what's collected and how to opt out.

by @qgallouedec in #5758

Other

Fixes

  • Fix exponential backtracking in qwen3 / qwen3_5 / glm4moe response parsing — GRPOTrainer was hanging indefinitely on truncated <tool_call> blocks (a degenerate case that happens naturally when generation hits max_completion_length mid-tool-call). Rewrote the regex to be non-backtracking — worst case goes from O(2ⁿ) to O(n). By @xodn348 in #5798
  • CUDA memory leak: release BNB dequantization buffers & stale state in OffloadActivations — follow-up to v1.4's activation-offloading leak fix. By @butterwecksolutions in #5730
  • Invalidate ZeRO-3 param coordinator trace in add_hooks by @roycho96 in #4693
  • Fix nested vocab_size for DistillationTrainer and GOLDTrainer by @Beichen-Ma in #5592
  • Fix MPS support in experimental empty_cache() by @jamie-peterson-ml in #5799
  • Fix metric_for_best_model for trainer-specific eval metrics by @qgallouedec in #5811
  • Fix generate_batch: inference tensors blocking inplace ops in background thread by @albertvillanova in #5818
  • Replace deprecated torch_dtype with dtype across examples, docs, notebooks, tests, and experimental distillation / gold trainers by @qgallouedec in #5717

Documentation and Examples

  • docs(grpo): align model to Qwen2.5 and add GRPO OOM tab in quickstart by @xodn348 in #5740

CI

New Contributors

What's Changed

Read more

Contributors

butterwecksolutions, aazizyan, and 11 other contributors
Loading