Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning
English | 中文
This repository contains the official code of Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning.
SAT (Staleness-Adaptive Trust Region) is a trust-region-motivated sampled-ratio clipping plug-in for PPO-like objectives. Rather than a fixed clip radius, it gives every token a directional, staleness-adaptive clip radius: using the observed staleness proxy
Python 3.10+ is required. The core CUDA stack (torch, sglang, transformer_engine, and Megatron-LM) should match your CUDA toolchain — install it from the official channels first, then install the remaining Python dependencies:
pip install -r requirements.txtTo install the package itself (sat_slime) in editable mode:
pip install -e .sat_slime/ Training framework and SAT/DPPO loss implementation
sat_slime_plugins/ Package namespace for optional training plugins
scripts/ Experiment launchers
assets/results/ Static figures used by this README
tests/utils/ SAT unit tests
Only README.md and README_zh.md are retained as Markdown documentation.
All launchers train on the merged DAPO-Math-17k + Dolci-RL-Zero-Math dataset, shipped with the repository under run_all_exp/data/. The scripts default to the original GY4 paths, but every path can be overridden:
export HF_MODEL_PATH=/path/to/Qwen3-30B-A3B-Base
export REF_LOAD_PATH=/path/to/Qwen3-30B-A3B-Base_torch_dist
export TRAIN_DATA_PATH=/path/to/sat/run_all_exp/data/dapo-dolci-merged/dapo-dolci-math-30k.jsonl
export AIME24_DATA_PATH=/path/to/sat/run_all_exp/data/aime-2024/aime-2024.jsonl
export CKPT_SAVE_BASE=/path/to/checkpoints
export MASTER_ADDR=<ray-head-ip>Optional W&B logging:
export WANDB_API_KEY=<key>
export WANDB_PROJECT=satThe reward function is passed as a real training CLI argument:
--custom-rm-path sat_slime.custom_rewards.prime_math_reward.compute_reward
Each method launcher runs both async steps 1 and 8 by default.
Before launching, the four external paths must be provided; no machine-specific absolute path is committed:
export HF_MODEL_PATH=/path/to/Qwen3-30B-A3B-Base
export REF_LOAD_PATH=/path/to/Qwen3-30B-A3B-Base_torch_dist
export TRAIN_DATA_PATH=/path/to/sat/run_all_exp/data/dapo-dolci-merged/dapo-dolci-math-30k.jsonl
export AIME24_DATA_PATH=/path/to/sat/run_all_exp/data/aime-2024/aime-2024.jsonlThe experiment datasets ship with the repository under run_all_exp/data/, so TRAIN_DATA_PATH and AIME24_DATA_PATH can point directly at those files.
All the experiment scripts corresponding to the experiments in the Main results tables can be found in the scripts/ folder — see scripts/README.md for the full list and a reference for every launch variable.
The following results reproduce the three result groups reported in the included HTML analysis and paper. Experiments use Qwen3-30B-A3B-Base and configured asynchronous weight-update intervals lag = 1 and lag = 8. Each reported cell is a single-seed run; the values should not be interpreted as multi-seed significance estimates.
Best observed AIME24 avg@8. Parenthesized values are the training steps of the best checkpoints. “collapse” means that the run later collapsed after reaching the reported best checkpoint.
| Method | lag = 1 | lag = 8 |
|---|---|---|
| Base Model (Qwen3-30B-A3B-Base) | 9.38 | 9.38 |
| GRPO | 31.25 (429) | 30.17 (429, collapse) |
| GSPO | 32.25 (474) | 31.46 (424, collapse) |
| GRPO w/ R3 | 31.83 (314) | 29.58 (214) |
| GSPO w/ R3 | 34.00 (379) | 33.13 (214) |
| GRPO w/ TIS | 31.62 (254) | 31.46 (374) |
| GSPO w/ TIS | 32.88 (319) | 31.21 (279) |
| GRPO w/ TIS w/ R3 | 31.79 (324) | 31.79 (239) |
| GSPO w/ TIS w/ R3 | 31.88 (329) | 32.92 (359) |
| DPPO | 33.96 (324) | 32.71 (289) |
| SAT-GRPO | 32.71 (404) | 32.08 (299) |
| SAT-GSPO | 34.17 (254) | 32.71 (359) |
| SAT-GRPO w/ R3 | 33.75 (284) | 33.13 (289) |
| SAT-GSPO w/ R3 | 35.83 (379) | 34.79 (359) |
SAT-GSPO w/ R3 ranks first at both configured lags, reaching 35.83 at lag 1 and 34.79 at lag 8. The plain GRPO and GSPO lag-8 runs reach competitive best checkpoints but later collapse in the logged window.
The table reports the last-epoch mean sampled-token training–inference mismatch
Lower values indicate less observed sampled mismatch; they are not a complete full-vocabulary
| Method | lag = 1 ↓ | lag = 8 ↓ |
|---|---|---|
| GRPO | 0.0103 | 0.0110 |
| GSPO | 0.0097 | 0.0109 |
| GRPO w/ R3 | 0.0058 | 0.0218 |
| GSPO w/ R3 | 0.0091 | 0.0158 |
| GRPO w/ TIS | 0.0108 | 0.0169 |
| GSPO w/ TIS | 0.0151 | 0.0208 |
| GRPO w/ TIS w/ R3 | 0.0048 | 0.0081 |
| GSPO w/ TIS w/ R3 | 0.0054 | 0.0068 |
| DPPO | 0.0108 | 0.0118 |
| SAT-GRPO | 0.0101 | 0.0108 |
| SAT-GSPO | 0.0095 | 0.0107 |
| SAT-GRPO w/ R3 | 0.0056 | 0.0079 |
| SAT-GSPO w/ R3 | 0.0056 | 0.0076 |
For every paired configuration, the last-epoch mismatch is higher at lag 8 than at lag 1. The R3-augmented SAT configurations remain in the low-mismatch band, while the largest value is observed for GRPO w/ R3 at lag 8 (0.0218).
At lag 1, every R3-augmented curve finishes below the corresponding baseline range, while SAT without R3 remains close to its base recipe. The reported trailing-25% means are approximately: GRPO 0.0102, DPPO 0.0103, GSPO w/ TIS 0.0101, SAT-GSPO 0.0104, GSPO w/ TIS w/ R3 0.0061, GRPO w/ R3 0.0055, GSPO w/ R3 0.0057, and SAT-GSPO w/ R3 0.0058.
At lag 8, the logged GSPO baseline diverges after approximately step 330 and eventually leaves the plotted range, whereas the R3-augmented runs occupy the lower-mismatch group. Reported trailing values include SAT-GRPO w/ R3 ≈0.0071, SAT-GSPO w/ R3 ≈0.0074, GRPO w/ R3 ≈0.0097, GSPO w/ R3 ≈0.0073, and GSPO w/ TIS w/ R3 ≈0.0068. SAT-GSPO remains near 0.0108.
Together, Results 4.1–4.3 show complementary roles in these runs: R3 primarily lowers the observed mismatch level, while SAT changes the directional sampled clipping geometry and intercepts high-mismatch outward updates earlier. These curves describe associations in the reported single runs; they do not establish a unique causal decomposition.
This project is built on THUDM/slime. This implementation builds on and is grateful to the following projects:
If you use SAT in your research or project, please consider citing us:
@misc{yang2026sat,
title = {Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning},
author = {Yang, Junyao and Shi, Yucheng and Li, Zongxia
and Li, Zhongzhi and Wang, Ruhan and Zhou, Xiangxin
and Panaganti, Kishan and Mi, Haitao and Liang, Leowei},
year = {2026},
note = {Tencent Hy LLM Frontier. Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning.}
}