Skip to content
jyyang26Public

About

Staleness-Adaptive Trust Region (SAT), a directional, token-adaptive clipping method for stabilizing asynchronous reinforcement learning of large language models.

Resources

Stars

24 stars

Watchers

0 watching

Forks

Repository files navigation

Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning

Blog arXiv

English | 中文

This repository contains the official code of Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning.

SAT (Staleness-Adaptive Trust Region) is a trust-region-motivated sampled-ratio clipping plug-in for PPO-like objectives. Rather than a fixed clip radius, it gives every token a directional, staleness-adaptive clip radius: using the observed staleness proxy $|d_{b,t}| = |\log r_{b,t}|$, SAT applies a batch-relative quantile gate so that only tokens on the current microbatch's high-staleness tail are affected, and contracts the sign-selected nominal endpoint of those tokens through staleness-based kernel scaling and effective contraction factors $c_{\pm,b,t}$. This suppresses the divergent, outward gradient updates that drive future distribution drift while leaving pull-back updates and ungated tokens untouched. For GRPO, SAT uses token-level ratios; for GSPO, it uses the sequence ratio broadcast to the response tokens. Full derivations and equations are in the blog and paper.

0. Setup

Python 3.10+ is required. The core CUDA stack (torch, sglang, transformer_engine, and Megatron-LM) should match your CUDA toolchain — install it from the official channels first, then install the remaining Python dependencies:

pip install -r requirements.txt

To install the package itself (sat_slime) in editable mode:

pip install -e .

1. Repository layout

sat_slime/          Training framework and SAT/DPPO loss implementation
sat_slime_plugins/  Package namespace for optional training plugins
scripts/            Experiment launchers
assets/results/     Static figures used by this README
tests/utils/        SAT unit tests

Only README.md and README_zh.md are retained as Markdown documentation.

2. Data and paths

All launchers train on the merged DAPO-Math-17k + Dolci-RL-Zero-Math dataset, shipped with the repository under run_all_exp/data/. The scripts default to the original GY4 paths, but every path can be overridden:

export HF_MODEL_PATH=/path/to/Qwen3-30B-A3B-Base
export REF_LOAD_PATH=/path/to/Qwen3-30B-A3B-Base_torch_dist
export TRAIN_DATA_PATH=/path/to/sat/run_all_exp/data/dapo-dolci-merged/dapo-dolci-math-30k.jsonl
export AIME24_DATA_PATH=/path/to/sat/run_all_exp/data/aime-2024/aime-2024.jsonl
export CKPT_SAVE_BASE=/path/to/checkpoints
export MASTER_ADDR=<ray-head-ip>

Optional W&B logging:

export WANDB_API_KEY=<key>
export WANDB_PROJECT=sat

The reward function is passed as a real training CLI argument:

--custom-rm-path sat_slime.custom_rewards.prime_math_reward.compute_reward

3. Training scripts

Each method launcher runs both async steps 1 and 8 by default.

Before launching, the four external paths must be provided; no machine-specific absolute path is committed:

export HF_MODEL_PATH=/path/to/Qwen3-30B-A3B-Base
export REF_LOAD_PATH=/path/to/Qwen3-30B-A3B-Base_torch_dist
export TRAIN_DATA_PATH=/path/to/sat/run_all_exp/data/dapo-dolci-merged/dapo-dolci-math-30k.jsonl
export AIME24_DATA_PATH=/path/to/sat/run_all_exp/data/aime-2024/aime-2024.jsonl

The experiment datasets ship with the repository under run_all_exp/data/, so TRAIN_DATA_PATH and AIME24_DATA_PATH can point directly at those files.

All the experiment scripts corresponding to the experiments in the Main results tables can be found in the scripts/ folder — see scripts/README.md for the full list and a reference for every launch variable.

4. Main results

The following results reproduce the three result groups reported in the included HTML analysis and paper. Experiments use Qwen3-30B-A3B-Base and configured asynchronous weight-update intervals lag = 1 and lag = 8. Each reported cell is a single-seed run; the values should not be interpreted as multi-seed significance estimates.

Result 4.1 — Complete best-checkpoint grid

Best observed AIME24 avg@8. Parenthesized values are the training steps of the best checkpoints. “collapse” means that the run later collapsed after reaching the reported best checkpoint.

Method lag = 1 lag = 8
Base Model (Qwen3-30B-A3B-Base) 9.38 9.38
GRPO 31.25 (429) 30.17 (429, collapse)
GSPO 32.25 (474) 31.46 (424, collapse)
GRPO w/ R3 31.83 (314) 29.58 (214)
GSPO w/ R3 34.00 (379) 33.13 (214)
GRPO w/ TIS 31.62 (254) 31.46 (374)
GSPO w/ TIS 32.88 (319) 31.21 (279)
GRPO w/ TIS w/ R3 31.79 (324) 31.79 (239)
GSPO w/ TIS w/ R3 31.88 (329) 32.92 (359)
DPPO 33.96 (324) 32.71 (289)
SAT-GRPO 32.71 (404) 32.08 (299)
SAT-GSPO 34.17 (254) 32.71 (359)
SAT-GRPO w/ R3 33.75 (284) 33.13 (289)
SAT-GSPO w/ R3 35.83 (379) 34.79 (359)

SAT-GSPO w/ R3 ranks first at both configured lags, reaching 35.83 at lag 1 and 34.79 at lag 8. The plain GRPO and GSPO lag-8 runs reach competitive best checkpoints but later collapse in the logged window.

Result 4.2 — Complete last-epoch mismatch grid

The table reports the last-epoch mean sampled-token training–inference mismatch

$$ \bar d_\pi =\mathbb E_{(b,t)\in\mathcal T_{\mathrm{act}}} \left|\log \pi(y_{b,t}\mid s_{b,t})-\log \mu(y_{b,t}\mid s_{b,t})\right|. $$

Lower values indicate less observed sampled mismatch; they are not a complete full-vocabulary $D_{\mathrm{TV}}$ measurement and do not by themselves define a return ranking.

Method lag = 1 ↓ lag = 8 ↓
GRPO 0.0103 0.0110
GSPO 0.0097 0.0109
GRPO w/ R3 0.0058 0.0218
GSPO w/ R3 0.0091 0.0158
GRPO w/ TIS 0.0108 0.0169
GSPO w/ TIS 0.0151 0.0208
GRPO w/ TIS w/ R3 0.0048 0.0081
GSPO w/ TIS w/ R3 0.0054 0.0068
DPPO 0.0108 0.0118
SAT-GRPO 0.0101 0.0108
SAT-GSPO 0.0095 0.0107
SAT-GRPO w/ R3 0.0056 0.0079
SAT-GSPO w/ R3 0.0056 0.0076

For every paired configuration, the last-epoch mismatch is higher at lag 8 than at lag 1. The R3-augmented SAT configurations remain in the low-mismatch band, while the largest value is observed for GRPO w/ R3 at lag 8 (0.0218).

Result 4.3 — Training–inference mismatch trajectories

Result 4.3a — Configured lag = 1

Training-inference mismatch over training at lag 1

At lag 1, every R3-augmented curve finishes below the corresponding baseline range, while SAT without R3 remains close to its base recipe. The reported trailing-25% means are approximately: GRPO 0.0102, DPPO 0.0103, GSPO w/ TIS 0.0101, SAT-GSPO 0.0104, GSPO w/ TIS w/ R3 0.0061, GRPO w/ R3 0.0055, GSPO w/ R3 0.0057, and SAT-GSPO w/ R3 0.0058.

Result 4.3b — Configured lag = 8

Training-inference mismatch over training at lag 8

At lag 8, the logged GSPO baseline diverges after approximately step 330 and eventually leaves the plotted range, whereas the R3-augmented runs occupy the lower-mismatch group. Reported trailing values include SAT-GRPO w/ R3 ≈0.0071, SAT-GSPO w/ R3 ≈0.0074, GRPO w/ R3 ≈0.0097, GSPO w/ R3 ≈0.0073, and GSPO w/ TIS w/ R3 ≈0.0068. SAT-GSPO remains near 0.0108.

Together, Results 4.1–4.3 show complementary roles in these runs: R3 primarily lowers the observed mismatch level, while SAT changes the directional sampled clipping geometry and intercepts high-mismatch outward updates earlier. These curves describe associations in the reported single runs; they do not establish a unique causal decomposition.

5. Acknowledgements

This project is built on THUDM/slime. This implementation builds on and is grateful to the following projects:

6. Citation

If you use SAT in your research or project, please consider citing us:

@misc{yang2026sat,
  title  = {Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning},
  author = {Yang, Junyao and Shi, Yucheng and Li, Zongxia
            and Li, Zhongzhi and Wang, Ruhan and Zhou, Xiangxin
            and Panaganti, Kishan and Mi, Haitao and Liang, Leowei},
  year   = {2026},
  note   = {Tencent Hy LLM Frontier. Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning.}
}

About

Staleness-Adaptive Trust Region (SAT), a directional, token-adaptive clipping method for stabilizing asynchronous reinforcement learning of large language models.

Resources

Stars

24 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages