Reinforcement Learning with Verifiable Rewards (RLVR) algorithms such as GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: the per-token entropy variation decomposes into the product of the trajectory-level advantage and an entropy sensitivity function over the next-token distribution, yielding an advantage–surprisal four-quadrant structure and a near-criticality property.
Building on this analysis, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability):
- Entropy-Critical Token Identification: Locating entropy-critical token subsets via batch-internal surprisal quantiles
- Selective Advantage Reweighting: Selectively amplifying the effective advantage of positive-advantage high-surprisal tokens (L+) to sustain exploration
- Closed-Loop Target-Entropy Gating: Activating intervention only when batch mean entropy falls below target entropy H_tgt; reverting to standard GRPO when above the threshold, achieving stable entropy regulation
Below we illustrate the RL training dynamics on two representative scenarios: Qwen2.5-7B Short CoT and Multi-Turn Tool-Use Agent.
This codebase is built on verl v0.7.0. We recommend using the official Docker image:
docker pull verlai/verl:vllm011.latestAlternatively, refer to the verl installation guide for manual environment setup.
We recommend mixing and deduplicating the following open-source datasets: DeepScaleR, Skywork-OR1, Polaris, DAPO-Math-17k
Quick Start (using DAPO-Math-17k as an example):
python recipe/STARE/datasets/dapo_17k_text_train.pyOutput is saved to recipe/STARE/datasets/train/dapo_17k_text_train.parquet, which the training scripts use by default.
We evaluate on 6 math benchmarks, stored in recipe/STARE/datasets/eval/:
- avg@32: AIME24, AIME25, AMC23
- avg@4: MATH-500, MinervaMath, OlympiadBench
The training scripts evaluate on AIME24 and AIME25 by default. Modify the test_files variable to include additional benchmarks.
We use Qwen series models. Download example:
huggingface-cli download Qwen/Qwen2.5-Math-7B --local-dir /root/model_path/Qwen2.5-Math-7B --resume-downloadThe training pipeline fully inherits the standard GRPO framework; enabling STARE only requires toggling a configuration switch.
bash recipe/STARE/scripts/run_short_cot_qwen2.5_math_7b_GRPO_ds.shbash recipe/STARE/scripts/run_short_cot_qwen2.5_math_7b_STARE.shPlease modify
MODEL_PATHandCKPTS_DIRin the scripts before running.
Default configuration (7B): stare_enabled=True, stare_variant=O1, stare_top_p_ratio=0.1, stare_reweight_w=1.1, stare_reweight_m=0.9, stare_target_entropy=0.3.
For 14B/32B models, we recommend milder reweighting: stare_reweight_w=1.05, stare_reweight_m=0.95, and correspondingly increasing max_response_length.
Variant descriptions:
- O1 (default): One-sided entropy amplification: applies W > 1 weight only to L+ (positive-advantage high-surprisal tokens), amplifying entropy-increasing signals
- C2: Two-sided regulation: additionally applies M < 1 attenuation to L- (negative-advantage high-surprisal tokens), simultaneously reducing entropy-decreasing pressure
Training scripts default to 2 nodes × 8 GPUs. Launch procedure:
# Head node
ray start --head --node-ip-address <HEAD_NODE_IP> --num-gpus 8
# Worker node
ray start --address <HEAD_NODE_IP>:6379 --num-gpus 8Submit the training job:
ray job submit --address="http://127.0.0.1:8265" \
--runtime-env-json='{"env_vars": {
"TOKENIZERS_PARALLELISM": "true",
"NCCL_DEBUG": "WARN",
"VLLM_LOGGING_LEVEL": "WARN",
"VLLM_ALLOW_RUNTIME_LORA_UPDATING": "true",
"CUDA_DEVICE_MAX_CONNECTIONS": "1",
"RAY_DEBUG": "legacy",
"VLLM_USE_V1": "1"
}}' \
-- bash recipe/STARE/scripts/run_short_cot_qwen2.5_math_7b_STARE.sh \
2>&1 | tee -a /root/verl_logs/run_stare.logFor more details on distributed training, see the verl documentation.
The core STARE algorithm is implemented in verl/trainer/ppo/core_algos.py (L1070–1293), and the closed-loop controller resides in recipe/STARE/stare_ray_trainer.py.
Algorithm Pipeline:
Input: token log-probs, trajectory-level advantage Â, response mask
Output: token-level reweighting factor ω
1. Partition by advantage sign: T+ = {Â > 0}, T- = {Â < 0}
2. Within each subset, rank tokens by surprisal s = -ln π(o) in descending order,
select Top-P% to form entropy-critical sets L+ (and L-)
3. Closed-loop gating: gate = 𝟙[H̄_batch < H_tgt]
4. Reweighting:
ω = W, if (i,t) ∈ L+ (amplify entropy-increasing signal)
ω = M, if (i,t) ∈ L- [C2] (attenuate entropy-decreasing signal)
ω = 1, otherwise
5. STARE objective: J_STARE = (1/N) Σ ω_i,t · min(ρ_i,t · Â_i, clip(ρ_i,t, 1-ε, 1+ε) · Â_i)
Setting ω ≡ 1 recovers standard GRPO. Because ω > 0, STARE preserves all token-level gradient directions: tokens with positive advantage remain reinforced, while those with negative advantage remain suppressed.
Across model scales from 1.5B to 32B and three task families (Short CoT, Long CoT, and Multi-Turn Tool Use), STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%–8% in average accuracy, with reflection tokens and response length growing in tandem, indicating sustained exploration–exploitation balance.
We thank the verl distributed RL framework, Qwen open-source models, and DAPO, DeepScaleR, Skywork-OR1, Polaris for their open-source data. All competitive baselines can be easily reproduced within verl.