EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic RL

EnvACE is an agentic RL framework that removes the external environment from the training loop. A single shared policy alternates between two roles:

• Act — generate the next tool call / environment-facing action, conditioned on the interaction history.
• Rehearse — play the role of the environment and generate the response induced by that action.

Subsequent decisions are conditioned on the policy's own rehearsed responses, so the trajectory unfolds without querying a real environment or a separate simulator. Both roles are trained jointly with a role-wise GRPO objective (separate baseline per role, shared parameters), so the policy internalizes environment dynamics as an implicit world model. At inference, that internalized world model enables test-time scaling by private rehearsal: run N imagined trajectories in parallel or sequentially, summarize them into a rehearsal memory, then use that memory to guide one committed execution in the real environment.

Real Environment · External Simulator · EnvACE World Rehearsal

• [2026-08-07] We released the code and paper for EnvACE.

• World-rehearsal training. Environment response generation is a role of the acting policy, not a separate module.
• Role-wise GRPO. Per-role advantage baselines, shared policy parameters, jointly optimized end-to-end from task-success reward.
• Test-time rehearsal. Two modes — Parallel (independent attempts) and Sequential (each attempt sees prior rehearsals + revisions) — condensed into a rehearsal memory before a single external execution.
• Reference results. EnvACE-8B: Overall 32.91 across BFCL-v4 / τ²-Bench / VitaBench; TF1 46.78 on FinMCP-Bench — outperforming Simulator-8B, TOUCAN-7B, EnvScaler-8B, AWM-8B/14B, ScaleEnv-8B under the same open-source scale.

conda create -n envace python==3.12 -y
conda activate envace

# EnvACE-specific pinned versions (torch 2.6.0 + cu124, sglang 0.4.6.post5,
# ray 2.49.2, transformers 4.53.2 — validated end-to-end).
pip install -r requirements_envace.txt
pip install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
pip install -e .

Three steps — a judge, a Ray cluster, then the launcher:

# 1. Qwen3-30B-A3B sglang judge (8 GPUs, any reachable node).
python -m sglang.launch_server \
  --model-path <A3B_path> --served-model-name Qwen/Qwen3-30B-A3B \
  --host 0.0.0.0 --port 10001 --tp 8 --mem-fraction-static 0.88

# 2. ray start --head  on node 0, then  ray start --address=<head>:6379  on the rest.

# 3. Train. The script's defaults are the reference run: Qwen3-8B share, 2 × 8 GPUs.
JUDGE_HOST=<judge_ip> bash examples/drmas_trainer/run_checklist_share.sh

Everything else is a ${VAR:-default} in the launcher, so you configure a run by exporting variables rather than editing the script. Read the header of run_checklist_share.sh first — it lists the four prerequisites the script assumes and documents each knob inline. Common overrides:

REFERENCE_MODEL_PATH=$PWD/models/Qwen/Qwen3-8B \
SIMULATOR_MODEL_PATH=$PWD/models/Qwen/Qwen3-8B   # backbone for both roles
trainer_nnodes=1 train_data_size=8               # single node, 8 GPUs
CALLER_MODE=frozen                               # train the Simulator only
bash examples/drmas_trainer/run_checklist_share.sh eval   # validation pass, no updates

Sibling launchers in the same directory cover the other variants: run_checklist_noshare.sh (separate Caller / Simulator actors) and run_checklist_caller_only.sh (no world rehearsal — the baseline).

@article{envace2026,
  title  = {EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning},
  author = {Xu, Zishan and Yao, Zhiyuan and Chen, Yuxin and Guo, Yifu and Lu, Zhengxi and
            Lu, Yuquan and Huang, Jinyang and Xu, Yan and Wang, Yasheng and Zhang, Weinan and
            Zeng, Xingshan and Liu, Weiwen},
  journal = {arXiv preprint arXiv:2608.06197},
  year   = {2026}
}

Code builds on Dr. MAS (multi-agent RL orchestration), verl-agent, and verl. The A3B judge is Qwen3-30B-A3B; the actor backbone is Qwen3-8B. Related environment-scaling baselines we compare against: Simulator, TOUCAN, EnvScaler, AWM, ScaleEnv (see the paper for citations).