Skip to content

dm_control soccer 2v2 self-play with opponent league (EnvPool) - #354

Open
Denys88 wants to merge 1 commit into
masterfrom
feat/dmc-soccer-selfplay
Open

dm_control soccer 2v2 self-play with opponent league (EnvPool)#354
Denys88 wants to merge 1 commit into
masterfrom
feat/dmc-soccer-selfplay

Conversation

@Denys88

@Denys88 Denys88 commented Jun 11, 2026

Copy link
Copy Markdown
Owner

Summary

Multi-agent self-play training for EnvPool's dm_control locomotion soccer envs (BoxheadSoccer2v2-v1, from Denys88/envpool#1): one shared PPO policy controls the home team across 256 parallel matches while an 8-type opponent league plays the away team — scripted anchors (chaser, keeper, random, zero) plus frozen past checkpoints of the training policy (lagged self-play), auto-refreshed from the run dir.

python runner.py --train --file rl_games/configs/dm_control/boxhead_soccer_2v2_selfplay.yaml
python -m rl_games.envs.dmc_soccer_tools video --camera 3 --out match.mp4
python -m rl_games.envs.dmc_soccer_tools tournament --run-dir runs/boxhead_soccer_2v2_selfplay/nn

What's included

  • rl_games/envs/dmc_soccer_selfplay.py — vecenv adapter: players-as-actors batching (egocentric, team-relative obs + within-team one-hot player id), shaped rewards with dense-annealing
  • rl_games/envs/dmc_soccer_opponents.py — opponent league + a minimal FrozenPolicy (rebuilds the actor from a checkpoint for cheap CPU inference; handles _orig_mod. torch.compile prefixes)
  • rl_games/envs/dmc_soccer_tools.py — match video rendering + checkpoint round-robin tournament (goal-diff/ep cross table)
  • config + docs/DMC_SOCCER_SELFPLAY.md — the validated recipe, documented as the seven failure modes each choice prevents (each observed and reproduced during tuning):
    1. concede penalty → ball-avoidance ("cowardice")
    2. dense stream ≥ goal reward + terminate-on-goal → dribble-farming
    3. two-sided ball-progress → punished by opponent attacks (learned helplessness)
    4. per-player chase rewards → everyone crowds the ball (fix: team-level chase)
    5. single-opponent training → late collapse (fix: the league)
    6. shaped-reward proxy plateau (fix: dense anneal → broke a 0.34→0.46 goals/ep plateau)
    7. entropy bonus + fixed_sigma → unbounded σ growth melts the policy into noise (fix: entropy_coef: 0)
  • Registry entries follow the existing pattern (_create_envpool-style lazy import in vecenv.py, env_configurations.py entry)

Validation

  • 1.64B frames / 50k epochs on laptop CPU (~64k frames/s end-to-end incl. PPO), zero collapses
  • Final checkpoint beats all scripted anchors (goal-diff/ep: chaser +0.46, keeper +0.56, random +0.53) and its own mid-training self both home and away
  • Smoke-tested in this branch: runner.py --train (3 epochs), tournament, and video render all pass

Dependencies

🤖 Generated with Claude Code

@DenSumy @claude
feat: dm_control soccer 2v2 self-play with opponent league (EnvPool)
ed5fe60 
Adds a multi-agent self-play training setup for EnvPool's dm_control
locomotion soccer envs:

- rl_games/envs/dmc_soccer_selfplay.py: vecenv adapter exposing each
  controlled player as an independent actor sharing one policy (egocentric,
  team-relative obs + within-team one-hot player id), with shaped rewards
  (asymmetric goal, one-sided ball progress, team-level chase, dense-anneal).
- rl_games/envs/dmc_soccer_opponents.py: opponent league for the away team —
  scripted anchors (chaser/keeper/random/zero) plus frozen past checkpoints
  of the training policy (lagged self-play), auto-refreshed from the run dir.
- rl_games/envs/dmc_soccer_tools.py: match video rendering and a
  checkpoint round-robin tournament (goal-diff/episode cross table).
- config + docs: the validated recipe and the seven failure modes each
  reward/algo choice prevents (cowardice, dribble-farming, learned
  helplessness, ball-crowding, single-opponent collapse, proxy plateau,
  entropy-driven sigma explosion).

Validated at 1.64B frames / 50k epochs on laptop CPU (~64k fps end-to-end,
zero collapses); final checkpoint beats all scripted anchors and its own
mid-training self both home and away.

Requires EnvPool soccer envs (Denys88/envpool#1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ViktorM

ViktorM commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Fine to merge low-key for research use after a rebase and three small
guards. Core-safety verified: purely additive, lazy imports (core suite
passes with envpool/cv2 absent), no changes to training code paths.

Before merge:

  • Rebase: now conflicts with master in env_configurations.py /
    vecenv.py — both sides append registry entries, keep both.
  • OpponentLeague._refresh_league's bare except Exception: pass silently
    replaces league opponents with weak-random on any checkpoint mismatch —
    this can invalidate a self-play experiment invisibly. Warn once on
    failure.
  • opponent='random' uses the unseeded global numpy RNG while 'league'
    mode is seeded — use a per-instance RNG for reproducibility.
  • _shaped_reward mutates the envpool obs dict in place through a reshape
    view — copy before modifying (cheap insurance even if currently benign).

Cosmetics (same commit): duplicate numpy import in
dmc_soccer_tools.py; dead comment referencing nonexistent train.py.

Deferred by design (planned self-play improvements pass): offline unit
tests for league/FrozenPolicy/reward shaping, FrozenPolicy's hardcoded
ELU/fixed_sigma/normalize_input assumptions, detailed docs + training
results.

Release note line: "Experimental: dm_control soccer 2v2 self-play with
opponent league (requires an envpool build with BoxheadSoccer2v2-v1;
interfaces may change)."

@ViktorM
ViktorM self-requested a review August 16, 2026 03:25
@ViktorM ViktorM self-assigned this Aug 16, 2026
@ViktorM ViktorM mentioned this pull request Aug 21, 2026
Open
ViktorM added a commit that referenced this pull request Sep 16, 2026
@ViktorM @claude
Soccer self-play review fixes: loud league degradation, seeded RNG, n…
69658d2 
…o obs mutation

Review follow-ups on the self-play addition (#354):
- OpponentLeague._refresh_league warns once when checkpoint loading fails
  instead of silently degrading the league to weak-random opponents (which
  invalidates a self-play run while looking like healthy training).
- The 'random' opponent mode uses a per-instance seeded RNG like league
  mode, instead of the global numpy RNG.
- _shaped_reward copies the reshape view before the team-chase broadcast
  writes into it -- never mutate the env's obs dict.
- Cosmetics: duplicate numpy import, dead train.py comment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ViktorM ViktorM Awaiting requested review from ViktorM

At least 1 approving review is required to merge this pull request.

Assignees

@ViktorM ViktorM

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Denys88 @ViktorM @DenSumy