Skip to content

Fix RL training objectives and PPO updates - #56

Merged
GeminiLight merged 1 commit into
mainfrom
agent/fix-rl-training-updates
Aug 12, 2026
Merged

GeminiLight merged 1 commit into
mainfrom
agent/fix-rl-training-updates

Conversation

@GeminiLight

Copy link
Copy Markdown
Owner

Summary

  • correct the A2C entropy term so entropy acts as an exploration bonus
  • freeze PPO reward advantages from rollout-time critic values across repeated updates
  • apply the same PPO advantage contract to ARPPO, safe-PPO, and heterogeneous-GNN variants
  • remove the unused PPO policy deep copy and calculate mini-batch update counts without the extra divisible batch
  • add regression tests for entropy direction, critic-independent PPO actor loss, and update-count boundaries

Root cause

A2C minimized positive entropy, PPO recomputed advantages from the changing critic during repeated updates, and the mini-batch count used 1 + floor(...), which over-updated exactly divisible rollouts. PPO also deep-copied a policy that was never consumed.

Validation

  • uv run pytest -q tests/solver/learning/test_rl_boundaries.py — 31 passed
  • uv run pytest -q — 269 passed
  • uv run python -m compileall -q virne tests
  • git diff --check

@GeminiLight
GeminiLight merged commit df16e04 into main Aug 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant