Article: "Aggregate in the Advantage, Not the Ratio: A Canonical-Form Analysis of Cooperative Multi-Agent Policy Optimization" (under review)
This repository reproduces all figures and tables from the paper. The core finding is straightforward: aggregate neighbors in the advantage, but keep the likelihood ratio per-agent. We systematically analyze two key design decisions in cooperative multi-agent PPO:
- Advantage support (
S^A): which agents' rewards contribute to the advantage - Ratio support (
S^R): which agents' likelihood ratios enter the clipped weight
**Repository Structure: **
credit-support-marl/
├── toy/ # Synthetic cooperative games
│ ├── single_step/ # Single-step gradient-estimator study (Fig 1, Table 1)
│ │ ├── envs.py # Coupling families (NumPy)
│ │ ├── supports.py # Support-matrix construction
│ │ └── make_biasvar2x2.py # Generates Fig 1, prints Table 1
│ ├── training/ # Full PPO training on all four families
│ │ ├── pkg/ # PyTorch package (server-ready)
│ │ │ ├── environments.py, supports.py, learner.py
│ │ │ ├── run_experiment.py # advantage_sweep / ratio_sweep / scaling
│ │ │ └── README.md
│ │ └── np_runner.py # NumPy fallback (no PyTorch) for two families
│ └── plots/ # NPZ → paper figures
│ ├── plot_advsweep.py # → Fig 2
│ ├── plot_fig4adv.py # → Fig 3
│ ├── plot_fig4.py # → Fig 4
│ └── plot_fig5scaling.py # → Fig 5
├── traffic/ # SUMO traffic-signal control (196 intersections)
│ ├── run_traffic.py # One condition per invocation
│ ├── support_mappo.py # Configurable-support MAPPO
│ ├── reward_fns.py # Per-agent local rewards
│ ├── probe_env.py # Builds coupling graph from coupling_adj.npy
│ ├── run_step1.bat/.sh # Advantage study (Fig 6, Table 2)
│ ├── run_step2.bat/.sh # Ratio study with neighbor advantage (Fig 7, Table 3)
│ ├── run_ratio_adv0.bat # Ratio study with independent advantage (Fig 7, Table 3)
│ ├── plot_traffic_errbars.py # → Fig 6, prints Table 2
│ └── plot_traffic_ratio.py # → Fig 7, prints Table 3
├── results/ # Pre-computed NPZ outputs (unzip results.zip)
│ ├── toy/
│ └── traffic/
├── figures/ # Regenerated PNG figures
└── results.zip # All NPZ data packed
| Paper Element | Script | Data Source | Notes |
|---|---|---|---|
| Bias/variance/MSE (2×2) | toy/single_step/make_biasvar2x2.py |
None (computed online) | Self-contained Monte Carlo; also prints Table 1 |
| P vs Q variance | toy/single_step/make_biasvar2x2.py |
None | Printed to stdout |
| Advantage sweep (4 families) | toy/plots/plot_advsweep.py |
results/toy/advsweep_*.npz |
Final return vs ρ_A |
| Advantage training curves | toy/plots/plot_fig4adv.py |
results/toy/advsweep_*.npz |
Team return vs iteration |
| Ratio redundancy curves | toy/plots/plot_fig4.py |
results/toy/ratiosweep_*.npz + npret_*.npz |
Three ratio supports coincide |
| Agent-count scaling | toy/plots/plot_fig5scaling.py |
results/toy/scaling_*.npz + npscale2_*.npz |
Standard vs off-policy stress |
| Traffic advantage study | traffic/plot_traffic_errbars.py |
results/traffic/traffic_adv_*.npz |
3 seeds; also prints Table 2 |
| Traffic advantage metrics | traffic/plot_traffic_errbars.py |
results/traffic/traffic_adv_*.npz |
Printed to stdout |
| Traffic ratio study (2×3) | traffic/plot_traffic_ratio.py |
results/traffic/traffic_ratio_*.npz |
Two advantage baselines; also prints Table 3 |
| Traffic ratio metrics | traffic/plot_traffic_ratio.py |
results/traffic/traffic_ratio_*.npz |
Printed to stdout |
The four coupling families are: directed_dilemma, dense_pairwise, local_congestion, block_community.
# Extract pre-computed results
unzip results.zip # → results/toy/, results/traffic/
# Install dependencies (figures require only NumPy and Matplotlib)
pip install numpy matplotlib
# Toy experiments
python toy/single_step/make_biasvar2x2.py
DATA=results/toy python toy/plots/plot_advsweep.py
DATA=results/toy python toy/plots/plot_fig4adv.py
DATA=results/toy python toy/plots/plot_fig4.py
DATA=results/toy python toy/plots/plot_fig5scaling.py
# Traffic experiments
DATA=results/traffic python traffic/plot_traffic_errbars.py
DATA=results/traffic python traffic/plot_traffic_ratio.py All figures are saved to figures/. Each plotting script reads its data directory from the DATA environment variable (defaults are specified at the top of each script), so you can easily point to your own experimental runs.
The PyTorch package under toy/training/pkg/ generates all sweep data.
cd toy/training/pkg
pip install torch numpy # + matplotlib for built-in plotters
# Advantage sweep — one per family
python run_experiment.py --exp advantage_sweep --env directed_dilemma --seeds 5
python run_experiment.py --exp advantage_sweep --env dense_pairwise --seeds 5
python run_experiment.py --exp advantage_sweep --env local_congestion --seeds 5
python run_experiment.py --exp advantage_sweep --env block_community --seeds 5
# Ratio sweep
python run_experiment.py --exp ratio_sweep --env directed_dilemma --seeds 5
# ... repeat for each family
# Agent-count scaling — standard and stress configs
python run_experiment.py --exp scaling --env dense_pairwise --ns "10,20,40,80,160,320" --seeds 5
# ... repeat per family; use --lr 0.15 --epochs 10 --batch 16 --tag _stress for stress configOutput NPZ files go to pkg/results/ — move them to results/toy/ for the plotting scripts. See toy/training/pkg/README.md for the complete argument list.
Note:
local_congestionandblock_communityratio/scaling studies were generated with the Torch-freetoy/training/np_runner.py(outputs:npret_*.npz,npscale2_*.npz). The plotting scripts merge these with server runs.
Requires sumo-rl, torch, and a SUMO installation. Each condition runs as a separate invocation (SUMO is computationally intensive; parallelize across processes/GPUs).
cd traffic
python probe_env.py # Builds coupling_adj.npy (road adjacency)
# Advantage study (fix per-agent ratio, vary advantage radius)
bash run_step1.sh # or run_step1.bat on Windows
# → traffic_adv_advR{0,1,2,joint}_s{0,1,2}.npz
# Ratio study with two advantage baselines
bash run_step2.sh # 1-hop neighborhood advantage fixed
./run_ratio_adv0.bat # Independent advantage fixed
# → traffic_ratio_rat{Indep,Nbr,Nbr2,Joint}[_A0]_s{0,1,2}.npzMove resulting NPZ files to results/traffic/. Stress-configuration files are not used in the current paper and are omitted from the packed results.
run_traffic.py usage:
--exp {adv,ratio} --adv_spec {0,1,2,joint} --ratio_spec {indep,1,2,joint} --seed S --tag NAME
adv_spec/ratio_spec: k-hop radius (orjoint= all 196 agents)--tag: output filename suffix