SilksongRL

A PPO agent that learns to beat the first Hollow Knight: Silksong boss (Moss Mother). Other bosses tbd

• Observation — pixels only. The policy sees an 84×84 grayscale frame (4 stacked for motion), captured from the game window. It learns to play from vision, nothing else.
• Reward / termination — ground-truth game state. A small BepInEx mod reads exact values (Hornet HP, silk, boss HP) from the running game and streams them to Python over UDP (127.0.0.1:51234, CSV hp,maxHealth,silk,bossHp).

Why hybrid: vision-based hit detection capped at ~30% precision — Hornet's own attack VFX is as bright as the boss's hit-flash, so pixels can't reliably tell "I hit the boss" from "I swung." Reading the exact boss-HP delta gives a clean, unfarmable offensive reward plus a real win signal (boss HP → 0), while the agent still learns purely from pixels.

Reward (per step):

reward = boss_damage * DMG_SCALE(0.3) + hp_delta * HP_SCALE(1.0)
       + WIN_BONUS(50) when the boss dies

Episode ends on Hornet death (HP 0) or boss death. A near-death guard (KILL_MAX_BOSS_HP) prevents the boss's scream/phase pauses — where its HP reads -1 at high HP — from false-firing a kill.

envs/
  capture.py       ScreenCapture: grab the game window (mss + pywin32)
  input.py         KeyboardController: send keystrokes (pynput)
  game_state.py    GameStateClient: receive ground-truth state over UDP  ← HP/silk/boss source
  bench.py         BenchDetector: template-match the bench (used for respawn confirmation)
  silksong_env.py  SilksongEnv: the gym Env — capture + input + reward + respawn
  hp.py            HPDetector: pixel HP reader — RETIRED (mod replaced it), kept as calibration tool
  hit.py           HitDetector: pixel hit-flash — RETIRED (mod replaced it)
training/
  train.py         PPO (CnnPolicy) + frame-stack + checkpoints + auto-resume
  play.py          deterministic eval, records obs video to videos/eval/
scripts/           calibration / recording tools: record_path, test_replay, test_bench,
                   test_hit, smoke_test_env
tests/             pytest suite (detectors, game_state parsing, env reward/state machine)
resources/
  templates/       bench match templates (bench_template_A.png is the active one)
  calibration/     reference screenshots used to calibrate detectors
  misc/            stray debug captures

The BepInEx plugin (SilksongRLExporter, a separate C# project, not in this repo) is the ground-truth source. It targets netstandard2.1 (Silksong is Mono) and reads PlayerData.instance.{health,maxHealth,silk} + the boss HealthManager.hp, sending them over UDP each frame.

• Stable-Baselines3 PPO, CnnPolicy (NatureCNN over 84×84×4); action space Discrete(14)
• gymnasium Env interface
• mss + pywin32 — screen capture
• pynput — keyboard input
• OpenCV — image processing
• PyTorch + CUDA — the net (RTX 5090 needs cu128 wheels)
• BepInEx 5 + a C# plugin — ground-truth state over UDP
• pytest — unit tests

Python side:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
# RTX 5090 / Blackwell needs CUDA 12.8 wheels; cu121 only supports up to sm_90
pip install --upgrade --force-reinstall torch torchvision --index-url https://download.pytorch.org/whl/cu128

Mod side (one-time):

1. Install BepInEx 5.x (x64) into the Silksong folder; launch once to bootstrap.
2. Build SilksongRLExporter (targets netstandard2.1); drop the DLL in BepInEx/plugins/.
3. Launch Silksong with the mod; confirm SilksongRL exporter loaded in BepInEx/LogOutput.log and that the UDP feed reports live values.

Prereqs every run: Silksong open with the mod, Hornet on the calibration bench, nothing else holding UDP 51234 (the env binds it), Silksong unobscured. All commands run from the repo root.

python -m training.train     # train (auto-resumes from the newest prior run by default)
python -m training.play      # watch a checkpoint play; records to videos/eval/
python -m pytest             # unit tests (no game needed)
tensorboard --logdir runs --host 0.0.0.0 --port 6006   # metrics (host 0.0.0.0 for LAN access)

Built as a solo learning project (with Claude's help). Current focus: ground-truth reward vs Moss Mother — watching per-episode boss_dmg (in monitor.csv) climb toward 120 as the agent learns offense. Two from-scratch runs plateaued at ~28 dmg/fight under DMG_SCALE=0.1; the current lever is raising DMG_SCALE (now 0.3) so aggression outweighs the HP it costs. Images are taken from 1440p resolution Currently, this works only with the closest bench to Moss Mother; I did not look for a teleport mod to restart the boss fight in the arena. There is a set of key actions that are executed to have hornet travel to the boss arena from the bench.