An AI agent that plays The Legend of Zelda: A Link to the Past (SNES) through RetroArch using vision analysis and input simulation.
┌─────────────────────────────────────────────────────────┐
│ Zelda Player Agent │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Observe │──▶│ Orient │──▶│ Decide │ │
│ │ (vision) │ │ (LLM) │ │ (LLM) │ │
│ └──────────┘ └──────────┘ └────┬─────┘ │
│ ▲ │ │
│ │ ┌──────────┐ ▼ │
│ │ │ Verify │◀─── ┌──────────┐ │
│ └────────┘ │ │ Act │ │
│ (vision) │ │ (inputs) │ │
│ └──────────┘ │
│ │
├─────────────────────────────────────────────────────────┤
│ RetroArch Bridge │
│ ┌─────────────┐ ┌──────────────┐ │
│ │ NCI (UDP) │ │ Keyboard │ │
│ │ port 55355 │ │ Simulation │ │
│ └─────────────┘ └──────────────┘ │
│ screenshots · save states · status · memory │
│ pause · frame advance · fast forward │
└─────────────────────────────────────────────────────────┘
| Feature | macOS | Linux |
|---|---|---|
| RetroArch NCI | ✅ | ✅ |
| Screenshots | ✅ (NCI) | ✅ (NCI) |
| Keyboard input | ✅ Quartz CGEvents | ✅ xdotool / ydotool |
| Window focus | ✅ AppleScript | ✅ xdotool / wmctrl |
Auto-detects RetroArch, core, and ROM paths across macOS and Linux. All paths can be overridden via environment variables or CLI flags.
- NCI (Network Control Interface): UDP commands on port 55355 for screenshots, save states, status queries, pause/frame-advance, and memory reads.
- Keyboard Simulation: Platform-specific input injection (Quartz on macOS, xdotool/ydotool on Linux).
- Observe — Take a screenshot via NCI, analyze with vision model
- Orient — Understand game state: location, HP, objective, obstacles
- Decide — Choose next actions (move, fight, interact, navigate menus)
- Act — Send keyboard inputs to RetroArch
- Verify — Screenshot after actions to confirm results
# Install cliclick (fallback input method)
brew install cliclick
# Install the package with macOS extras
uv pip install -e ".[macos]"# Install input tools
# Debian/Ubuntu: sudo apt install xdotool wmctrl
# Fedora: sudo dnf install xdotool wmctrl
# Arch: sudo pacman -S xdotool wmctrl
# Wayland (no X11): install ydotool instead
# Install the package
uv pip install -e .You need RetroArch with the Snes9X core and a Zelda ROM. The tool auto-detects paths but you can override:
export ZELDA_RA_BINARY=/path/to/retroarch
export ZELDA_RA_CONFIG=/path/to/retroarch.cfg
export ZELDA_SNES9X_CORE=/path/to/snes9x_libretro.so
export ZELDA_ROM="/path/to/zelda.smc"
export ZELDA_SCREENSHOT_DIR=/path/to/screenshotsOr pass via CLI flags:
zelda-player --ra-binary /path/to/retroarch --core /path/to/core.so --rom /path/to/rom.smc launch# Show what was detected
zelda-player config
# Check all paths and dependencies
zelda-player verify
# Enable NCI in retroarch.cfg
zelda-player enable-nci
# Launch RetroArch with Zelda loaded
zelda-player launch --windowed
# Query RetroArch status via NCI
zelda-player status
# Take a screenshot
zelda-player screenshot -o /tmp/shot.png
# Send a single command to RetroArch NCI
zelda-player cmd GET_STATUS
# Press a SNES button
zelda-player press start -d 0.1
# Start the autonomous play loop
zelda-player play --max-steps 10
# Show SNES → keyboard mapping
zelda-player map-keys| SNES Button | Keyboard Key |
|---|---|
| D-Pad Up/Down/Left/Right | ↑ ↓ ← → |
| A | x |
| B | z |
| X | s |
| Y | a |
| L | q |
| R | w |
| Select | Right Shift |
| Start | Enter |
Action commands (return responses): VERSION, GET_STATUS, GET_CONFIG_PARAM, READ_CORE_MEMORY, WRITE_CORE_MEMORY, LOAD_STATE_SLOT, SAVE_FILES, LOAD_FILES, LOAD_CORE, SCREENSHOT
State/button commands (no response): MENU_TOGGLE, PAUSE_TOGGLE, FRAMEADVANCE, FAST_FORWARD, RESET, QUIT, SAVE_STATE, LOAD_STATE, STATE_SLOT_PLUS, STATE_SLOT_MINUS
| Variable | Purpose |
|---|---|
ZELDA_RA_BINARY |
Override RetroArch executable path |
ZELDA_RA_CONFIG |
Override retroarch.cfg path |
ZELDA_RA_CORES |
Override cores directory |
ZELDA_SNES9X_CORE |
Override Snes9X core path |
ZELDA_ROM |
Override ROM path |
ZELDA_SCREENSHOT_DIR |
Override screenshot directory |