gptntPlays: An HTTP Control Interface for Keep Talking and Nobody Explodes

gptntPlays is a Unity mod that exposes Keep Talking and Nobody Explodes (KTANE) as a programmatic environment for AI agents and research. It adds an HTTP server to the game so an external process (a model, a controller, a benchmark harness) can start missions, observe the bomb, send mouse/rotation actions, and step the simulation deterministically.

The mod is built on top of the TwitchPlays mod by samfun123 and others; the original Twitch-chat command interface has been removed and replaced with the HTTP control surface described below.

This repository accompanies a research paper. If you are looking for the code, builds, or want to reproduce experiments, this README is the primary entry point. The bulk of new code lives under TwitchPlaysAssembly/.

• Overview
• Architecture
• Installation
• Configuration
• HTTP API
  • Lifecycle
  • Observation
  • Actions
  • Time control
  • Utilities and debugging
• Game state schema
• Supported modules
• Coordinate system
• Game-state machine
• Tracing and logging
• Building the mod
• Repository layout
• Citation
• Acknowledgements and license

KTANE is a real-time, partially observable, mouse-driven puzzle game in which a player must defuse a procedurally generated bomb made up of independent puzzle modules. The game is well suited as a benchmark for embodied / GUI-driven agents because:

• Each mission is seeded and reproducible.
• Each module is a small, self-contained reasoning task (sequences, mazes, ciphers, perception under noise) that can be evaluated independently.
• The action space is low-level: 2D screen-space clicks, holds, releases, and bomb rotations — the same affordances available to a human player.
• Difficulty scales naturally by adding modules, strikes, time pressure, and needy modules.

gptntPlays makes this environment scriptable. It boots a standard KTANE install, exposes a local HTTP server on port 8085, and lets a controller:

1. Start a mission with a chosen seed and module list.
2. Read a structured JSON description of the current bomb (the state), including per-module ground-truth attributes.
3. Read pixel observations (RGB screenshots plus segmentation masks of interactable elements).
4. Issue actions (click, hold, release, rotate the bomb, zoom out).
5. Pause, resume, and time-step the simulation so agents that are slower than real time still have a fair evaluation.

The HTTP server runs on a worker thread; all requests that touch Unity objects are marshalled back to the main thread, so handlers are safe to call concurrently from an external runner.

The new mod code lives in TwitchPlaysAssembly/Src/ and is organised around a small set of MonoBehaviour components attached to a single mod host object:

The HTTP server only binds to localhost. The mod is designed to run alongside a controller on the same machine (typically inside a single container) — not to be exposed on a network.

You need a copy of KTANE. The game loads mods from a mods/ directory that sits next to the game executable.

We have only tested against the Humble Store build of KTANE; the Steam build probably works but has not been validated by us. Any storefront that ships the standard Unity build of the game should be fine.

1. Install KTANE, then create an empty mods/ directory next to the game executable. The install directory should look like:

   <KTANE install>/
   ├── Keep Talking and Nobody Explodes.app   # or the .exe / Linux binary
   └── mods/
   

2. Download the latest prebuilt assembly from the build/ directory of the upstream repo.

3. Drop the built GptntPlays folder into the game's mods/ directory, so the path looks like <KTANE install>/mods/GptntPlays/.

4. Launch KTANE. The HTTP server should bind to http://localhost:8085/ on startup. Confirm with:

   curl http://localhost:8085/health
   # → Setup

If you prefer to build from source, see Building the mod.

The mod reads the following environment variables at startup:

Observation frames in /buffer are PNG-encoded at this resolution; the ring buffer holds the most recent 16 frames captured at one frame every 0.25 in-game seconds (see GptntGameHost).

All endpoints accept GET with query-string parameters and return either text/plain or application/json. CORS is enabled (Access-Control-Allow-Origin: *) and traceparent / tracestate headers are accepted for distributed tracing.

Responses use standard status codes:

• 200 OK — handled successfully.
• 400 Bad Request — request issued in a game state that does not permit it, or invalid parameters.
• 408 Request Timeout — segmentation render did not complete within 500 ms.
• 500 Internal Server Error — handler threw.

Starts a new mission. Only valid from Setup.

Example:

/startmission?seed=1&timeLimit=300&numStrikes=3&needyTime=90&isFront=true&optWidgets=5&components=Venn&timeScale=1.0&timeStepSize=250

Returns the seed on success.

Returns the game to the setup screen from any non-setup state. Used between episodes.

Returns the current game state as a single token: one of Gameplay, Setup, LightsOn, LightsOff, Transitioning, PostGame. Excluded from request logs so it is safe to poll.

Force-detonates the bomb. Only valid in LightsOn. Intended for terminating failed episodes.

Force-solves every remaining module. Only valid in LightsOn. Intended as an upper-bound oracle and for tooling tests.

Returns the full bomb state as JSON. Only available once the bomb has reached its first lights-on and not after reset. See Game state schema.

Returns the most recent buffered RGB frames plus a single fresh segmentation mask as a packed binary payload (Content-Type: application/octet-stream). The wire format avoids the cost of PNG-encoding + base64 + JSON parsing on the hot observation path.

Layout (little-endian, written via System.IO.BinaryWriter):

Each frame is exactly frameWidth * frameHeight * 3 bytes; GPU row padding from Texture2D.GetRawTextureData() is stripped before serialisation. The ring buffer holds up to 16 frames captured at one frame every 0.25 in-game seconds. If the segmentation render does not complete within 500 ms the response returns 408 Request Timeout and no body is written.

The segmentation mask colours all currently active interactable Selectables on a dedicated render layer; pixel colour identifies which selectable a pixel belongs to.

Legacy JSON form of the same payload, kept for backwards compatibility with older controllers and notebooks:

{
  "screenshot": ["<base64 PNG>", "<base64 PNG>", ...],
  "segmentation": "<base64 PNG>"
}

Prefer /buffer for new code — /old-buffer is materially slower because it PNG-encodes and base64-wraps every frame.

The single action endpoint. Behaviour is controlled by the action query parameter. Only valid during LightsOn / LightsOff once the mission has actually started; otherwise 400.

x_pos and y_pos are floats in [0, 1]. See Coordinate system.

These endpoints let an external controller decouple game time from wall-clock time so that slow agents can be evaluated under the same effective time pressure as fast ones.

Sets Time.timeScale. 0 pauses the game. If a pause is requested during a Transitioning state, the mod waits for the transition to end before applying the pause (transitions never play correctly under timeScale=0).

Sets the duration of one step, in in-game milliseconds.

Synchronously advances the game by exactly one setstepunit and pauses it again before returning. The handler:

1. Sets Time.timeScale = 1 and waits setstepunit ms.
2. Waits until the game is no longer in a Transitioning state.
3. Waits until no module is mid-emerge (bombState.isEmerging == false), so that the next observation reads a quiescent bomb.
4. Sets Time.timeScale = 0 and returns.

Each of the three waits has a 10 s wall-clock timeout; if any times out the response is 408 Request Timeout with a message indicating which phase stalled. The typical control loop is: /settimescale?value=0 once at episode start, then alternate /state (and/or /buffer) ↔ /action ↔ /timestep.

GET /state serialises a BombState:

{
  "seed": 1,
  "maxStrikes": 3,
  "isDetonated": false,
  "isSolved": false,
  "isLightOn": true,
  "bombSide": "front",      // "front" or "back"
  "isEmerging": false,      // true while any module's interactables are mid-emerge animation
  "timerModule": { "secondsRemaining": 287.4, "onFront": true, "index": 0, "name": "Timer" },
  "widgets":  [ /* BatteryWidgetState | IndicatorWidgetState | PortWidgetState | SerialNumberWidgetState */ ],
  "modules":  [ /* one entry per solvable module, see below */ ],
  "strikes":  [ "ButtonModuleState: wrong release", ... ]
}

Every module entry inherits from SolvableModuleState and includes:

• name — module identifier (e.g. BigButton, Keypad, Wires).
• onFront, index — anchor position on the bomb. index is the closest anchor slot on the chosen face.
• isSolved, inFocus — solve state and whether this module is currently zoomed in on.
• Module-specific fields (the wire array for Wires/Venn/WireSequence, the maze coordinates for Maze, etc.).

Widgets carry their bomb-face position (front / back / top / bottom / left / right) and per-widget attributes such as battery type and count, indicator label and lit state, port set, or serial-number string.

The schema is intentionally exposed as ground truth: it bypasses what a human or vision-only agent could read from the screen. For vision-only experiments, use /buffer.

StateClasses.cs defines structured state for the following stock vanilla modules:

• The Button (BigButton)
• Keypad
• Simon Says (Simon)
• Wires (WireSet)
• Complicated Wires (Venn)
• Wire Sequence
• Maze (Invisible Walls)
• Memory
• Morse Code
• Password
• Who's On First

The timer and the four widget types (battery, indicator, port, serial number) are also fully serialised.

Any modded module can be spawned via /startmission using its string module ID (discoverable at runtime via GET /modules). Modded modules not listed above will appear in /state with basic solve and focus tracking via a generic fallback state class. To add structured, module-specific attributes to /state for a modded module, see docs/adding-modded-modules.md.

x_pos and y_pos are normalised screen coordinates in [0, 1]:

• x = 0 is the left edge of the rendered frame, x = 1 the right.
• y = 0 is the top of the frame, y = 1 the bottom. (Internally the mod converts to Unity's bottom-origin convention, so this matches how images returned by /buffer are oriented.)

A click ray is cast from the main camera through that point against the interactable layer. If the ray misses, the click is a no-op; otherwise it triggers the corresponding Selectable. See GptntActions.Click.

Reported by /health and used to gate other endpoints:

OnFirstLightsOn (the moment the bomb is first picked up and the lights come up) enables /state. OnReset disables it again.

Every HTTP request creates an OpenTelemetry span via OpenTelemetrySpan, exporting OTLP/JSON to http://localhost:4318/v1/traces (the default OTLP HTTP collector endpoint). Long-running handlers create child spans (game.action.mainthread, game.action.clickstart, observation.buffer, buffer.start, …) so action latency and main-thread queueing can be measured.

The mod honours W3C traceparent headers on incoming requests, so a controller that is already inside a trace can attach the game-side spans to the same trace.

Logs are written through log4net and tagged with trace_id / span_id properties during a request, via GptntDebug.FormatMessage.

If you do not run a collector, requests still succeed; the OTLP export call simply fails silently per span.

The mod is a Unity-side C# assembly that targets the same KTANE managed runtime as TwitchPlays — net35, Unity's Mono profile for this version of the game. The general flow follows the upstream TwitchPlays build instructions:

1. Build TwitchPlaysAssembly/TwitchPlaysAssembly.csproj from your IDE.
2. In Unity, wait for the editor to pick up the new scripts, then build the mod.
3. Drop the built mod into the game's mods/ directory and restart KTANE.

You do not need the Steam version of KTANE — the Humble Store build works fine and, in our experience, was less awkward to point the build at than the Steam install (your mileage with Steam may vary).

The committed TwitchPlaysAssembly.csproj targets a Windows install by default; override GameFolder (in the csproj) if your install lives elsewhere.

Two macOS-specific gotchas:

1. IDE. Visual Studio for Mac has been discontinued by Microsoft, so the build requires an older installer. We are currently building with Visual Studio for Mac 17.6.14 (build 413); newer "Visual Studio Code + C# Dev Kit" setups have not been validated.

2. Paths. The Mac KTANE bundle has a different managed-DLL layout than the Windows install. On macOS the managed directory is:

<wherever KTANE lives>/Keep Talking and Nobody Explodes.app/Contents/Resources/Data/Managed

— there is no ktane_Data\Managed\ segment. So Mac users need a one-time patch to the csproj before building:

• Set GameFolder to the full .../Keep Talking and Nobody Explodes.app/Contents/Resources/Data/Managed directory.
• Strip the \ktane_Data\Managed\ segment from every <HintPath> that references it, so each path becomes $(GameFolder)/<Dll>.dll. The affected references are Assembly-CSharp, every UnityEngine.*Module, UnityEngine.UI, and log4net.
• Update the post-build Exec command's destination from $(GameFolder)mods\Twitch Plays\… to $(GameFolder)/../../../../mods/Twitch Plays/… (or copy the DLL manually after each build) — the mods/ directory lives next to the .app, not inside Managed/.

A small sed does most of the work on a fresh checkout:

sed -i '' 's|\\ktane_Data\\Managed\\|/|g' TwitchPlaysAssembly/TwitchPlaysAssembly.csproj

We chose not to commit the Mac-friendly variant as the default because the upstream TwitchPlays csproj — and the majority of likely contributors — target Windows. Keep these edits in your working tree (or behind an MSBuild Condition); do not push them as the project default.

1. Edit C# sources under TwitchPlaysAssembly/.
2. Build the project in Visual Studio — the DLL is copied into the game's mods/Twitch Plays/ folder by the post-build step.
3. Open the Unity project once and wait for the editor to recompile / refresh.
4. Build the Unity mod, move the output into the game's mods/ directory.
5. Restart KTANE to pick up the new mod.

.
├── Assets/                  # Unity project assets (untouched fork content)
├── Packages/                # Unity package manifest
├── ProjectSettings/         # Unity project settings
├── TwitchPlaysAssembly/     # The mod assembly (all gptntPlays code lives here)
│   ├── RequestHandlers.cs   # HTTP route handlers
│   ├── Src/
│   │   ├── Gptnt*.cs        # HTTP server, state, actions, buffer, host, debug
│   │   ├── StateClasses.cs  # JSON-serialisable bomb/module/widget state
│   │   ├── Segmentation.cs  # Per-selectable instance segmentation pass
│   │   ├── MagicSolver.cs   # Oracle baseline controller
│   │   ├── OpenTelemetrySpan.cs
│   │   └── ...              # Inherited TwitchPlays sources kept for compatibility
│   └── TwitchPlaysAssembly.{csproj,sln}
├── docs/                    # Generated docs (Doxygen output)
└── README.md                # This file

The TwitchPlaysAssembly directory contains both the new code added for this project and the original TwitchPlays sources it inherits from. The new control surface is contained in the Gptnt* files, RequestHandlers.cs, StateClasses.cs, Segmentation.cs, MagicSolver.cs, and OpenTelemetrySpan.cs.

If you use gptntPlays in academic work, please cite the accompanying paper. A BibTeX entry will be added here once the paper has a public preprint / DOI.

This project is a fork of and extension to KtaneTwitchPlays by samfun123 and contributors. We are grateful for their work, which made the input plumbing, module abstractions, and mission scaffolding used here possible. The original repository's license terms (see LICENSE) carry over to this fork.

Keep Talking and Nobody Explodes is © Steel Crate Games. This mod is an unofficial third-party modification and is not affiliated with or endorsed by Steel Crate Games.