wibbly

Your camera is the controller.
Camera-gesture games for magnetite — a gesture is an input event like any other.
Frames never leave the device. No install, no download, no depth sensor, no controller.

Vulos — rooted in vula, the Zulu and Xhosa word for open.

Status: early. Most of this repo is a plan, not a product.

This README will not pretend otherwise. What plays today is one game — single-camera tennis — driven by one gesture (swing) from one player. The input library is real and unit-tested without a camera. Hand tracking (pinch, point) is built as a standalone, unit-tested library, but nothing composes it into the running pipeline or a game yet. Palmworks is folded in with its full history and builds, but nothing drives it with gestures yet.

Read WIBBLY.md for the full spec and backlog. Read the status table below for what is real.

wibbly is a game for magnetite — the decentralized, self-hostable Rust games platform. Magnetite is the platform; wibbly is what it looks like to build on it when your controller is a webcam.

The thesis is narrow and testable: a webcam is a controller, a GestureEvent is an input event like a keypress, and a game should never name a model, a runtime, or a vendor to receive one. Games run in a browser tab with zero install. Pose inference runs locally on the machine holding the camera, so the wire carries gesture events — not video.

What comes from magnetite: a real magnetite AuthoritativeGame — the reference arena-shooter, compiled to wasm — runs client-side in the browser tab via @vulos/wibbly-authority, stepped from the tennis render loop as a Topology::SingleRoom match: the bottom rung of magnetite's own topology ladder, hosted by the tab with no server. That is the concrete, current sense in which wibbly is built on magnetite. It changes nothing about camera gestures, which magnetite types InputClass::Attested and are, by construction, never replay-verifiable — determinism is a property of the simulation given its inputs, not of the inputs themselves. That boundary is enforced by magnetite, in code, on purpose. Nothing here claims otherwise.

What stays here: the camera. packages/wibbly-input is a standalone, vendor-neutral library with no magnetite dependency, so anything can consume it — magnetite ships the contract, not a pose pipeline.

The games live in games/ — see that README to submit one.

Spec · Quick start · Demo mode · Architecture · Model selection · Docs

Audited against the tree, not against the pitch.

Generated by npm run screenshots against a real build. Chromium runs with a synthetic camera (--use-fake-device-for-media-stream) — a rolling test pattern, not a person — so no skeleton is detected and no gesture fires in any shot. Nothing here is composited or staged.

Title — game selection; soccer and boxing show as planned, not playable	Setup — the camera is explained before the browser prompt fires
Setup — handedness, driven by the real Calibration API	Tennis (/play) — court, players, camera rig
In-game menu — salvaged from the CAMERA branch, now wired	In-game menu — settings
Mini-site — dark	Mini-site — light

Also captured: in-game-menu-camera.png, not-found.png. See docs/screenshots/README.md.

One caveat, stated plainly. In headless Chromium under SwiftShader, TensorFlow.js cannot initialize a WebGL or WebGPU backend and falls back to its CPU backend, which takes ~10-12s to come up. The pipeline does therefore run — the setup screenshots show a live camera feed — but Chromium's synthetic stream is a rolling test pattern, not a person, so no skeleton is drawn and checkFraming() correctly reports "No one detected". That is a real verdict from the real function, not a placeholder. Regenerate with npm run screenshots -- --headed on a machine with a GPU, and a person in frame, to see tracking succeed.

git clone https://github.com/vul-os/wibbly
cd wibbly
npm install
npm run dev            # http://localhost:5173

Then open /play, allow camera access, stand about two metres back with your upper body in frame, and swing. One player, one gesture — the honest extent of it. The spacebar works as a fallback and the game stays fully playable if the camera is denied or the model fails to load, which is the correct degradation.

npm run build          # production bundle → dist/
npm run preview        # serve dist/
npm test               # 338 unit tests (input seams + wibbly-p2p + wibbly-authority + app), no camera required
npm run typecheck
npm run screenshots    # requires: npm i -D playwright && npx playwright install chromium

npm run build:demo     # → dist-demo/, based at /products/magnetite/wibbly/play/
npm run verify:demo    # builds it, serves it at that sub-path under the REAL
                       # production CSP, and drives it with Chromium
npm run verify:model   # re-check the vendored weights against their manifest

build:demo produces a self-contained single-surface build for embedding in a same-origin iframe. See Demo mode below.

The MoveNet MultiPose Lightning weights are vendored into this repo at public/models/movenet-multipose-lightning/ (~9.3 MiB, Apache-2.0, provenance and hashes in its README) and served same-origin. This is the default in both builds.

That is not a preference, it is a requirement: the demo is embedded on a page served under connect-src 'self', where a fetch to tfhub.dev is refused outright and the detector would never initialise — silently, with no error and no 404, just a game that ignores you.

To skip the vendored weights and pull from TF Hub at runtime instead:

VITE_WIBBLY_MODEL=cdn npm run build

Demo builds reject that setting rather than producing a build that cannot work.

VITE_WIBBLY_MODE=demo builds a different shell, not the app with bits hidden. It exists for one deployment — a same-origin iframe on the marketing site — and that context has constraints the normal app does not.

npm run dev:demo       # http://localhost:5173
npm run build:demo     # → dist-demo/, base /products/magnetite/wibbly/play/
npm run verify:demo    # the proof, below

Either build can be flipped at runtime by the host page with window.__WIBBLY_MODE__ = 'demo' | 'full', which follows the same pattern magnetiteConfig() already used. All of it resolves in src/mode.js through pure functions, so the gating is unit-tested rather than trusted.

What demo mode does not do. It does not skip the camera explanation. A cold getUserMedia prompt with no context is the worst moment in this UX, and inside an iframe it is worse: the prompt names the embedding origin, so the visitor sees a permission dialog from a site they were reading rather than the game they were looking at. Explain, then ask — the handedness pick shares that card so it costs no extra step.

It also does not hide what this is. There is a permanent "Demo" marker, a link to this repo, and a line stating that this is one game driven by one gesture from one player against a local AI.

After a few swings, a dismissible panel appears in the corner. It never blocks play, and it says this:

In progress, plainly: wibbly multiplayer does not work yet. The transport side is built and tested on its own — the peer session, the WebRTC offer/answer handshake, a compact code for putting a connection in a link — with no code path that touches a server anywhere in it. What is missing is a lobby: nothing in wibbly turns that into a screen you can actually use, so there is no invite link and nothing to click yet.

That warning is the point of the panel, not a disclaimer bolted to it. The pitch above it is only that solo tennis needed no server, and that a second player is designed as peer-to-peer — one browser tab holds authority, a WebRTC DataChannel carries the other player's gesture events, opened by a one-time link or QR code exchange, no server for anyone to run or rent. packages/wibbly-p2p has no magnetite dependency; this panel has nothing to do with the magnetite wasm authority described above — that link runs entirely in the full app (never in this demo embed, since its CSP blocks wasm) and has no bearing on whether a second player can join.

It builds the demo, serves dist-demo/ under /products/magnetite/wibbly/play/ (not at the root, and not through vite preview, which knows the base and would hide a base mistake), applies the real production CSP copied verbatim from vulos-management/pkg/cproutes/spa.go, and drives it with Chromium using a synthetic camera. 26 checks, all passing:

• Zero external network requests. Every request in the run, in full:

  /products/magnetite/wibbly/play/
  /products/magnetite/wibbly/play/assets/index-*.js
  /products/magnetite/wibbly/play/assets/index-*.css
  /products/magnetite/wibbly/play/assets/pose-detection.esm-*.js
  /products/magnetite/wibbly/play/assets/shared-*.js
  /products/magnetite/wibbly/play/models/court.glb
  /products/magnetite/wibbly/play/models/TennisCourt_BaseColor.png
  /products/magnetite/wibbly/play/models/movenet-multipose-lightning/model.json
  /products/magnetite/wibbly/play/models/movenet-multipose-lightning/group1-shard{1,2,3}of3.bin
  

• the detector initialises from the local model, on the WebGL backend

• no CSP violations with any consequence, and no 4xx

• no localStorage key and no cookie survives the page

• the spacebar path still produces swings with the camera denied

• it renders in a 420px-wide iframe with no horizontal overflow

Two real bugs were found by running this rather than assuming it: court.glb was loaded from a hardcoded /models/court.glb that 404s under any sub-path, and its 2 MB base-colour texture was embedded in the .glb, so GLTFLoader loaded it through a blob: URL that connect-src 'self' refuses — the court rendered untextured in production with no error and no failed request. Both are fixed; the texture is now an external same-origin file (scripts/externalize-glb-textures.mjs), which also shrank court.glb from 2,086,144 to 26,452 bytes.

All input lives in packages/wibbly-input. Game code names only these interfaces — never a model, a runtime, or a vendor. Swapping MoveNet for something else is a constructor argument, not a rewrite. Every seam ships a working default.

camera ──► FrameSource ──► PoseTracker ──► PlayerBinder ──► GestureRecognizer ──► game
           (pixels)        (skeletons)     (stable ids)     (events)
                                    ▲
                              Calibration
                        (handedness, reach, framing)

FrameSource — where pixels come from. Default WebcamFrameSource wraps getUserMedia and owns its own capture loop. A Rust-side NativeFrameSource is specified for the Tauri phase; frames never cross IPC there, only landmarks do.

PoseTracker — pixels to skeletons. estimate() returns Person[]; the plural is the whole point. The default is MoveNetMultiPoseTracker. A composable hand tracker is specified and not built.

PlayerBinder — which skeleton is which player. MoveNet returns up to six people per frame with no stable identity across them, so a binder assigns durable PlayerIds. SpatialBinder does greedy nearest-centroid matching with per-player claim zones for the couch case. This is the highest-risk component in the project and its tests are synthetic — occlusion, a player leaving and returning, and two players crossing over are handled in code but have not been validated against real bodies.

GestureRecognizer — skeletons to game events. Recognizers are pure functions over landmark history, which is what makes them unit-testable without a camera — the old poseDetection.js was not. SwingRecognizer is the only implementation. It emits handedness-relative strokes (forehand/backhand) rather than raw image-space directions, so a left-handed player's forehand behaves like a forehand instead of mirroring into the wrong shot.

Calibration — per-player handedness, reach envelope, camera-framing checks and lighting warnings, persisted locally and keyed to PlayerId. Reach widens continuously during play rather than being captured in a setup wizard.

A GestureEvent carries a playerId, a kind, a confidence in 0..1 that games must handle, an optional vector, and tCapture — the capture timestamp, not the detection timestamp.

Chosen after a licensing and benchmark review, written down in WIBBLY.md §4 so nobody relitigates it.

Body — MoveNet MultiPose Lightning (TF.js, Apache 2.0). Up to six people, and TF.js documents that person count does not affect inference speed — a flat cost curve, which is exactly the property a 2–4 player couch game needs. It is the only multi-person model with published in-browser numbers: 54 FPS on a MacBook Pro 15", 62 on a desktop i9-10900K, 24 on an iPhone 12, all WebGL. Vendor-published, but browser-real.

Hands — MediaPipe HandLandmarker (Apache 2.0). 21 landmarks per hand, multi-hand, first-class web support, and realistically the only browser option. Its eight canned gestures are too thin for games, so classification belongs in our own recognizers over raw landmarks. Planned — not built.

The licensing traps here are the useful part.

One runtime correction worth knowing: MediaPipe Tasks Web's "GPU" delegate is WebGL, not WebGPU — WebGPU for vision tasks is still an open upstream feature request. Reaching WebGPU means ONNX Runtime Web. The MoveNet numbers above are WebGL.

v1 is browser-first, and that is not a compromise. Zero install is the platform's single biggest asset — a link, a wave, a game. For seeding a game library from nothing it beats anything native offers.

Tauri-with-webview-ML was researched and rejected. WebKitGTK has no WebGPU and, per a WebKit developer, nobody is working on it; WKWebView on macOS 26 is unconfirmed and must be tested empirically. Distro WebKitGTK ships without WebRTC, so getUserMedia on Linux requires compiling WebKitGTK yourself, X11-only. macOS camera-permission bugs are open upstream (wry#1195, tauri#11951). And you cannot escape it by shipping frames to Rust: Tauri IPC is JSON-serialized, a 1080p RGBA frame is ~8MB, and 10MB has been measured at ~200ms on Windows.

Tauri as an app shell is still right — phase 2, with capture and inference in Rust. nokhwa for capture via V4L2, ort for inference with CoreML/DirectML execution providers, a native wgpu preview surface under a transparent webview, and only landmarks crossing IPC — kilobytes at 30Hz. That path also unlocks RTMO, which the browser cannot reach. None of it exists in this repo yet.

Two distinct problems that should not be conflated.

Local (same camera, 2–4 players) is PoseTracker.maxPeople plus PlayerBinder. MoveNet's flat cost curve makes it cheap, and it is the differentiated case. The plumbing is in place; two humans have not tested it.

Networked (different cameras) has each client run its own tracker and transmit GestureEvents, not video — camera frames never leave the device. Sessions and discovery are peer-to-peer (packages/wibbly-p2p, WebRTC, no magnetite dependency and no server anywhere) — see Relationship to magnetite below for the separate, real magnetite link, which is unrelated to this transport.

Be clear about what anti-cheat can and cannot promise here. Magnetite's replay verification assumes deterministic input. Camera gesture input is a noisy, nondeterministic sensor stream — it is the one input class that cannot be replay-verified. Gesture games therefore run client-attested: the host simulates authoritatively over received events, and events are rate-limited and plausibility-checked against human-reachable velocities and cooldowns, but a determined cheater can synthesise them. That is a real limitation, documented rather than papered over — and it is also the reason wibbly stays its own repo instead of merging into magnetite and blurring the property magnetite sells.

wibbly is a client of magnetite, not a fork of it — and, as of @vulos/wibbly-authority, a real one: public/magnetite/arena-authority.wasm is magnetite's own reference AuthoritativeGame, compiled to wasm32-unknown-unknown, running in the browser tab as a Topology::SingleRoom match (the bottom rung of magnetite's topology ladder, hosted client-side with no server), stepped once per tennis frame by src/game/magnetite-authority.js. That module is what makes "wibbly is built on magnetite" literally true, not just a stated intention.

This does not change anything about the anti-cheat boundary. magnetite is Rust and sells deterministic authoritative simulation with replay verification for InputClass::Deterministic input; camera gestures are typed InputClass::Attested and are, by construction, never replay-verifiable — a property of the input, not of where the authority runs. Running a real magnetite simulation in the tab does not verify a swing was real; it means the same game module a dedicated or sharded magnetite host would run, runs here instead. See Multiplayer & anti-cheat for the full argument.

This is unrelated to wibbly's own networked multiplayer: packages/wibbly-p2p is pure WebRTC peer-to-peer with no magnetite dependency at all — see its README for why the earlier magnetite-node design was retired in favor of it.

packages/wibbly-input/     the seams — TypeScript library, no DOM injection, 221 tests
packages/wibbly-p2p/       peer-to-peer multiplayer transport (WebRTC, no magnetite dependency), 70 tests
packages/wibbly-authority/ the real magnetite link — AuthoritativeGame compiled to wasm, run client-side
src/game/                tennis: court, ball physics, player rig, AI opponent, camera rig
src/pages/               title, setup, play, 404 — the whole app shell
site/                    house-style static mini-site + docs (the honest surface)
scripts/screenshots.mjs  Playwright screenshotter
WIBBLY.md                the spec and the program backlog

• WIBBLY.md — the spec: vision, seams, model research, runtime targets, phased backlog, open questions
• site/docs/ — overview, architecture, models, runtime targets, multiplayer, incentives, roadmap, getting started
• docs/screenshots/README.md — what each screenshot shows and how it was captured

MIT OR Apache-2.0 — © VulOS. wibbly is a VulOS project; source and issues at github.com/vul-os/wibbly. This covers the whole repository, including packages/wibbly-input and the tennis reference game.

The models wibbly loads are separately licensed, and both are permissive: MoveNet ships under Apache-2.0 via tfjs-models, and MediaPipe under Apache-2.0. The MoveNet weights are vendored into this repo under public/models/movenet-multipose-lightning/, which carries its own LICENSE (Apache-2.0, Google LLC) and provenance — those files are not covered by this repo's licence. That was a selection criterion, not luck — see Model selection for the alternatives ruled out on licensing alone.

_{vulos — open by design}