mudra — control Onshape with your hands

MediaPipe hand tracking (in your browser) → gesture engine → WebSocket → Node server → trusted CDP mouse input into a dedicated Chrome window running Onshape (or a local testbed with identical mouse bindings).

Design + milestones: see PLAN.md. Prompt that produced the plan: onshape-hand-control-prompt.md.

cd mudra
npm install
node server.mjs              # launches the driver Chrome window on the TESTBED
# then open http://localhost:8787/ in YOUR browser and allow the camera

Two windows side by side: the gesture console (webcam + skeleton overlay + live control values) and the driver window (the 3D scene being controlled; the red ring is the virtual pointer).

Keep the console tab visible and foreground — browsers throttle background tabs, which would starve the control stream (the watchdog would then release everything, by design).

node server.mjs --target onshape

Log into Onshape in the driver window (the login persists in .chrome-profile/) and open a document. Same gestures, same code. If your Onshape account remaps the mouse, edit bindings in config.mjs.

The console also has quick-action buttons (fit / front / top / right / iso / undo) that inject Onshape's default keyboard shortcuts — the exact keystrokes live in config.mjs (keys). If a view button does the wrong thing, check your shortcuts in Onshape (press Shift+/ in a document) and adjust config.mjs to match.

Resting open hands do nothing: every control has an engage/release latch with hysteresis, a per-frame dead zone, and re-zeroes at the moment of engagement. Orbit and zoom also arm on movement — the mouse button is pressed only once the fist actually travels, so a misdetected fist can never fire a phantom right-click (which Onshape would turn into a context menu).

• Watchdog: if the control stream goes silent >600 ms while a mouse button is held (console crash, tab closed), the server releases it.
• Hand lost mid-gesture → the engine force-releases (never a stuck drag).
• Kill switch releases everything immediately.

npm test          # engine unit tests — synthesized landmark replay
npm run smoke     # E2E: synthetic controls -> CDP -> testbed, asserts
                  # azimuth/zoom/hover/selection changed + watchdog works
                  # (needs `node server.mjs` running)

Debug endpoints while the server runs: /health (held buttons, viewport), /state (testbed camera + selection), /screenshot (PNG of the driver page).

• Gains and bindings: config.mjs (orbitPxPerUnit, zoomWheelPerUnit, invert flags).
• Gesture thresholds: ENGINE_DEFAULTS in public/gesture-engine.mjs (all engage/release pairs).
• The console's record button saves the raw landmark stream as JSONL — replay it through createEngine() in a test to tune thresholds against a real session instead of live hands.