Dexterity Lab

A local browser simulator for a five-finger robotic hand. Explore articulated motion, object contact, grasping, and camera-based hand control with Three.js, Rapier physics, and MediaPipe hand tracking.

The default robot uses the supplied original meshes, joint definitions, masses, and inertias: 27 links and 20 actuated joints. A simplified reference hand is also available. Camera inference and simulation run in the browser; the Python launcher only serves static files.

The local simulator with the original robot and the Pinch preset selected in Manual mode.

• Articulated robotics: a 20-joint hand with URDF/SRDF import, joint limits, and six-degree-of-freedom wrist control.
• Interactive physics: Rapier rigid-body simulation with object contact, friction, and hand self-collision filtering.
• Camera-based control: local MediaPipe hand tracking, open-hand calibration, and palm-relative motion mapping.
• Geometry-aware pinching: inverse kinematics coordinates the original robot's thumb and index fingertips.
• Inspectable sessions: record and export joint targets, measured angles, wrist poses, and object trajectories as JSON.

• Choose a setup
• Quick start
• Development setup
• Python environment
• Using the simulator
• Camera control and pinching
• Recording and export
• Importing a robot
• Tests and formatting
• Controller API
• Project structure
• Troubleshooting
• Physics and tracking limits
• References and notices

Run commands from the project root—the directory containing run.py and package.json.

The Python tools have been checked with Python 3.13.3. Use a browser with WebGL2; browser verification has used Chromium/Chrome. Camera control additionally requires a webcam, camera permission, and a secure context such as localhost or HTTPS.

A complete bundled build needs no Python packages, Node.js installation, API key, or external inference service. Development dependency installation needs registry access; asset preparation may download the camera model if missing. Once assets are present, the app runs locally without a network connection.

On macOS or Linux:

python3 run.py

On Windows:

py -3 run.py

Open http://localhost:8000/ if the browser does not open automatically. Keep the terminal running and press Ctrl+C to stop the server.

Start-Mac.command and Start-Windows.bat launch the same app using the system Python command. A virtual environment is optional for launching.

python3 run.py --port 8001
python3 run.py --no-browser
python3 run.py --help

Use py -3 instead of python3 on Windows. The server binds to 127.0.0.1, serves dist/, and disables caching to avoid stale files after rebuilding. Do not open index.html directly with a file:// URL.

Source edits do not update the bundled app automatically. Use the development server while editing, or rebuild dist/ before refreshing a page served by run.py.

Install the locked JavaScript dependencies, prepare assets, and start Vite:

npm ci
npm run assets
npm run dev

Open the URL printed by Vite, normally http://127.0.0.1:5173/. Vite reloads source changes during development. A Python environment is not required for this workflow.

npm run assets:

• Restores public/robot/ from bundled dist/robot/ when the source robot assets are missing.
• Restores the camera model from dist/models/ when available, otherwise downloads it.
• Copies MediaPipe's WASM runtime from node_modules/ into public/wasm/.

Some packaged copies keep large assets only in dist/, so prepare assets before the first development build. If both public/robot/ and dist/robot/ are missing, restore the supplied robot assets from the project package; the script cannot download them.

npm run build
npm run preview

Preview serves the generated build, normally at http://127.0.0.1:4173/; use the URL printed in the terminal. Alternatively, python3 run.py serves the same dist/ directory.

Both dev and build generate public/tracking-worker.js from src/tracking-worker.js. Edit the source worker, not the generated file. After changing worker code during development, restart npm run dev and reload the page to regenerate the worker bundle.

The optional audit and proxy-generation scripts use NumPy and SciPy. Their pinned versions in requirements.txt require Python 3.12 or later. Use an interpreter meeting that requirement when creating the environment.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m pip check

py -3 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python -m pip check

If PowerShell blocks activation, use the environment directly:

.\.venv\Scripts\python.exe -m pip install -r requirements.txt
.\.venv\Scripts\python.exe scripts/audit-urdf.py public/robot

For an existing .venv, activate it and install requirements as needed; do not recreate it each time. Run deactivate to leave it. In your editor, select .venv/bin/python on macOS/Linux or .venv\Scripts\python.exe on Windows. The environment and Python cache files are ignored by Git.

After activating the environment and preparing robot assets:

python scripts/audit-urdf.py public/robot

The audit prints a JSON report without modifying inputs. It checks URDF/SRDF structure, masses, inertias, limits, mesh references, and collision exclusions.

The STL audit expects the original export directory layout, containing printed_parts_hi_res/, sim_visual/, and sim_collision/:

python scripts/audit-stl.py /path/to/original/export

Replace the example path with your export directory. Passing the reorganized public/robot/ directory does not inspect its meshes because its directory names differ.

python scripts/build-supplied-proxy.py regenerates and overwrites the legacy public/models/supplied-hand.json. It is not needed for the default original-mesh robot. Archived generators in public/reference/ also require original CAD source modules that are not supplied by requirements.txt.

1. Launch the app and wait for Original hand · supplied meshes to load.
2. In Manual, try Open, Pinch, Power, Fist, and Spread.
3. Expand a finger's joint controls to adjust it individually. Wrist position uses millimeters; wrist rotation and joint sliders use degrees.
4. Select Run grasp demo to watch an approach, close, lift, and release sequence with a 40 mm, 25 g cube. This switches to Manual, resets the world, and replaces the scene's objects.
5. Select Reset world to restore the current robot and default objects. Reset also clears camera calibration and stops an active recording.

The original robot has its own presets. Pinch brings the thumb/index pads together; it does not find an object or move the wrist toward one. Grasping depends on object placement, contact, and friction. Motors obey joint limits, and contacts can stop a finger before its commanded target.

Self-collision starts on for the original hand and off for the approximate reference hand. The active pair count reflects self-contact; the scene's Hand contacts metric counts hand-to-object contact.

Shortcuts are ignored while an input/select is focused or the quick guide is open.

1. Select Camera → Enable camera and grant camera access.
2. Choose the hand being tracked. Left → right robot mirrors left-hand control onto the right robot; it does not load a left robot model.
3. Hold your open palm toward the camera with the wrist and all fingertips visible. Extend your thumb and separate it from your index finger.
4. Press Calibrate open hand and hold still while six stable samples are collected. Calibration times out after 30 seconds; movement, curled fingers, or a pinching thumb can restart or prevent sampling.
5. Move and rotate your palm gently, then bend and spread your fingers. Bring thumb and index fingertips together to pinch; separate them to reopen.
6. Use Clutch to hold the current commands while repositioning your real hand. Press it again with your hand visible to resume from the current wrist position. Physics continues while clutched; use Pause to freeze simulation time.

On the original robot, pinch assistance uses tracked fingertip separation and robot joint geometry to coordinate the thumb and index finger. Wrist movement positions this pinch workspace in the world. This compensates for different human/robot proportions; it does not attach objects or bypass collision physics. Explicit imported retarget maps do not receive built-in pinch assistance.

Increase thumb gain if the thumb bends correctly but stops short; decrease it if it closes too early. Near a pinch, fingertip alignment takes priority over independent thumb/index angles. Keep the camera fixed after calibration. Recalibrate after changing the tracked hand, resetting/recentering, switching robots, or updating the app.

Slow means frames arrive with noticeable delay; it differs from No hand. Use good lighting, keep fingertips visible, and move slowly. Tracking loss holds the last commands; reacquisition reanchors the wrist to avoid a jump. Stopping the camera, switching to Manual, or hiding the tab releases the camera and clears calibration. Camera frames and landmarks remain in the browser.

1. Select Record, then interact with the hand.
2. Select Stop when finished.
3. Select Export JSON to download the session.

The export includes robot/joint metadata, collision policy, physics settings, joint targets and measured angles, wrist pose, object trajectories, and contact counts. Its schema is dexterity-lab/1; distances are meters, angles radians, masses kilograms, time seconds, and quaternions use [x, y, z, w] order.

Recording samples at most about 30 times per simulated second and stops at 18,000 samples. Rendering performance can reduce the sampling rate. A new recording replaces the previous in-memory recording; reloading loses unexported data. There is no built-in session replay/import interface.

Select + Import URDF folder and choose a directory containing the URDF and every referenced mesh. The bundled layout is:

robot/
├── hand_right_printed.urdf
├── hand_right_printed.srdf
├── meshes/
│   ├── visual/       # Referenced STL or OBJ files
│   └── collision/    # Referenced collision meshes
├── retarget-map.json       # Optional
└── simulation-frame.json   # Optional

With multiple URDFs, the importer selects a uniquely named right-hand URDF; otherwise provide one URDF. Prefer public/robot/ (after asset preparation) or dist/robot/ over the archived raw export: bundled copies contain corrected paths, mesh units, and SRDF names.

Supported imports include STL/OBJ, box/sphere/cylinder shapes, fixed/revolute/continuous joints, origins and mesh scales, material colors, joint limits, mass, center of mass, and full inertia. Continuous controls are restricted to ±π. Collision meshes become convex hulls; concave geometry needs externally prepared convex decomposition.

A matching SRDF supplies self-collision exclusions. Without one, only directly connected links are excluded. Mismatched robot names, unknown links, conflicting matching SRDFs, missing/ambiguous meshes, and unsupported constructs cause import errors. Expand Xacro first. DAE, textures, mimic joints, prismatic/floating/planar joints, and tendon transmissions are unsupported.

For different joint names or axis conventions, include retarget-map.json:

{
  "index_pip": { "joint": "actual_index_joint", "sign": 1, "offset": 0 },
  "thumb_cmc_abd": { "joint": "actual_thumb_joint", "sign": -1, "offset": 0 }
}

Keys are tracker joint names; joint names the destination URDF joint. The mapping applies sourceAngle * sign + offset, with offsets in radians. These signs are examples, not universal settings. Check joints in Manual before camera use.

Standard names include <finger>_mcp_abd, <finger>_mcp_flex, <finger>_pip, and <finger>_dip for index/middle/ring/pinky, plus thumb_cmc_abd, thumb_cmc_flex, thumb_mcp_flex, and thumb_ip_flex.

An optional simulation-frame.json sets the source-to-display Euler XYZ rotation in radians:

{ "rpy": [-1.57079632679, 0, -1.57079632679] }

The bundled hand20_printed_right has recognized display and control conventions. Other robots need their own mapping and pose validation; the grasp demo is not calibrated for arbitrary imports.

After development setup:

npm test
npm run format:check

Tests cover inertia, original/reference dynamics, collision filtering, joint limits, grasp/release, native Pinch contact, camera-pinch commands for both hands, palm-axis rotation, calibration, and tracking-loss recovery. Original-hand tests use real bundled URDF/SRDF and STL files from public/robot/, falling back to dist/robot/.

For per-joint idle-motion measurements:

node scripts/check-hand-stability.mjs

Format JavaScript with:

npm run format

Prettier uses .prettierrc.json. Source, scripts, tests, and JavaScript configuration are formatted; dependencies, the Python environment, dist/, copied WASM loaders, and the generated camera worker are excluded by .prettierignore.

Once the app has loaded, window.dexterity is available in the browser console. Select Manual first so camera commands do not overwrite your commands.

const lab = window.dexterity;
lab.setPaused(true);
lab.setJointTargets({ index_pip: 0.9, index_dip: 0.5 }); // Radians
lab.setWrist([0, 0.16, 0]); // Meters; keep the current wrist orientation
lab.spawn('cube', [0, 0.2, 0.1], 0.04, 0.035); // Size in meters, mass in kg
const state = lab.advance(120); // One simulated second, only while paused
console.log(state.qMeasured);
lab.setPaused(false);

setWrist(position, quaternion) also accepts an optional [x, y, z, w] quaternion. advance() accepts 1–1200 whole steps. snapshot() reads the current state; reset() restores the scene. model, simulation, tracker, and scene expose current instances for development. Targets use the same physical motors as the UI; the API does not bypass contacts or limits.

• Physics: Rapier 0.19.3, fixed 120 Hz stepping, a kinematic wrist, dynamic links, and continuous collision detection. Solver iterations are 32 plus 8 additional hand iterations for the original robot, and 12 plus 4 for the reference.
• Motion: wrist targets are rate-limited to 0.65 m/s and 4 rad/s. Joint target speed is the smaller of 3.5 rad/s and the imported velocity limit. Contact impulses can produce higher instantaneous body velocities.
• Motors: ideal force-based position servos. URDF effort limits are metadata, not hard torque caps. Tendons, gearbox dynamics, saturation, and compliance are not modeled.
• Geometry: supplied visual detail and inertia are retained; collision meshes use convex hulls. The original hand filters 31 SRDF exclusions plus directly connected pairs. Other hand contacts remain active.
• Tracking: MediaPipe Tasks Vision 0.10.32 uses the bundled Hand Landmarker float16 model in a classic Web Worker, with GPU inference and CPU fallback. It provides 21 landmarks, not a dense surface or reliable absolute depth. Wrist depth is inferred from image scale.
• Accuracy: occlusion, foreshortening, finite solver tolerances, and differing human/robot proportions limit alignment. The full configuration space and arbitrary grasps are not validated. On slow machines, simulated time can progress more slowly than real time.

Automated and browser checks do not establish performance with your actual camera or physical robot. This is an interactive simulation workspace, not hardware validation or an exact digital twin.

• Change history
• URDF and geometry review
• Original-hand stability correction
• Earlier verification scope
• Third-party components and notices
• Bundled license texts

Reports describe the versions and fixtures under which they were produced. The stability report supersedes earlier grasp/collision measurements; historical reports do not replace current tests. Supplied robot/reference assets are not relicensed by this README.