lstk-eye

Hackable AI glasses: a see-through HUD, a camera and a mic on your own frames, with a laptop as the brain. The device is a thin client — capture in, pixels out; the server does the seeing, listening and thinking. What the glasses do is a set of voice-switched modes, and modes are plugins:

• Point and explain — ask a question about what you're looking at; the answer comes back as step-by-step slides with arrows anchored to the real objects ("how do I check battery voltage" → arrow at the VΩmA jack, then COM, then the dial).
• Find and track — name a thing; the HUD locks a marker onto it and follows it as you move.
• Drive other machines — a robot arm with its own camera points where you look (head-aim) or keeps a person centered (person-follow). Same glasses, new mode.

An AI that can finally point at things while it talks — that is the whole idea.

• Robot arm modes: head-aim and person-follow (Vimeo, 44 s)

Status: working prototype — the demos are the real device. Early development; expect rough edges.

 glasses (ESP32-S3)                       laptop (Python server)
┌─────────────────────┐   JPEG + WAV    ┌──────────────────────────────┐
│ camera ── button ── │ ──── HTTP ────▶ │ speech-to-text (Whisper)     │
│ mic ── gyro         │  + gyro (UDP)   │ vision LLM (any provider)    │
│                     │                 │ tracking / registration      │
│ 128x64 OLED ◀────── │ ◀─── scenes ─── │ modes = plugins              │
│ (beamsplitter HUD)  │   (JSON draw    │ camera↔display calibration   │
└─────────────────────┘    primitives)  │ optional: robot arm loop     │
                                        └──────────────────────────────┘

The glasses are a thin client. Firmware does capture (photo, voice, preview stream, gyro), button gestures, and drawing of a handful of primitives (text, markers, chevrons, reticles) that the server sends as tiny JSON scenes. Zero intelligence onboard, so the device never needs a reflash when the software gets smarter.

The server owns all state and thinking, and every stage is an interface with at least two implementations (a real one and a mock): speech-to-text, the vision LLM, segmentation, tracking, the arm's target sources. --profile mock runs the whole system with no models, no keys and no hardware; a device simulator plays the glasses.

What the glasses do is a mode. A mode is a plugin: it is entered by a voice cue, takes over the HUD (and the arm, if it needs one), receives previews/photos/speech while active, and ends on a stop phrase or a double click. Adding a behavior is one class registered by name — no session or firmware changes. Modes shipped today:

• Point and explain — a photo is segmented (SAM-style), numbered marks are drawn on it (Set-of-Marks), and the vision LLM answers with a step plan that references objects only by mark number (no raw coordinates → no coordinate hallucination). Steps render as arrows on the real objects; double-click advances.
• Find and track — the vision LLM returns the target's center directly and asks for zoom crops of small details (a pin on a board resolves as precisely as a chair); the marker then follows the object at ~5 Hz through edge-verified template matching on the preview stream. Look away and it hides; targets outside the display window get edge chevrons.
• Head-aim (robot arm) — the arm points where you look: the glasses' gyro gives the instant response, SuperPoint + LightGlue registration between the glasses' frame and the arm camera anchors the aim to the exact spot.
• Person-follow (robot arm) — the arm keeps a person centered in its camera: pose-gated detection, lock-on that survives dropouts, edge pursuit.

Latency, honestly: speech-to-text 0.3–0.5 s (Whisper large-v3-turbo on Apple-silicon GPU); pointing 1–2 s with a Flash-class model, 3–6 s with a large one; marker refresh ~5 Hz by the physics of the WiFi round trip; head-aim reacts to a head turn within ~100 ms via the gyro. Text rendering on the device is instant and never degrades.

Full build guide: docs/HARDWARE.md. Bill of materials (~$40 on top of your own glasses):

Everything mounts on the left temple of regular glasses. Hot glue is the official assembly method — dielectric, reversible with isopropyl.

Requires Python 3.11+ and uv (or plain pip).

git clone https://github.com/Artod/lstk-eye
cd lstk-eye
uv sync

Run the server in mock mode (no ML models, no API key) and drive it with the built-in device simulator:

uv run lstk-eye serve --profile mock

uv run lstk-eye simulate --ask "how do I check battery voltage"

The simulator plays the role of the glasses: it "captures" a photo, sends the question, renders the returned scenes as PNG frames, and advances slides — the full nervous system without a soldering iron.

For the real pipeline install the ML extras and provide a vision-LLM API key (Anthropic by default; Gemini and any OpenAI-compatible server are one config line — see lstk-eye.example.toml):

uv sync --extra full
export ANTHROPIC_API_KEY=sk-ant-...
uv run lstk-eye serve

Extras: full = stt (faster-whisper) + seg (ultralytics) + arm (servo SDK, ByteTrack); mlx adds Apple-silicon Whisper (stt.backend = "mlx"); gemini and openai add the other planner backends, which read GEMINI_API_KEY / OPENAI_API_KEY (or planner.api_key, and planner.base_url for a local OpenAI-compatible server).

Check your environment anytime with:

uv run lstk-eye doctor

The two arm modes above (head-aim, person-follow) need a small robot arm carrying its own camera; the glasses hand it their point of attention. Setup, in order:

Hardware: an SO-101-class arm on Feetech STS3215 bus servos, a USB camera bolted to the arm (many webcams cap at 30 fps; a 60–120 fps one tracks visibly better), and — for head-aim — the MPU6050 on the glasses.

Setup ([arm] in lstk-eye.toml, full list with defaults in lstk-eye.example.toml):

• backend = "feetech", port (servo bus serial port), calibration_path — a lerobot-style calibration JSON; an arm already calibrated with lerobot needs no re-calibration.
• camera_name (substring of the device name, survives USB re-plugs) or camera_index.
• ready_pose / sleep_pose — the aiming stance and the parked stance, per joint in degrees. Record them with lstk-eye arm pose: the servos go limp for a few seconds, you hold the arm where you want it, and the pose it locks into is printed.
• pan_sign / tilt_sign — from lstk-eye arm probe --wiggle, which steps each joint and measures which way the image moved.
• imu_yaw_axis / imu_pitch_axis — from lstk-eye arm imu: turn your head left-right, then nod, and the axes with the largest swings (sign included) are yaw and pitch.
• gaze_matcher = "auto" takes SuperPoint + LightGlue when torch and LightGlue are installed, and falls back to a SIFT matcher otherwise — head-aim works either way, the learned one just registers far more often on low-texture scenes.

Using it: say "protect me" / "guard me" (RU «защити», «охраняй») or "manual mode" / "head aim" / "follow my head" (RU «ручной режим»); a stop phrase — "stop", "stand down", "at ease", «стоп», «отбой» — or a double click ends the mode, and another cue switches modes in place. All phrase lists live in [modes].

Watching it: GET /api/v1/arm/stream is an MJPEG view of the arm camera with telemetry overlays (open it in a browser for demos), GET /api/v1/arm/status returns the same state as JSON. With arm.record_runs every mode activation is recorded to runs/arm/ (overlay mp4 + per-tick jsonl telemetry); head-aim additionally dumps its first few hundred registration attempts to runs/gaze/ as eye frame and arm frame side by side — the two things worth looking at when tuning.

Modes never touch joints directly: each installs a target source into the arm loop, so a new behavior is one class. Details in docs/ARCHITECTURE.md.

Arduino sketch for the XIAO ESP32S3 Sense lives in firmware/lstk_eye_glasses. Board setup, library list, and flashing instructions: firmware/README.md.

src/lstk_eye/          server package
  protocol/            device<->server wire schemas (pydantic)
  pipeline/            STT, segmentation, planner, relocalization (all pluggable)
  arm/                 robot arm: servo bus, camera, visual-servo aim, target sources
  modes/               voice-switched behavior plugins (protect, head-aim)
  simulator/           software stand-in for the glasses
firmware/              Arduino sketch for the glasses
docs/                  architecture, protocol, hardware, interaction, calibration
tests/                 pytest suite (runs fully offline on mocks)

Every pipeline stage is an interface with at least two implementations (real + mock); adding a backend (different STT, a new segmenter, another LLM) means implementing one class and registering it in pipeline/factories.py. See docs/ARCHITECTURE.md.

Issues and PRs welcome — see CONTRIBUTING.md. The test suite runs offline in seconds (uv run pytest -m "not slow"); CI is just ruff + pytest.

MIT