mcp-voice

Local voice agent: STT → Gemini 2.5 Flash (+ MCP tools) → Jarvis TTS.

Terminal 1 — gateway:

# after venv + deps + voices/jarvis_conds.pt + GEMINI_API_KEY in .env
uvicorn apps.gateway.main:app --host 0.0.0.0 --port 8000

Terminal 2 — Jarvis Orb UI:

cd apps/jarvis-orb
npm install
npm run dev

Open http://127.0.0.1:5173 → hold HOLD TO TALK → release → hear Jarvis.

If attendees only need a minimal local flow (agent + MCP tools + voice), skip FastAPI/UI and run one script:

python scripts/voice_agent_cli.py --text "Remember my workshop is Friday" --print-tools

Or with audio input:

python scripts/voice_agent_cli.py --audio-in sample.wav --audio-out reply.wav --print-tools

This uses the same core pipeline modules but avoids HTTP routes/websockets during the workshop.

Say "Hey Jarvis" instead of holding the button. Uses openWakeWord's pretrained hey_jarvis model, running fully local on the gateway (CPU via ONNX Runtime — no account, no API key, no per-attendee signup). The mic stream for wake-word listening never leaves your machine except to your own gateway process.

The model auto-downloads the first time the gateway starts (needs internet once; cached after that). Click ENABLE WAKE WORD in the UI, then say "Hey Jarvis" followed by your request — it auto-stops recording after you go quiet. If WAKE WORD UNAVAILABLE shows up, check the gateway logs for the download/load warning.

If Gemini fails at runtime and Groq fallback is configured, the fallback path also runs the same tool set before answering.

Use VS Code Search for all coding checkpoints:

• Query: TODO(Workshop|:)
• Regex: on
• Files to include: apps/gateway/**,mcp_servers/assistant/**

Also see the attendee checklist in WORKSHOP_TODOS.md.

Try saying:

• “What's the weather in Mumbai?”
• “Any news about AI chips?”
• “Search the web for the latest AI chips headlines.”
• “Remember that my workshop is on Friday.”
• “What notes do you have saved for me?”

Standalone MCP server (for Cursor / other MCP clients):

python -m mcp_servers.assistant.server

cp .env.example .env          # set GEMINI_API_KEY
# voices/jarvis-5s.wav (>5s)

docker compose build
docker compose run --rm gateway python scripts/build_jarvis_conds.py --device cpu --backend turbo
docker compose up

docker compose up now starts two services:

• gateway (FastAPI on :8000)
• mcp-server (stdio MCP server for external MCP clients)

If you only want the HTTP gateway:

docker compose up gateway

If you only want the standalone MCP server:

docker compose up mcp-server

UI still runs on the host:

cd apps/jarvis-orb && npm install && npm run dev

docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build

Gateway-only on GPU hosts:

docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build gateway

Python 3.11 or 3.12:

python -m venv .venv
# Windows: .\.venv\Scripts\Activate.ps1
# Mac:     source .venv/bin/activate

pip install -U pip wheel
pip install "setuptools>=70,<81"

# Windows NVIDIA:
pip install torch==2.6.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
# Mac / CPU:
# pip install torch==2.6.0 torchaudio==2.6.0

pip install -r requirements.txt
cp .env.example .env   # GEMINI_API_KEY, TTS_DEVICE=auto
python scripts/build_jarvis_conds.py --device auto --backend turbo
uvicorn apps.gateway.main:app --host 0.0.0.0 --port 8000

Then start apps/jarvis-orb as above.

The gateway now supports STT tuning via .env:

# model
WHISPER_MODEL=medium.en

# decode profile: fast | balanced | accurate
STT_PROFILE=balanced

# auto chooses int8_float16 on cuda, int8 otherwise
STT_COMPUTE_TYPE=auto

# force English for better speed/stability (or use auto)
STT_LANGUAGE=en

# trim silence and tune search width
STT_VAD_FILTER=true
STT_BEAM_SIZE=2

Recommended low-latency setup on NVIDIA + conda:

WHISPER_MODEL=medium.en
STT_DEVICE=cuda
STT_PROFILE=fast
STT_COMPUTE_TYPE=auto
STT_LANGUAGE=en
STT_VAD_FILTER=true
STT_BEAM_SIZE=1

If GPU memory is still tight, use small.en with STT_PROFILE=fast.

apps/gateway/     FastAPI + STT/TTS + Gemini agent + tools
apps/jarvis-orb/  React orb UI (voice + wake word)
mcp_servers/      MCP stdio server (same tools)
data/notes.db     saved notes (created at runtime)
voices/           jarvis wav + conds
scripts/          smoke tests

Workshop guide: see WORKSHOP.md.

• Phase 1 — STT + TTS + conds
• Phase 2 — Gemini turn pipeline
• Phase 3 — MCP tools + React voice UI