Hey Dot

Hey Dot is a fully local, voice-first desktop AI assistant.

It stays on top of your desktop, listens for "Hey Dot", captures your screen, runs local vision-language inference, and speaks back a concise answer. No cloud APIs, no internet dependency, and no data leaving your machine.

• 100% local AI runtime
• Wake word + voice query flow
• On-device screen understanding
• Spoken answers with low latency
• No API keys required

1. Hey Dot listens in the background for a wake phrase.
2. On wake, it starts a short active listening window.
3. It captures the current screen locally.
4. It sends the image + prompt to an Ollama model.
5. It speaks the response using macOS text-to-speech.

• Python 3
• customtkinter (desktop overlay UI)
• SpeechRecognition + PyAudio (microphone capture)
• faster-whisper (offline speech-to-text)
• mss + Pillow (screen capture)
• Ollama Python client (local model inference)
• numpy, python-dotenv
• macOS say command (voice output)

• hey_dot.py: Main desktop assistant runtime
• requirements.txt: Python dependencies
• frontend/: React + Vite showcase site for the project

• macOS
• Python 3.10+
• Ollama installed and running
• Microphone permission enabled for Terminal/Python
• Screen recording permission enabled for Terminal/Python

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Pull the default local vision-language model:

ollama pull moondream

source venv/bin/activate
python hey_dot.py

Runtime flow:

1. Wait for "Dot is Sleeping"
2. Say "Hey Dot"
3. Ask your question
4. Hear the spoken answer

• OLLAMA_MODEL: Local Ollama model name (default: moondream)
• WHISPER_MODEL: Whisper checkpoint (default: base.en)

The frontend communicates the same positioning as the app: fully local, private, and cloud-free.

cd frontend
npm install
npm run dev

• Wake-word detection includes phrase, token, and fuzzy matching for robustness.
• The current capture path targets mss().monitors[1].
• Responses are intentionally short for natural voice playback.

• Wake word detection is STT-driven, not a dedicated keyword spotting model.
• Performance depends on hardware and local model choice.
• Multi-monitor selection is not yet configurable.