Voicy

A native macOS voice assistant. Speak a coding task → a locally-run LLM (LM Studio / Ollama / MLX) refines it into a structured spec → you correct it by voice or text → copy it to the clipboard for your coding agent.

Speech-to-text is on-device via WhisperKit (Whisper on CoreML). Nothing leaves your machine — transcription is local, and the LLM is a local OpenAI-compatible server you run yourself (LM Studio / Ollama / mlx_lm.server).

• ✅ First feature: Voice → Coding Task (push-to-talk + continuous capture, refine, voice/text correction, clipboard output).
• ✅ Scaffolded: Notes (continuous capture lands here), Knowledge base (stub), Settings, onboarding, menu-bar item.
• ⏳ Deferred: knowledge-base embeddings/semantic search & retrieval into prompts.

• macOS 14 (Sonoma) or later.
• Xcode 15+ (full Xcode, not just Command Line Tools — needed for the SwiftData macros and to build an app target).
• XcodeGen — brew install xcodegen.
• A local LLM server exposing an OpenAI-compatible API (run whichever you like):
  • LM Studio — server on http://127.0.0.1:1234/v1 (serves the MLX/GGUF models you already have under ~/.lmstudio/models).
  • Ollama — ollama serve (http://127.0.0.1:11434/v1).
  • MLX — mlx_lm.server (http://127.0.0.1:8899/v1).

# 1. Point the toolchain at full Xcode (once, after installing it):
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer

# 2. Generate the Xcode project from project.yml:
xcodegen generate

# 3. Open it:
open Voicy.xcodeproj

In Xcode:

1. Let Swift Package Manager resolve WhisperKit and KeyboardShortcuts (File ▸ Packages ▸ Resolve, or it happens automatically).
2. Select the Voicy target ▸ Signing & Capabilities ▸ set Team to your personal team or "Sign to Run Locally."
3. Build & run (⌘R).

Or from the command line:

xcodebuild -project Voicy.xcodeproj -scheme Voicy -configuration Debug build
xcodebuild -project Voicy.xcodeproj -scheme Voicy test   # runs VoicyTests

Onboarding walks you through three steps:

1. Microphone access — grant it.
2. Speech model — pick one (Base English is the fast default) and download/load it. First download is a few hundred MB.
3. LLM server — start LM Studio / Ollama / mlx_lm.server, then in Settings select the matching provider and hit Test to confirm it's reachable.

1. Go to Tasks.
2. Hold the "Hold to talk" button (or press ⌥⌘Space from anywhere) and describe the coding task. Release to transcribe.
3. Click Refine into task — the local model streams a structured draft (title, context, request, acceptance criteria, constraints, out of scope).
4. Correct it: edit fields directly, type a correction, or hold the correction mic and speak ("use SwiftData not CoreData"). Changed sections are highlighted.
5. Copy to clipboard (⇧⌘C) and paste into your coding agent.

Continuous mode: toggle it in the sidebar. Spoken utterances are saved as Notes; say your wake phrase ("new task …", configurable in Settings) to spin a new task straight from speech.

Audio (AVAudioEngine, 16kHz) → Transcription (WhisperKit) → CaptureCoordinator
   → TaskRefiner (LLMClient, OpenAI-compatible streaming) → Task Composer UI
   → ClipboardService.  Persistence via SwiftData.

• Audio/ — AudioCaptureService (mic → 16 kHz mono frames + level).
• Transcription/ — SpeechTranscribing protocol, WhisperKitTranscriber (the only file importing WhisperKit), TranscriptionService, ModelManager.
• Capture/ — CaptureCoordinator (push-to-talk / continuous / correction), VoiceActivity (energy VAD), WakePhrase, global Hotkeys.
• LLM/ — ChatEngine protocol + RemoteChatEngine (LLMClient, streaming OpenAI-compatible), ProviderConfig presets, Prompts, TaskRefiner.
• Models/ — SwiftData models + TaskDraft value type.
• Output/ — TaskMarkdownFormatter, ClipboardService.
• UI/ — RootWindow, TaskComposerView, SettingsView, NotesView, KnowledgeView, MenuBarView, OnboardingView, components.

• Speech models: WhisperKit downloads + caches the selected model on first use (its own cache). The STT backend is isolated to Transcription/WhisperKitTranscriber.swift; targets the WhisperKit 0.9.x API.
• App Sandbox is OFF (Sources/Voicy/Resources/Voicy.entitlements). A sandboxed AVAudioEngine input path crashes with a PRECONDITION FAILURE when it can't Mach-look-up com.apple.audioanalyticsd. Running unsandboxed avoids the whole mach-lookup class (hardened runtime stays on; mic gated by the usage string
  • audio-input entitlement). To ship sandboxed, re-enable it and add a com.apple.security.temporary-exception.mach-lookup.global-name array with com.apple.audioanalyticsd.
• Audio tap: the AVAudioEngine tap is installed with format: nil and the 16 kHz converter is built lazily from the real buffer format — installing with an explicit format throws a "format mismatch" fault on multi-channel mics.
• Requests send only temperature — no top_k/top_p/repetition penalties — so the local server's own defaults stand.