OmniVoice Rust Port
GPU-first Rust workspace for OmniVoice inference, parity validation, and CLI execution with Candle.

• What is this?
• Demo
• Key Features
• Installation & Setup
• How to Start Using
• System Requirements
• Acknowledgments
• License

This repository is a Rust workspace focused on porting OmniVoice inference to Candle with GPU-first execution.

It contains:

• crates/omnivoice-infer — the two-stage inference pipeline
• crates/omnivoice-cli — CLI for prompt prep, stage0/stage1 inspection, and full inference
• crates/omnivoice-server — separate OpenAI-compatible HTTP speech server binary
• docs/contracts — phase-by-phase behavior contracts
• tools — local reference and support scripts

The project prefers CUDA and Metal when available, but full OmniVoice inference is available on CPU as well as GPU backends.

There is no public GUI demo in this workspace. The primary interface is the CLI.

cargo run -p omnivoice-cli --features cuda -- infer `
  --text "Hello, this is a test of zero-shot text-to-speech." `
  --language en `
  --output artifacts\demo.wav `
  --device cuda:0 `
  --dtype f32 `
  --seed 1234

If you want to force a local bundle or a specific Hugging Face repo, pass --model:

cargo run -p omnivoice-cli --features cuda -- infer `
  --model model `
  --text "Hello from a local model bundle." `
  --output artifacts\demo-local.wav

For an OpenAI-compatible speech endpoint, run the separate server binary:

$env:OMNIVOICE_API_KEY="local-dev-token"
cargo run -p omnivoice-server --features cuda -- `
  --host 127.0.0.1 `
  --port 8000 `
  --device cuda:0 `
  --dtype f32

• Two-stage OmniVoice inference pipeline in Rust
• GPU-first runtime selection: CUDA -> Metal -> CPU fallback
• Full inference paths available on CPU and GPU
• Stage0 and Stage1 parity harnesses against official OmniVoice references
• Voice clone, voice design, auto voice, batch, and long-form chunked inference
• CLI workflows for prepare-prompt, stage0-debug, stage1-decode, and infer
• Separate omnivoice-server binary with /, /health, /v1/models, and /v1/audio/speech
• Server-side readiness reporting, base-path mounting, CORS/OPTIONS, and JSON or multipart request support for TTS

• Rust toolchain
• CPU-only inference works; GPU acceleration is optional
• For CUDA: NVIDIA GPU and compatible driver/toolkit
• For Metal: macOS with Metal support
• Either a local OmniVoice bundle or network access to download models into the Hugging Face cache
• Official upstream reference materials available locally

cargo fmt
cargo test -p omnivoice-infer --features cuda --test phase_status -- --nocapture --test-threads=1
cargo test -p omnivoice-cli --features cuda --test phase10_cli_cuda -- --nocapture --test-threads=1

cargo fmt
cargo clippy --workspace --all-targets
cargo test -p omnivoice-infer --features cuda --test phase10_cuda_acceptance -- --nocapture --test-threads=1
cargo test -p omnivoice-cli --features cuda --test phase10_cli_cuda -- --nocapture --test-threads=1

1. Either pass --model <local-path> or let the CLI/server auto-resolve k2-fsa/OmniVoice from Hugging Face.
2. If you omit --asr-model, Whisper defaults to oxide-lab/whisper-large-v3-turbo-GGUF and downloads the Candle-compatible config.json, tokenizer.json, and whisper-large-v3-turbo-q4_0.gguf.
3. Keep official upstream reference materials available locally for behavior verification.
4. Run sequential GPU tests instead of launching everything at once.
5. Use omnivoice-cli infer for end-to-end synthesis.

• Windows, macOS, or Linux
• Enough RAM and VRAM for OmniVoice weights and runtime tensors
• For GPU acceleration:
  • NVIDIA GPU for CUDA
  • Apple Silicon / supported macOS GPU for Metal

This workspace builds on top of the official upstream projects:

• OmniVoice
• candle
• mistral.rs

Official upstream projects remain the source of truth for behavior and engineering reference, but they are not tracked as part of this repository.

See THIRD_PARTY_LICENSES.md for focused upstream attribution relevant to this workspace.

Apache-2.0 — see LICENSE

Copyright (c) 2026 FerrisMind