OminiX-API

OpenAI-compatible API server for OminiX-MLX models on Apple Silicon.

• OpenAI-compatible endpoints - Drop-in replacement for OpenAI API
• LLM chat - Chat completions with Qwen3, Mistral, GLM models
• Speech-to-text - Audio transcription with Paraformer ASR
• Text-to-speech - Voice cloning with GPT-SoVITS (named voices, few-shot, pre-computed codes)
• Image generation - Text-to-image with FLUX.2-klein and Z-Image-Turbo
• WebSocket streaming TTS - MiniMax T2A compatible protocol with per-message voice switching
• Voice registry - Named voices with aliases, configurable via voices.json
• Dynamic model loading - Switch models at runtime without server restart
• Memory efficient - One model per category, automatic unloading when switching
• Pure Rust - No Python dependencies at runtime

curl -fsSL https://raw.githubusercontent.com/OminiX-ai/OminiX-API/main/install.sh | sh

Requires macOS 14.0+ on Apple Silicon (M1/M2/M3/M4). The script will:

• Check for Xcode Command Line Tools (needed for Metal GPU acceleration) and prompt to install if missing
• Download the latest release binary and verify its SHA-256 checksum
• Install ominix-api to /usr/local/bin
• Create ~/.OminiX/ config directory with default settings

To pin a version or change install location:

VERSION=1.0.0 INSTALL_DIR=~/.local/bin curl -fsSL https://raw.githubusercontent.com/OminiX-ai/OminiX-API/main/install.sh | sh

# Clone both repositories
git clone https://github.com/OminiX-ai/OminiX-API.git
git clone https://github.com/OminiX-ai/OminiX-MLX.git

cd OminiX-API
cargo build --release
# Binary: target/release/ominix-api

# Run with LLM only (downloads model automatically)
LLM_MODEL=mlx-community/Qwen3-4B-bf16 ominix-api

# Run with all models
PORT=8080 \
LLM_MODEL=mlx-community/Qwen3-4B-bf16 \
ASR_MODEL_DIR=./models/paraformer \
TTS_REF_AUDIO=./audio/reference.wav \
IMAGE_MODEL=zimage \
ominix-api

# Run with app manifest validation
ominix-api --app-manifest my-app.ominix.toml

The TTS engine uses GPT-SoVITS for voice cloning. It supports three quality tiers, selected automatically based on what's available for each voice:

┌─────────────────────────────────────────────────────┐
│              Voice Quality Tiers                     │
├─────────────────────────────────────────────────────┤
│                                                      │
│  1. Pre-computed codes  (best quality)               │
│     ref_audio + ref_text + codes_path                │
│                                                      │
│  2. Few-shot / HuBERT   (good quality)               │
│     ref_audio + ref_text  (HuBERT extracts codes)    │
│                                                      │
│  3. Zero-shot            (baseline)                  │
│     ref_audio only  (mel spectrogram matching)       │
│                                                      │
└─────────────────────────────────────────────────────┘

Named voices are configured in a JSON file. The server loads this at startup from $VOICES_CONFIG or the default path ~/.dora/models/primespeech/voices.json.

{
  "default_voice": "doubao",
  "models_base_path": "~/.dora/models/primespeech",
  "voices": {
    "doubao": {
      "ref_audio": "moyoyo/ref_audios/doubao_ref_mix_new.wav",
      "ref_text": "这家resturant的steak很有名",
      "codes_path": "gpt-sovits-mlx/doubao_mixed_codes.bin",
      "speed_factor": 1.1,
      "aliases": ["default"]
    },
    "luoxiang": {
      "ref_audio": "moyoyo/ref_audios/luoxiang_ref.wav",
      "ref_text": "复杂的问题背后也许没有统一的答案...",
      "codes_path": "gpt-sovits-mlx/codes/luoxiang_codes.bin",
      "speed_factor": 1.1,
      "aliases": ["luo"]
    }
  }
}

Paths can be absolute or relative to models_base_path. Tilde (~) is expanded to $HOME.

When a voice name is provided (via HTTP voice field or WebSocket voice_id), it resolves in this order:

voice: "luo"
    │
    ├─ 1. Registry lookup (case-insensitive)
    │     ├─ Direct name match ("luo"?)  → no
    │     └─ Alias match ("luo" in luoxiang.aliases?)  → yes
    │           │
    │           ├─ Has codes_path?  → pre-computed codes mode
    │           ├─ HuBERT available?  → few-shot mode
    │           └─ Otherwise  → zero-shot mode
    │
    └─ 2. File path fallback (if not in registry)
          └─ Validated for path traversal, then used as zero-shot reference

Voice switching is on-demand. The base models (BERT, T2S, VITS, HuBERT) stay loaded; only the reference conditioning changes per voice. Switching overhead is ~100-700ms depending on the voice.

POST /v1/audio/speech

Synthesize speech from text. Supports dynamic voice switching per request.

Request:

{
  "input": "Hello, this is a test.",
  "voice": "luoxiang",
  "model": "gpt-sovits",
  "response_format": "wav",
  "speed": 1.0,
  "instruct": "用兴奋激动的语气说话，充满热情和活力"
}

Response formats:

• Default (PCM streaming): Raw PCM audio (16-bit signed LE, mono, 24kHz) with Content-Type: audio/pcm and Transfer-Encoding: chunked. Chunks are sent as they're generated (~800ms per chunk). Headers include X-Audio-Sample-Rate: 24000, X-Audio-Channels: 1, X-Audio-Bits-Per-Sample: 16.

• WAV (buffered): Complete WAV file with Content-Type: audio/wav. Triggered by ?format=wav query parameter, "response_format": "wav", or when reference_audio is present (voice cloning always returns WAV).

For Qwen3-TTS endpoints, instruct enables speaker+instruct mode for preset voices and clone+instruct mode for voice cloning. JSON clients may also send prompt as a compatibility alias.

Examples:

# Use a named voice from the registry
curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "你好世界", "voice": "doubao"}' \
  -o doubao.wav

# Use a voice alias
curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "你好世界", "voice": "luo"}' \
  -o luoxiang.wav

# Switch voices between requests (no reconnection needed)
curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "主持人说话", "voice": "marc"}' -o host.wav

curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "嘉宾回应", "voice": "luoxiang"}' -o guest.wav

Python example with dynamic voice switching:

import requests

API = "http://localhost:8080/v1/audio/speech"

dialogue = [
    ("marc", "大家好，欢迎来到今天的节目。"),
    ("luoxiang", "谢谢主持人，很高兴来到这里。"),
    ("yangmi", "我也很开心参加今天的讨论。"),
]

for i, (voice, text) in enumerate(dialogue):
    resp = requests.post(API, json={"input": text, "voice": voice})
    with open(f"line_{i+1}_{voice}.wav", "wb") as f:
        f.write(resp.content)
    print(f"[{voice}] {len(resp.content)} bytes")

GET /ws/v1/tts (upgrade to WebSocket)

Streaming TTS over a persistent WebSocket connection. Follows the MiniMax T2A protocol with an extension for per-message voice switching.

Client                                    Server
  │                                         │
  │  ──── WebSocket Connect ────────────►   │
  │  ◄──── connected_success ───────────    │
  │                                         │
  │  ──── task_start ───────────────────►   │
  │  ◄──── task_started ───────────────     │
  │                                         │
  │  ──── task_continue (text) ─────────►   │
  │  ◄──── task_progress (audio chunk) ─    │
  │  ◄──── task_progress (audio chunk) ─    │
  │  ◄──── task_progress (final chunk) ─    │
  │                                         │
  │  ──── task_continue (text) ─────────►   │  (repeat)
  │  ◄──── task_progress ... ───────────    │
  │                                         │
  │  ──── task_finish ──────────────────►   │
  │  ◄──── connection closed ───────────    │

Connect to the WebSocket endpoint. The server sends a confirmation:

{"event": "connected_success"}

Configure voice and audio settings for the session:

{
  "event": "task_start",
  "voice_setting": {
    "voice_id": "marc",
    "speed": 1.0
  },
  "audio_setting": {
    "format": "wav"
  }
}

Server responds:

{"event": "task_started"}

Send text to synthesize. Optionally override the voice per message:

{
  "event": "task_continue",
  "text": "要合成的文本",
  "voice_id": "luoxiang"
}

The server synthesizes the audio using native streaming generation and sends PCM chunks as they're produced (~800ms of audio per chunk, hex-encoded):

{"event": "task_progress", "data": {"audio": "52494646..."}, "is_final": false}
{"event": "task_progress", "data": {"audio": "a1b2c3d4..."}, "is_final": false}
{"event": "task_progress", "data": {"audio": "e5f6a7b8..."}, "is_final": true}

The data.audio field contains hex-encoded raw audio bytes. The is_final flag indicates the last chunk for this text segment. Decode with bytes.fromhex(audio_hex).

Processing is sequential: the next task_continue is not processed until the current one finishes streaming all chunks.

Close the session:

{"event": "task_finish"}

Errors are sent as:

{"event": "error", "message": "Description of the error"}

Common errors:

• "Missing 'text' field" - task_continue without text
• "TTS error: ..." - Synthesis failure
• "Inference unavailable" - TTS model not loaded
• "Invalid JSON" - Malformed message
• "Unknown event: ..." - Unrecognized event type

Use per-message voice_id to switch speakers within a single connection:

import asyncio
import json
import struct
import websockets

async def multi_voice_dialogue():
    dialogue = [
        ("marc",     "大家好，欢迎来到今天的节目。"),
        ("luoxiang", "谢谢主持人。我来谈谈法律方面的问题。"),
        ("yangmi",   "我觉得AI在创意领域也很有潜力。"),
        ("marc",     "两位说得都有道理，感谢参与讨论！"),
    ]

    async with websockets.connect("ws://localhost:8080/ws/v1/tts") as ws:
        await ws.recv()  # connected_success

        # Configure session (default voice, can be overridden per message)
        await ws.send(json.dumps({
            "event": "task_start",
            "voice_setting": {"voice_id": "marc", "speed": 1.0},
            "audio_setting": {"format": "wav"}
        }))
        await ws.recv()  # task_started

        all_pcm = b""

        for speaker, text in dialogue:
            # Send text with per-message voice override
            await ws.send(json.dumps({
                "event": "task_continue",
                "text": text,
                "voice_id": speaker
            }))

            # Collect audio chunks
            audio = b""
            while True:
                resp = json.loads(await ws.recv())
                if resp["event"] == "error":
                    print(f"Error: {resp['message']}")
                    break
                hex_data = resp.get("data", {}).get("audio", "")
                if hex_data:
                    audio += bytes.fromhex(hex_data)
                if resp.get("is_final"):
                    break

            print(f"[{speaker:8s}] {len(audio):>7d} bytes | {text}")

            # Save individual line
            with open(f"{speaker}_{text[:10]}.wav", "wb") as f:
                f.write(audio)

            # Accumulate PCM (skip 44-byte WAV header)
            if len(audio) > 44:
                all_pcm += audio[44:]

        # Write combined WAV
        with open("full_dialogue.wav", "wb") as f:
            f.write(b"RIFF")
            f.write(struct.pack("<I", 36 + len(all_pcm)))
            f.write(b"WAVEfmt ")
            f.write(struct.pack("<IHHIIHH", 16, 1, 1, 32000, 64000, 2, 16))
            f.write(b"data")
            f.write(struct.pack("<I", len(all_pcm)))
            f.write(all_pcm)

        print(f"\nSaved full_dialogue.wav ({len(all_pcm) / 2 / 32000:.1f}s)")

        await ws.send(json.dumps({"event": "task_finish"}))

asyncio.run(multi_voice_dialogue())

import asyncio
import json
import websockets

async def simple_tts():
    async with websockets.connect("ws://localhost:8080/ws/v1/tts") as ws:
        await ws.recv()  # connected_success

        await ws.send(json.dumps({
            "event": "task_start",
            "voice_setting": {"voice_id": "doubao", "speed": 1.0},
            "audio_setting": {"format": "wav"}
        }))
        await ws.recv()  # task_started

        # Send multiple sentences (processed sequentially)
        for text in ["第一句话。", "第二句话。", "第三句话。"]:
            await ws.send(json.dumps({
                "event": "task_continue",
                "text": text
            }))

            audio = b""
            while True:
                resp = json.loads(await ws.recv())
                hex_data = resp.get("data", {}).get("audio", "")
                if hex_data:
                    audio += bytes.fromhex(hex_data)
                if resp.get("is_final"):
                    break

            print(f"{len(audio)} bytes for: {text}")

        await ws.send(json.dumps({"event": "task_finish"}))

asyncio.run(simple_tts())

const WebSocket = require('ws');

const ws = new WebSocket('ws://localhost:8080/ws/v1/tts');

ws.on('open', () => {
  console.log('Connected');
});

ws.on('message', (data) => {
  const msg = JSON.parse(data);

  switch (msg.event) {
    case 'connected_success':
      // Configure voice
      ws.send(JSON.stringify({
        event: 'task_start',
        voice_setting: { voice_id: 'doubao', speed: 1.0 },
        audio_setting: { format: 'wav' }
      }));
      break;

    case 'task_started':
      // Start synthesizing
      ws.send(JSON.stringify({
        event: 'task_continue',
        text: '你好，这是一个测试。'
      }));
      break;

    case 'task_progress':
      const audioHex = msg.data.audio;
      const audioBuffer = Buffer.from(audioHex, 'hex');
      // Process audio chunk (e.g., write to file, play back)
      console.log(`Received ${audioBuffer.length} bytes (final: ${msg.is_final})`);

      if (msg.is_final) {
        ws.send(JSON.stringify({ event: 'task_finish' }));
      }
      break;

    case 'error':
      console.error('Error:', msg.message);
      break;
  }
});

POST /v1/chat/completions

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "temperature": 0.7,
    "max_tokens": 256
  }'

Response:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1706000000,
  "model": "qwen3",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "The capital of France is Paris."
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 8,
    "total_tokens": 33
  }
}

POST /v1/audio/transcriptions

# Encode audio as base64
AUDIO_B64=$(base64 -i audio.wav)

curl http://localhost:8080/v1/audio/transcriptions \
  -H "Content-Type: application/json" \
  -d "{
    \"file\": \"$AUDIO_B64\",
    \"model\": \"paraformer\"
  }"

Response:

{
  "text": "Hello, this is a test.",
  "language": "zh",
  "duration": 2.5
}

POST /v1/images/generations

# Text-to-image with Z-Image (faster)
curl http://localhost:8080/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A beautiful sunset over the ocean, digital art",
    "model": "zimage",
    "size": "512x512",
    "n": 1
  }'

# Text-to-image with FLUX.2-klein (higher quality)
curl http://localhost:8080/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A cat sitting on a windowsill",
    "model": "flux",
    "size": "1024x1024"
  }'

Response:

{
  "created": 1706000000,
  "data": [{
    "b64_json": "/9j/4AAQSkZJRg...",
    "revised_prompt": "A beautiful sunset over the ocean, digital art"
  }]
}

# Encode reference image as base64
IMG_B64=$(base64 -i reference.png)

curl http://localhost:8080/v1/images/generations \
  -H "Content-Type: application/json" \
  -d "{
    \"prompt\": \"Make it look like a watercolor painting\",
    \"model\": \"zimage\",
    \"size\": \"512x512\",
    \"image\": \"$IMG_B64\",
    \"strength\": 0.75
  }"

The server supports one model per category (LLM, ASR, TTS, Image). Models can be loaded, switched, and unloaded at runtime without restarting the server.

┌─────────────────────────────────────────────────────────┐
│                   Model Slots                            │
├─────────────┬─────────────┬─────────────┬───────────────┤
│     LLM     │     ASR     │     TTS     │    Image      │
│  (1 slot)   │  (1 slot)   │  (1 slot)   │   (1 slot)    │
├─────────────┼─────────────┼─────────────┼───────────────┤
│ Qwen/Mistral│  Paraformer │  GPT-SoVITS │  FLUX/Z-Image │
└─────────────┴─────────────┴─────────────┴───────────────┘

curl http://localhost:8080/v1/models/status

Response:

{
  "status": "success",
  "models": {
    "llm": "mlx-community/Qwen2.5-7B-Instruct-4bit",
    "asr": null,
    "tts": null,
    "image": "zimage"
  }
}

# Load LLM
curl http://localhost:8080/v1/models/load \
  -H "Content-Type: application/json" \
  -d '{"model": "mlx-community/Qwen2.5-7B-Instruct-4bit", "model_type": "llm"}'

# Load ASR
curl http://localhost:8080/v1/models/load \
  -H "Content-Type: application/json" \
  -d '{"model": "/path/to/paraformer/model", "model_type": "asr"}'

# Load TTS (provide reference audio)
curl http://localhost:8080/v1/models/load \
  -H "Content-Type: application/json" \
  -d '{"model": "/path/to/reference/audio.wav", "model_type": "tts"}'

# Load Image model
curl http://localhost:8080/v1/models/load \
  -H "Content-Type: application/json" \
  -d '{"model": "zimage", "model_type": "image"}'

# Unload one model type
curl http://localhost:8080/v1/models/unload \
  -H "Content-Type: application/json" \
  -d '{"model_type": "llm"}'

# Unload all models
curl http://localhost:8080/v1/models/unload \
  -H "Content-Type: application/json" \
  -d '{"model_type": "all"}'

Models download automatically from HuggingFace. Recommended:

# Download model files
huggingface-cli download funaudiollm/paraformer-large-mlx --local-dir ./models/paraformer

Required files:

models/paraformer/
├── paraformer.safetensors   # Model weights
├── am.mvn                   # CMVN normalization
└── tokens.txt               # Vocabulary (8404 tokens)

GPT-SoVITS uses few-shot voice cloning. Provide a reference audio file:

# Reference audio requirements:
# - WAV format, 16kHz or higher
# - 3-10 seconds of clean speech
# - Single speaker, minimal background noise

TTS_REF_AUDIO=./audio/reference.wav cargo run --release

The voice registry (voices.json) is loaded automatically if present. See the Voice Registry section for configuration.

Image models download automatically from HuggingFace. Choose between:

FLUX.2-klein (MLX format):

Z-Image-Turbo (MLX format):

Original Models (for reference):

# Use specific model (auto-downloads if not present)
IMAGE_MODEL=flux cargo run --release    # Use FLUX.2-klein
IMAGE_MODEL=zimage cargo run --release  # Use Z-Image-Turbo

# Use custom local model path (optional)
FLUX_MODEL_DIR=/path/to/flux-model cargo run --release
ZIMAGE_MODEL_DIR=/path/to/zimage-model cargo run --release

# Download FLUX.2-klein
huggingface-cli download black-forest-labs/FLUX.2-klein-4B --local-dir ./models/flux

# Download Z-Image-Turbo
huggingface-cli download uqer1244/MLX-z-image --local-dir ./models/zimage

# Or using git lfs
git lfs install
git clone https://huggingface.co/black-forest-labs/FLUX.2-klein-4B ./models/flux
git clone https://huggingface.co/uqer1244/MLX-z-image ./models/zimage

models/flux/                          # FLUX.2-klein
├── transformer/
│   └── diffusion_pytorch_model.safetensors
├── text_encoder/
│   ├── model-00001-of-00002.safetensors
│   └── model-00002-of-00002.safetensors
├── vae/
│   └── diffusion_pytorch_model.safetensors
└── tokenizer/
    └── tokenizer.json

models/zimage/                        # Z-Image-Turbo
├── transformer/
│   └── model.safetensors
├── text_encoder/
│   └── model.safetensors
├── vae/
│   └── diffusion_pytorch_model.safetensors
└── tokenizer/
    └── tokenizer.json

OminiX-API includes a capability versioning system. Each OminiX-MLX model crate declares its capabilities in an ominix.toml manifest. At build time, these are compiled into a registry that apps can query at runtime or validate against at startup.

GET /v1/version

Returns all available capabilities, their versions, and currently loaded models:

curl http://localhost:8080/v1/version

{
  "ominix_api": "0.1.0",
  "capabilities": {
    "qwen3-tts": {
      "name": "qwen3-tts",
      "version": "1.0.0",
      "category": "tts",
      "description": "Qwen3 TTS with preset voices and voice cloning",
      "capabilities": ["streaming", "voice_cloning", "preset_voices"]
    },
    "qwen3-asr": {
      "name": "qwen3-asr",
      "version": "0.1.0",
      "category": "asr",
      "description": "Qwen3 ASR speech recognition",
      "capabilities": ["multilingual"]
    },
    "flux-klein": {
      "name": "flux-klein",
      "version": "0.1.0",
      "category": "image",
      "description": "FLUX.2-klein image generation",
      "capabilities": ["quantization", "text_to_image"]
    }
  },
  "models_loaded": []
}

Apps can declare their requirements in an ominix.toml file. OminiX-API validates these at startup when launched with --app-manifest:

ominix-api --app-manifest my-app.ominix.toml
# or via environment variable:
OMINIX_APP_MANIFEST=my-app.ominix.toml ominix-api

Example manifest:

[app]
name = "my-voice-bot"
version = "0.1.0"
ominix_api = ">=0.1.0"

[requires]
tts = ">=1.0.0"
asr = ">=0.1.0"
llm = ">=0.1.0"

Requirements are checked by category (tts, asr, llm, image, vlm, ocr). Version constraints use semver syntax.

Advanced: require specific capabilities:

[app]
name = "voice-cloning-app"
version = "0.1.0"

[requires.tts]
version = ">=1.0.0"
[requires.tts.capabilities]
voice_cloning = true
streaming = true

If any requirement is not satisfied, the server exits with a clear error message listing what's missing.

Each model crate in OminiX-MLX has an ominix.toml describing what it provides:

# qwen3-tts-mlx/ominix.toml
[package]
name = "qwen3-tts"
version = "1.0.0"
category = "tts"
description = "Qwen3 TTS with preset voices and voice cloning"

[capabilities]
streaming = true
voice_cloning = true
preset_voices = true

These are read at build time by build.rs and compiled into the binary. When you update a crate's version or capabilities, just rebuild OminiX-API.

src/
  main.rs              Entry point — tracing, channels, thread spawning, server start
  config.rs            Config struct (from environment variables and CLI args)
  state.rs             AppState (shared across HTTP handlers)
  error.rs             render_error() helper for OpenAI-compatible error responses
  router.rs            Builds the Salvo Router with all endpoint routes
  version.rs           Capability registry, app manifest parsing, requirement validation

  inference/
    request.rs         InferenceRequest enum — the message protocol between handlers and inference thread
    thread.rs          inference_thread() — owns all models, processes requests via channel

  handlers/
    helpers.rs         get_state(), send_and_wait() — dedup boilerplate across handlers
    health.rs          GET /health, /v1/models, /v1/models/status, /v1/models/report
    version.rs         GET /v1/version — capability registry and version info
    chat.rs            POST /v1/chat/completions
    audio.rs           POST /v1/audio/transcriptions, /v1/audio/speech
    image.rs           POST /v1/images/generations, /v1/models/quantize
    models.rs          POST /v1/models/load, /v1/models/unload
    training.rs        Voice training endpoints + GET /v1/voices
    ws_tts.rs          WebSocket streaming TTS (MiniMax T2A protocol)

  engines/
    llm.rs             LLM inference — Qwen3, GLM-4.7-Flash backends
    asr.rs             ASR inference — Paraformer, SenseVoice+Qwen backends
    tts.rs             TTS synthesis — GPT-SoVITS with voice registry
    qwen3_tts.rs       Qwen3-TTS engine — streaming synthesis, voice cloning (x-vector)
    image.rs           Image generation — FLUX.2-klein, Z-Image-Turbo
    vlm.rs             Vision-Language Model inference

  types/
    chat.rs            ChatCompletionRequest/Response, ChatMessage, ChatUsage
    audio.rs           TranscriptionRequest/Response, SpeechRequest
    image.rs           ImageGenerationRequest/Response, ImageData
    training.rs        VoiceTrainRequest, TrainingStage, TrainingProgressEvent
    error.rs           ApiError, ApiErrorDetail
    voice.rs           VoiceInfo, VoiceListResponse

  model_config.rs      Model registry, hub cache scanning, availability checking
  training.rs          Voice cloning pipeline (5 stages on dedicated thread)
  utils.rs             Path utilities and security helpers

build.rs               Reads OminiX-MLX ominix.toml manifests, generates capability_registry.rs
examples/
  voice-bot.ominix.toml  Example app manifest for startup validation

To add a new capability (e.g., video generation):

1. engines/video.rs — Engine struct with new() and generate()
2. types/video.rs — Request/response types (add pub use in types/mod.rs)
3. inference/request.rs — Add Video + LoadVideoModel variants to the enum
4. inference/thread.rs — Add model slot + match arms (3 lines each with load_model_slot helper)
5. handlers/video.rs — Handler function (~15 lines using send_and_wait)
6. router.rs — Add one .push() line

┌──────────────────────────────────────────────────────────────┐
│                   HTTP/WS Server (Salvo)                      │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌────────┐│
│  │  /chat  │ │  /asr   │ │  /tts   │ │ /images │ │ ws/tts ││
│  │  (REST) │ │  (REST) │ │  (REST) │ │  (REST) │ │  (WS)  ││
│  └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ └───┬────┘│
│       │           │           │            │          │      │
│       └───────────┴───────────┴────────────┴──────────┘      │
│                               │                              │
│                   mpsc::channel (bounded: 32)                │
│                               │                              │
└───────────────────────────────┼──────────────────────────────┘
                                ▼
┌──────────────────────────────────────────────────────────────┐
│              Inference Thread (owns all models)               │
│                                                               │
│  ┌──────────────┐ ┌──────────────┐ ┌───────────────────────┐ │
│  │  LLM Slot    │ │  ASR Slot    │ │   TTS Slot (legacy)   │ │
│  │  Qwen3 /     │ │  Qwen3-ASR / │ │  GPT-SoVITS engine    │ │
│  │  Mistral /   │ │  Paraformer /│ │  + voice registry     │ │
│  │  (empty)     │ │   (empty)    │ │  + on-demand switching│ │
│  └──────────────┘ └──────────────┘ └───────────────────────┘ │
│                                                               │
│  ┌──────────────┐ ┌──────────────┐ ┌───────────────────────┐ │
│  │ Image Slot   │ │  VLM Slot    │ │  Qwen3-TTS Slot       │ │
│  │ Z-Image /    │ │  Qwen2.5-VL /│ │  CustomVoice (preset) │ │
│  │ FLUX /       │ │  (empty)     │ │  Base (voice cloning) │ │
│  │ (empty)      │ │              │ │  + streaming synthesis│ │
│  └──────────────┘ └──────────────┘ └───────────────────────┘ │
└──────────────────────────────────────────────────────────────┘

Key Design Points:

• Actor Model: MLX models don't implement Send/Sync, so all models run on a dedicated inference thread. HTTP and WebSocket handlers communicate via bounded async channels.

• One Model Per Slot: Each category has exactly one model slot. Loading a new model automatically unloads the previous one to free GPU memory.

• Shared Inference Path: Both the REST POST /v1/audio/speech and WebSocket ws/v1/tts endpoints use the same InferenceRequest::SpeechStream channel to the inference thread. Voice switching works identically for both.

• Streaming TTS: The default response format is raw PCM (16-bit signed LE, mono, 24kHz) streamed via chunked transfer encoding. Each chunk is ~800ms of audio generated by qwen3-tts-mlx's native streaming API (Synthesizer::start_streaming → StreamingSession::next_chunk). This eliminates timeout issues for long text — chunks flow continuously regardless of total audio length. Clients that need WAV can request it explicitly via ?format=wav query parameter or "response_format": "wav" in the JSON body.

• TTS Model Auto-Switching: Two Qwen3-TTS model variants exist — Base (with ECAPA-TDNN speaker encoder for voice cloning) and CustomVoice (with preset speakers like vivian, serena, ryan). The inference thread auto-switches between them based on request type: preset voice → CustomVoice, reference audio → Base. Models are discovered dynamically by scanning ~/.OminiX/models/ for directories containing "tts" and "base" or "customvoice" in the name.

• Dynamic Loading: Models can be loaded, switched, and unloaded at runtime via /v1/models/load and /v1/models/unload endpoints.

• Memory Efficient: Unused model slots remain empty. Unload models you're not using to free GPU memory for other tasks.

The current architecture uses a single inference thread that processes all requests sequentially via an mpsc channel. This means:

• A long TTS generation (30s-2min) blocks all other inference (chat, ASR, image)
• Concurrent TTS requests queue up — session B waits until session A finishes
• Streaming solves client timeouts but not head-of-line blocking

Workaround: Run separate instances per modality. For production deployments (e.g. crew gateway), run dedicated ominix-api instances for each service:

# ASR instance on port 8080
ASR_MODEL_DIR=~/.OminiX/models/Qwen3-ASR-1.7B-8bit PORT=8080 ./ominix-api

# TTS instance on port 8081
QWEN3_TTS_MODEL=~/.OminiX/models/Qwen3-TTS-12Hz-1.7B-CustomVoice-8bit PORT=8081 ./ominix-api

This ensures ASR and TTS never block each other. No code changes needed.

For high-concurrency TTS workloads, the planned architecture is a dynamic instance pool:

┌──────────────────────────────────────────────────────┐
│                  TTS Instance Pool                     │
│                                                        │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐               │
│  │ Engine 0│  │ Engine 1│  │ Engine N│  (spawned on   │
│  │ (always)│  │ (on     │  │  demand)│   demand when  │
│  │         │  │  demand)│  │         │   RAM allows)  │
│  └────┬────┘  └────┬────┘  └────┬────┘               │
│       └────────────┼────────────┘                     │
│                    │                                   │
│           Request Router                               │
│     (route to idle instance,                           │
│      spawn new if all busy                             │
│      and RAM available)                                │
└──────────────────────────────────────────────────────┘

• Each Qwen3-TTS 1.7B 8-bit instance uses ~2GB RAM
• On a 128GB machine, 3-5 concurrent TTS instances are feasible
• Instances are spawned on demand when all existing instances are busy
• Idle instances are reclaimed after a configurable timeout
• System RAM is checked before spawning (sysctl hw.memsize - current usage)

The server identifies the host chip at startup and reports it:

INFO Host: chip=M5 gpu_neural_accelerators=true asr_ane=enabled

Every Apple silicon Mac has a Neural Engine, but reaching it requires a Core ML port — MLX has no ANE backend on any generation. Through M4 that port is not worth making for ASR: the ANE cannot beat the GPU path at transformer decode, and running it alongside returns little. M5 changes the arithmetic, because its GPU gained Neural Accelerators and the two became genuinely separate units worth running at once. So ANE use is enabled from M5 onward and off before it.

ASR_USE_ANE=1|0 overrides in either direction, for benchmarking a generation the default excludes. Note that the ANE encoder-offload path is not yet implemented — this setting currently expresses policy and gates what a future offload scheduler is permitted to do.

Measured against LibriSpeech test-clean — 2,620 real utterances, 5.4 h of audio, scored for word error rate so the throughput numbers cannot come from degenerate output. Sessions are open-loop: each simulated speaker produces one second of audio per second of wall clock.

Recommended operating point: ~95 sessions across 3 processes (p50 598 ms). Saturation is abrupt — latency goes near-vertical within a few extra sessions — so leaving headroom matters more than squeezing out the last five channels. At 90 sessions the GPU runs at 84% mean utilisation (95% p90); memory is 5.1 GB total, nowhere near a constraint on 128 GB.

Batching throughput at 16 concurrent clients:

Batching improves latency as well as throughput here: at a fixed number of waiting clients the queue drains roughly twice as fast. Correctness is unaffected — across 48 utterances, transcripts produced in batches of 8 were identical to the same utterances transcribed one at a time.

Model variants, single stream:

4-bit is both faster and more accurate than 8-bit on this model — decode is memory-bandwidth-bound, so halving the weight bytes attacks the dominant cost.

Requires MLX 0.32.0 or newer. Earlier builds compute RoPE incorrectly at sequence length 1 with batch > 1, which is exactly the shape batched decode uses; the server probes for the defect at startup and falls back to single-request decoding rather than returning corrupt transcripts. See docs/asr-batching.md.

Benchmarks on Apple M3 Max (128GB):

import openai

client = openai.OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"  # No auth required
)

# Chat completion
response = client.chat.completions.create(
    model="qwen3",
    messages=[
        {"role": "user", "content": "Hello!"}
    ]
)
print(response.choices[0].message.content)

# Image generation
response = client.images.generate(
    model="zimage",
    prompt="A cat in space",
    size="512x512"
)
print(response.data[0].b64_json[:50] + "...")

MIT OR Apache-2.0