Omega

Omega is a Windows-native, Go-first sparse runtime and coding-agent host for MiniMax-M3 GGUF models that are larger than physical RAM.

The target profile is deliberately specific:

• AMD Ryzen 9 7900X
• NVIDIA RTX 4080 16 GiB (SM89)
• 128 GiB host RAM
• MiniMax-M3-UD-IQ4_XS split GGUF (208 GB)
• Windows 11, Go 1.26, Visual Studio 2022, CUDA 13.3

Go owns GGUF indexing, the expert store and cache, scheduling, protocol rendering, tools, sessions, the HTTP API, and telemetry. Native code is exposed through a versioned C ABI for page-aligned memory and optimized CPU/CUDA kernels.

Omega now has a complete correctness-first text forward path and an integrated hybrid CUDA path. The following pieces are implemented and tested:

• exact multi-shard GGUF indexing and model fingerprint validation;
• resumable/checksummed 4 KiB-aligned expert-store preparation;
• configurable native 2Q expert cache (80 GiB dedicated default), route-frequency persistence, IOCP reads, and buffered-I/O fallback;
• versioned DLL loading, AVX-512/VNNI GGML kernels, SM89 CUDA allocation, resident Q8 matvec, RMSNorm, partial NeoX RoPE, OAI SwiGLU, residual, and Q8 KV/GQA attention kernels;
• exact dense-GQA/OAI graph execution, real routed/shared-expert execution, previous-route prefetch, expert-major GPU prefill, direct Q6_K/IQ3_S/IQ4_XS CUDA MMQ, placement planning, tuning, and telemetry;
• byte-preserving streaming of the final four shared-expert layers through one reusable Q8_0 device buffer, preserving WDDM headroom without changing model weights;
• two-slot expert pipelining with CUDA-pinned native staging and nonblocking transfer streams, with no Go-managed pointer retained by asynchronous work;
• 32K Q8 host/GPU KV caches with the model's orthonormal Hadamard rotations, chunked 1,024-token prefill, and exact reusable prefix-KV truncation across requests;
• the MiniMax tokenizer, exact prompt rendering, incremental think/tool XML parsing, Go sampling/generation, OpenAI-compatible API, sessions, and trusted-host coding tools;
• the embedded Omega Workbench with resumable real-time reasoning, response, tool, Plan, Goal, and runtime event streams.

omega serve --backend native is functional. When the dedicated VRAM profile is available it loads the dense graph on CUDA and overlaps the GPU shared expert with CPU routed experts. If conflicting GPU applications leave too little VRAM, it starts the much slower CPU correctness fallback and says so explicitly. The exact pinned llama.cpp differential matches four greedy decode tokens, the top-three first-token ordering, and nine of the first-token top ten. Real-model CPU/GPU oracles, the AVX2 fallback suite, race tests, and CUDA Compute Sanitizer pass. The full 80 GiB/32K dedicated release suite passes: its three measured runs produced all 256 requested tokens, with a 13.98 prompt tok/s median, a 1.83 generation tok/s median, an 81.92 GiB peak host working set, and a 13.88 GiB peak runtime VRAM footprint. The llama-http backend remains a compatibility path.

The release benchmark refuses to report success unless the native sparse backend reaches both 10 prompt tokens/s and 1.5 generated tokens/s while exposing the 80 GiB expert cache, 32K context, CUDA hybrid path, utilization telemetry, an observed 1,024-token rendered prefill after an exact rendered 8K seed, and the 84 GiB host / 14 GiB VRAM limits. The cache target leaves headroom inside the memory Windows actually makes available rather than relying on nominal installed RAM. Compatibility, reduced-profile, CPU-only, or mock backends never satisfy that gate.

omega doctor treats existing CUDA or inference processes as dedicated-profile failures and reports their identities, but never stops them. Shut those processes down manually before a release run.

git submodule update --init --recursive
.\scripts\build.ps1

The Go binary is written to bin\omega.exe. The native libraries are bin\omega-kernels.dll (AVX2 fallback) and bin\omega-kernels-avx512.dll (AVX-512/VNNI). The Go loader opens the safe base DLL first, checks CPUID, and selects the optimized sibling when supported.

The build script explicitly selects Visual Studio 2022 and CUDA 13.3, generates SM89 code, builds both CPU variants, runs the Go tests, and then builds the CLI. The current native ABI is version 11.

$model = "$env:USERPROFILE\.lmstudio\models\unsloth\MiniMax-M3-GGUF"

.\bin\omega.exe doctor --model $model
.\bin\omega.exe prepare --model $model
.\bin\omega.exe tune --model $model
.\bin\omega.exe serve --model $model --workspace C:\Projects\MyProject

Open http://127.0.0.1:8080/ for the local Omega Workbench. Its workspace is canonicalized and fixed for the life of the server. The UI is available only on a loopback listener.

For a functional bridge to a currently running llama.cpp-compatible server:

.\bin\omega.exe serve --model $model --backend llama-http `
  --upstream http://127.0.0.1:18000 `
  --workspace C:\Projects\MyProject

Start the built-in agent:

.\bin\omega.exe agent --workspace C:\Projects\MyProject `
  --endpoint http://127.0.0.1:8080

shell.exec is a trusted host shell, not a security sandbox. Commands run with the current Windows user's permissions. Omega supplies cancellation, timeouts, output bounds, a workspace working directory, and process-tree cleanup.

• Chat uses the trusted filesystem and host-shell tools for an interactive coding-agent turn.
• Plan is enforced read-only by the server. It exposes workspace inspection, allowlisted Git/system diagnostics, and immutable plan publishing, but never exposes filesystem writes, patches, or the trusted shell.
• Goal implements an accepted plan through a persistent autonomous loop. Goals journal checkpoints and evidence, survive restart, and support pause/resume/cancel. Default guards pause after 60 active minutes, 24,000 generated tokens, or 64 tool calls; reaching a guard is not treated as failure.

Model reasoning is displayed as a separately labelled, collapsible stream. It is model output rather than an infallible execution trace. Tool calls, terminal output, queue state, token rates, context use, RAM, and VRAM telemetry update alongside the visible response.

• omega doctor: inspect model topology and machine readiness.
• omega prepare: build the aligned, checksummed expert store.
• omega tune: benchmark and persist machine-specific tuning.
• omega bench --suite release: run the gated benchmark.
• omega serve: expose the OpenAI-compatible API and embedded local Workbench.
• omega agent: run the terminal coding agent and its workspace tools.

Run omega <command> --help for command-specific options.

.\scripts\test.ps1

.\scripts\test.ps1 -Actual `
  -ModelPath $model `
  -StorePath "$env:LOCALAPPDATA\Omega\models\<fingerprint>\experts.omegaexp"

The actual-model suite checks every quantization type present in the GGUF against scalar references, executes cached single/top-4 experts, probes the late mixed-quant layers that use Q6_K routed projections, compares a real GPU transformer layer against the CPU graph, and runs a 60-layer token forward. It also repacks byte-exact real tensor rows into randomized one-, two-, and three-block edge shapes across several CPU thread counts and each CUDA-capable quantization. The suite covers pinned asynchronous transfers, both tuned decode routes, process-tree cancellation, bounded shell output, streaming disconnects, and session restart recovery. When CUDA Compute Sanitizer is installed with the toolkit, the script runs the randomized CUDA matrix and the remaining CUDA kernels under memcheck.

Build the exact correctness oracle and include it in the differential suite:

.\scripts\build-reference.ps1

.\scripts\test.ps1 -Actual `
  -ModelPath $model `
  -StorePath "$env:LOCALAPPDATA\Omega\models\<fingerprint>\experts.omegaexp" `
  -LlamaEndpoint http://127.0.0.1:18001 `
  -LlamaCommit 0b78558a6f3e57830caffe2afb5000f4ed9e187a