TyloQuant MFQ

Next-generation quantization & inference infrastructure

Every Bit. Maximum Fidelity.

MFQ combines neural-network-aware SQ and VQ formats designed to approach the practical rate-distortion frontier, high-quality, fine-grained mixed-precision calibration and quantization, and efficient CUDA/Metal inference. The goal is to deliver the best model quality possible on existing hardware within its memory, storage, and latency limits.

English · 中文

DeepSeek V4.1 · Quick start · Features · Models · Benchmarks · Documentation

Run a 476 GB DeepSeek V4.1 Flash checkpoint directly from Hugging Face Safetensors on a single 512 GB Apple M3 Ultra—without converting it to an MFQ container first. The tuned Metal path keeps the backbone and MoE experts resident in unified memory while offloading only Engram storage to the Mac's internal SSD.

Measured locally at batch size 1 with a 32,768-token context, temperature 0, and warmed execution. Prefill uses representative 509-, 2,009-, and 15,969-token prompts. The process used approximately 287 GiB RSS after load; the 440 GiB wired budget is a ceiling rather than steady-state usage. The high-acceptance MTP result accepted 132/132 drafted tokens and matched the non-MTP output exactly; gains remain workload-dependent.

See the full protocol, lower-acceptance fallback result, and current limitations.

MFQ handles the full path from a source checkpoint to a deployable packed model. It measures activation and loss sensitivity, assigns precision at tensor, expert, and projection granularity under an exact serialized-size budget, stores the result in a self-contained .mfq container, and executes the packed weights with optimized C++ kernels. The formats and runtime are co-designed so fidelity-per-bit gains carry through to fast CUDA/Metal execution and one-command serving.

The simplest Apple silicon path is a prebuilt MFQ Studio package. Prebuilt packages are published through the project Releases when available and bundle the desktop console, MFQ Server, and C++ runtime. Open Studio, register a model directory, and load any supported .mfq model.

Prebuilt C++ workers can also be supplied directly to the server:

uv run mfq serve \
  --running-executable /path/to/mfq-decode-metal \
  --model /models/model.mfq

MFQ requires Git, uv, CMake 3.26+, and a C++ toolchain. Use CUDA 12+ on NVIDIA systems or Metal on Apple silicon.

git clone https://github.com/Tylogi/TyloQuant.git MFQ
cd MFQ

# NVIDIA / CUDA
uv sync
uv run mfq build --backend cuda

# Apple silicon / Metal
uv sync --extra metal
uv run mfq build --backend metal

Start an empty local server and open http://127.0.0.1:8090/:

uv run mfq serve

Detailed requirements and custom CMake options are in the mfq build guide.

Use the MFQ S4-M mixed-precision preset. It keeps sensitive tensors at higher precision and leaves Vision and MTP components at source precision by default.

# Quantization dependencies
uv sync --extra train

# Hugging Face safetensors -> self-contained MFQ
uv run mfq quantize \
  /models/Qwen3.8-27B \
  /models/Qwen3.8-27B-MFQ-S4-M.mfq \
  --preset S4-M \
  --backend auto

Load and serve the result:

uv sync                                # add --extra metal on Apple silicon
uv run mfq build --backend auto
uv run mfq serve \
  --model /models/Qwen3.8-27B-MFQ-S4-M.mfq \
  --context-size 32768

MFQ can also quantize full-precision MFQ and GGUF sources, consume per-tensor or Expert-Wise precision maps and activation importance matrices, and write sharded output. See mfq quantize and mfq calibrate.

• Fine-Grained Quantization & Precision Virtualization. MFQ virtualizes Expert-Wise and Sub-Tensor precision choices inside a single logical tensor container, producing clean, unified quantization formats instead of exposing every internal configuration as another dtype. A descriptor-driven JIT lowers these metadata-described tensors into compact execution plans and production kernels, keeping the runtime robust, simple, and efficient. Calibration and quantization can therefore search a much larger space—down to each expert, projection, chunk, or neuron—without multiplying execution paths.

• High-Quality Calibrated Models & Rapid Architecture Support. MFQ publishes fine-grained mixed-precision models, evaluates them against their source models under consistent model-level protocols, and rapidly extends both quantization and runtime support to important new architectures.

• Production Inference & Serving. One native serving stack provides Continuous Batching, tiered RAM/SSD Prefix and KV-cache management, multi-model loading and lifecycle management with LRU, and SSD-streamed expert execution for models beyond memory capacity. Scheduling, prefetch, and cache policies remain shared runtime capabilities rather than separate model-specific serving paths.

• High-Performance CUDA & Metal Backends. Backend-specific packed-weight, attention, cache, and I/O kernels sit behind a shared C++ model graph and runtime contract. CUDA and Metal are supported today; optimized ROCm and CPU backends are planned.

MFQ keeps heterogeneous MoE execution efficient at both supported precision granularities:

Warm end-to-end throughput on Apple M5 Max (40-core GPU, 128 GB).

Published .mfq models are available from Hugging Face and ModelScope.

The C++ runtime currently covers the following architecture groups. Exact coverage depends on checkpoint revision, embedded components, and backend.

See the runtime support matrix before deploying a specific artifact.

MFQ exposes several interfaces for applications and tooling:

• HTTP control API under /api/v1 for models, sessions, responses, jobs, datasets, evaluations, caches, and runtime state;
• Server-Sent Events for token and job streaming;
• WebSocket realtime APIs for audio and full-duplex sessions;
• OpenAI-compatible /v1/models and /v1/chat/completions endpoints from managed C++ workers and runtime adapters;
• MCP and function-tool execution through the server tool registry.

Start with the HTTP API, WebSocket API, or mfq serve reference.

The evaluation uses the official 0731 weights and a fixed WikiText-2 protocol covering 573 chunks and 146,115 scored tokens at ctx=512.

Against the nearest-size Unsloth Dynamic baselines used in this evaluation, MFQ lowers Mean KLD by 34.24–51.42%.

All plotted tiers use the same BF16 teacher and the complete 145-chunk, 148,335-scored-token evaluation. MFQ records lower raw Mean KLD at every matched precision point shown.

• Documentation index
• Build the C++ runtime
• Quantize a model
• Run MFQ Server
• Runtime support
• MiniCPM-o 4.5 multimodal runtime
• Contributing and architecture rules

MFQ is deeply inspired by and has learned from llama.cpp, oMLX, MLX, PyTorch, Transformers, and Unsloth.

MFQ is licensed under the Apache License 2.0.