ZML - Model to Metal

Laguna S 2.1

Tue, 21 Jul 2026 19:00:00 +0100

In close collaboration with Poolside, we’re happy to announce the support of Laguna S 2.1, their new open-weight model for agentic coding and long-horizon work.

It is a 118B total parameters Mixture-of-Experts model with 8B activated parameters per token and supports a context window of up to 1M tokens, available in thinking and no-thinking modes.

And it’s available today in ZML/LLMD with DFlash support. Most importantly (and as always with us), it’s available on all the supported platforms: NVIDIA, AMD, Intel, Google TPU and Apple Metal.

ZML/LLMD alpha

Wed, 08 Jul 2026 08:00:00 +0100

Today we’re releasing ZML/LLMD. It’s a self-contained inference server that runs LLaMa, Gemma, Qwen and Mistral LLMs transparently on 5 architectures: NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI and Apple Metal.

Modern serving features

ZML/LLMD supports modern serving features: continuous batching, paged attention, tensor parallel sharding, prefix caching, tool calling and does so on all platforms.

Metrics are also exposed in the Prometheus format via the /metrics endpoint.

10x faster tokenization

Tue, 07 Apr 2026 09:00:00 +0100

Today we merged a pretty nice performance improvement to ZML.

PR #416 switches zml.tokenizer.Tokenizer from using the Hugging Face tokenizers crate to using the IREE project tokenizer when using Hugging Face tokenizer.json files.

The (upcoming) problem

Tokenizers performance is often overlooked, but it can have a significant impact on the overall latency of LLM inference. In some cases, it can even become the bottleneck of the entire system. This is particularly important as context are becoming bigger and bigger. For code generation, a 1M context window is regular. But a 1M context means 4MB of data (u32).

Introducing zml-smi

Mon, 30 Mar 2026 16:00:00 +0100

zml-smi is a universal diagnostic and monitoring tool for GPUs, TPUs and NPUs. It provides real-time insights into the performance and health of your hardware.

It is a mix between nvidia-smi and nvtop.

It transparently supports all the platforms ZML supports. That is NVIDIA, AMD, Google TPU and AWS Trainium devices. It will be extended to support more platforms in the future as ZML continues to expand its hardware support.

Introducing ZML/v2

Tue, 24 Mar 2026 16:00:00 +0100

ZML is an inference stack built close to the hardware. It lowers models directly onto NVIDIA, AMD, TPU, and Trainium targets from a single codebase, without depending on and suffering from the Python-heavy runtime layers that most of the ecosystem is built around.

The guiding idea behind zml/v1 was simplicity: give ZML a model and its weights, and the system would take care of compilation, placement, and execution for you. That made the first version approachable and effective for standard deployments, but it also baked too much behavior into implicit global state. As the project pushed into partial compilation, custom passes, sharding, quantization, and more backend-specific execution paths, those implicit shortcuts became constraints. ZML/v2 is the rewrite that makes those concepts explicit: platform ownership, compilation, memory, IO, and placement are now first-class, so advanced use cases can be expressed directly instead of forced through workarounds.

About

Mon, 01 Jan 0001 00:00:00 +0000

The ZML Blog is a technical publication about running modern AI systems in production.

We write about:

inference systems
compiler architecture
hardware portability
deployment ergonomics
observability and operating discipline

The editorial bias is simple: practical speed, maintainable systems, and fewer hidden compromises.

ZML/LLMD alpha

Mon, 01 Jan 0001 00:00:00 +0000