yzma - Go with your own intelligence

yzma lets you write Go applications that directly integrate llama.cpp for fully local inference using hardware acceleration.

• Run the latest Vision Language Models (VLM) and Large/Small/Tiny Language Models (LLM) on Linux, macOS, or Windows.
• Use any available hardware acceleration such as CUDA, Metal, OpenVINO, or Vulkan for maximum performance.
• Run in a browser as well, with TinyGo and WebAssembly, on the CPU or on the GPU with WebGPU.
• yzma uses the purego and ffi packages so CGo is not needed.
• Works with the newest llama.cpp releases so you can use the latest features, performance improvements, and bugfixes.

The documentation is at yzma.ai.

This example uses the SmolLM2-135M-GGUF model:

package main

import (
	"fmt"
	"os"
	"path/filepath"

	"github.com/hybridgroup/yzma/pkg/download"
	"github.com/hybridgroup/yzma/pkg/llama"
)

var (
	modelFile            = "SmolLM2-135M.Q4_K_M.gguf"
	prompt               = "Are you ready to go?"
	libPath              = os.Getenv("YZMA_LIB")
	responseLength int32 = 12
)

func main() {
	llama.Load(libPath)
	llama.LogSet(llama.LogSilent())

	llama.Init()

	model, _ := llama.ModelLoadFromFile(filepath.Join(download.DefaultModelsDir(), modelFile), llama.ModelDefaultParams())
	ctx, _ := llama.InitFromModel(model, llama.ContextDefaultParams())

	vocab := llama.ModelGetVocab(model)

	tokens := llama.Tokenize(vocab, prompt, true, false)

	batch := llama.BatchGetOne(tokens)

	sampler := llama.SamplerChainInit(llama.SamplerChainDefaultParams())
	llama.SamplerChainAdd(sampler, llama.SamplerInitGreedy())

	for pos := int32(0); pos < responseLength; pos += batch.NTokens {
		llama.Decode(ctx, batch)
		token := llama.SamplerSample(sampler, ctx, -1)

		if llama.VocabIsEOG(vocab, token) {
			fmt.Println()
			break
		}

		buf := make([]byte, 36)
		len := llama.TokenToPiece(vocab, token, buf, 0, true)

		fmt.Print(string(buf[:len]))

		batch = llama.BatchGetOne([]llama.Token{token})
	}

	fmt.Println()
}

Install yzma, then download the model using the yzma command line tool:

$ yzma model get -u https://huggingface.co/QuantFactory/SmolLM2-135M-GGUF/resolve/main/SmolLM2-135M.Q4_K_M.gguf

And run the Go program:

$ go run ./examples/hello/


"Yes, I'm ready to go."

go install github.com/hybridgroup/yzma@latest
yzma install --lib /path/to/lib
export YZMA_LIB=/path/to/lib

The yzma command line tool downloads the llama.cpp prebuilt libraries for your platform. Your application can also download them itself, with auto-detection for CUDA and ROCm.

Each file that comes down is checked against the SHA-256 digest that the release publishes, and the yzma verify command checks an installation later.

See yzma.ai/getting-started/install for the instructions for macOS, Linux, Windows, Raspberry Pi, NVIDIA Jetson Orin, the Arduino UNO Q, and a browser.

We have several examples of how you can use yzma in our examples directory.

This example uses the Qwen2.5-VL-3B-Instruct-Q8_0 VLM model to process both a text prompt and an image, then displays the result.

$ go run ./examples/vlm/ -model ~/models/Qwen2.5-VL-3B-Instruct-Q8_0.gguf -mmproj ~/models/mmproj-Qwen2.5-VL-3B-Instruct-Q8_0.gguf -image ./images/domestic_llama.jpg -p "What is in this picture?"

The image features a white llama standing in a fenced-in area, possibly a zoo or a farm. The llama is positioned in the center of the image, with its body facing the right side. The fenced area is surrounded by trees, creating a natural environment for the llama.

See the code here.

You can use yzma to do inference on text language models. This example uses the qwen2.5-0.5b-instruct-fp16.gguf model for an interactive chat session.

$ go run ./examples/chat/ -model ./models/qwen2.5-0.5b-instruct-fp16.gguf
Enter prompt: Are you ready to go?

Yes, I'm ready to go! What would you like to do?

Enter prompt: Let's go to the zoo


Great! Let's go to the zoo. What would you like to see?

Enter prompt: I want to feed the llama


Sure! Let's go to the zoo and feed the llama. What kind of llama are you interested in feeding?

See the code here.

yzma also runs in a browser, with text or with images. llama.cpp becomes a WebAssembly module of its own, and a Go program compiled by TinyGo drives it through the pkg/llamawasm package. The model never leaves the machine of the reader, and no server does any of the work.

$ make download-llama.cpp-wasm
$ make wasm-example
$ make wasm-vlm-example
$ make serve-wasm

Then open http://localhost:8080 for chat, or http://localhost:8080/vlm.html to ask a question about an image. Each page says which backend it got:

backend: webgpu (WebGPU)

There are three builds of llama.cpp, and the JavaScript glue takes the best one the browser can run:

So a browser without WebGPU still works, on the CPU. The GPU is worth the most to a page that takes images: putting one through the projector of a model takes a second or two on the GPU against half a minute or more on the CPU.

The API in a browser is the smaller one of pkg/llamawasm: the calls that text generation, embeddings, and images need. The names and the order of the calls are the same as in pkg/llama and pkg/mtmd, so a program moves over with a change of the import.

See the code here, or the one that takes an image.

See wasm/README.md for how it works, what each build costs in speed, and what a page must do to use one.

See the examples directory for more examples of how to use yzma.

Who is using yzma? Check out some of the tools, applications, examples, and blog posts and videos!

yzma uses models in the GGUF format supported by llama.cpp. There are many models in GGUF format on Hugging Face (over 201k at last count):

https://huggingface.co/models?library=gguf&sort=trending

You can use the yzma command to download models for you!

For example, this downloads the gemma-3-1b-it-GGUF model:

$ yzma model get -u https://huggingface.co/ggml-org/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-Q4_K_M.gguf

Check out the Models page for the download command and the run command for each model.

yzma currently has support for over 96% of llama.cpp functionality. See ROADMAP.md for the complete list.

You can use multimodal models (image/audio) and text language models with full hardware acceleration on Linux, macOS, and Windows.

A browser is also a target:

There the API is the smaller one of the pkg/llamawasm package: text generation, embeddings, and images, with no audio, video, LoRA adapters, saved state, or quantization. See the WebAssembly example above and wasm/README.md.

Whenever there is a new release of llama.cpp, the tests for yzma are run automatically. This helps us stay up to date with the latest code and models.

Sometimes there are breaking changes to llama.cpp that require an update to yzma. Here are the known compatible versions for tagged releases:

A tagged release of yzma installs its own llama.cpp release by default, so yzma install without the -version flag gets the version in this table. Use -version latest to get the most recent nightly build instead. A build from the main branch always uses the most recent nightly build.

Here are some of the known compatible versions for the nightly builds of llama.cpp:

yzma is fast because it calls llama.cpp in the same process. No external servers needed!

For example, here is the Qwen3-VL-2B-Instruct Visual Language Model (VLM) performing multi-modal inference on an image and text prompt running on a Apple M4 Max with 128 GB RAM:

$ go test -run none -benchtime=10s -count=5 -bench BenchmarkMultimodalInference
goos: darwin
goarch: arm64
pkg: github.com/hybridgroup/yzma/pkg/mtmd
cpu: Apple M4 Max
BenchmarkMultimodalInference-16		10		1577948683 ns/op	788.9 tokens/s
BenchmarkMultimodalInference-16		12		1243692014 ns/op	910.8 tokens/s
BenchmarkMultimodalInference-16		 7		1654741804 ns/op	737.2 tokens/s
BenchmarkMultimodalInference-16		 7		1568106947 ns/op	771.9 tokens/s
BenchmarkMultimodalInference-16		10		1704669371 ns/op	706.1 tokens/s
PASS
ok  	github.com/hybridgroup/yzma/pkg/mtmd	76.644s

Want to see more benchmarks? Take a look at the BENCHMARKS.md document, which has one file for each platform. To make these numbers again on your own machine, run ./benchmarks/run.sh. See how to run the benchmarks.

yzma is now ready to be used to build complete applications that incorporate language models directly into your Golang code.

Here are some advantages of yzma with llama.cpp:

• Compile Go programs that use yzma with the normal go build and go run commands. No C compiler needed!
• Use the llama.cpp libraries with whatever hardware acceleration is available for your configuration. CUDA, Vulkan, etc.
• High performance from making function calls from within the same process. No external model servers!
• Download llama.cpp precompiled libraries directly from Github, or include them with your application.
• Update the llama.cpp libraries without recompiling your Go program, as long as llama.cpp does not make any breaking changes.

The idea is to make it easier for Go developers to use language models as part of "normal" applications without having to use containers or do anything other than the normal GOOS and GOARCH env variables for cross-complication.

yzma originally started with definitions from the https://github.com/dianlight/gollama.cpp package, but then has gone on to modify them rather heavily. Thank you!