bx

bx is a model-independent, lazy tensor runtime for fast on-device inference. The runtime lives in bx; GGUF loading, tokenization, and model definitions live in the separately publishable bx-lm crate.

Galaxy S24 Ultra. Reproduce from the checked-in CSV with makers plot-vlm-pareto.

Mobile inference is not one homogeneous workload, so bx does not force every phase onto one device:

The default hybrid path therefore runs vision and prefill on Vulkan, then decode on CPU. This is an explicit phase boundary, not an automatic fallback. Benchmark runs disable fallback so an unavailable backend fails visibly.

The rest of the design follows the same principle: keep semantics simple, then specialize execution without changing results.

• Immutable lazy graphs memoize shared nodes and realize them once.
• Views such as reshape, transpose, slice, and broadcast are zero-copy.
• The scalar CPU kernels are the semantic oracle.
• Parallel CPU and Vulkan kernels are differential-tested against that oracle.
• Dtypes and device movement are explicit; there are no implicit casts.
• Models stay outside the runtime core, so bx is not tied to Qwen or GGUF.

Requirements: stable Rust 1.88+, cargo-make, Android API 26+ arm64, adb, and an Android NDK for device tasks.

makers setup          # Android target, verified qwen35-0.8b weights, environment check
makers info-device    # inspect the connected device
makers test-device    # CPU, Vulkan, and AHardwareBuffer tests

Weights are placed as real files under models/<name>/{model,mmproj}.gguf. makers setup always provides the certified default; more models are fetched by name (a verified local copy is reused, otherwise the pinned Hugging Face file downloads straight into place — SHA-256 checked either way):

makers setup lfm25-450m lfm25-1.6b   # add the LFM2.5-VL pairs
makers doctor                        # shows which models are ready

Run a VLM with the recommended hybrid route (makers run is the host-side CPU pair of run-device):

makers run-device --image ~/photo.jpg \
  --prompt "Please tell me about all texts written in photo." -n 128
makers run-device --model lfm25-450m -p "hi"   # another registered model
makers run --model lfm25-450m                  # same request on the host

Images are auto-oriented, aspect-fit, and letterboxed to the certified 512×512 vision grid. --image-size selects a different square grid, rounded to a multiple of patch_size × spatial_merge_size the way llama.cpp and MLX round; past 768 px the learned position grid is extrapolated. Output reports preprocess, vision, prefill/TTFT, and decode separately; model loading is excluded from phase timings.

Build the pinned llama.cpp comparison once, then measure both engines with the same image, prompt, models, token budget, and protocol:

makers compare-text-device   # first time only
makers compare-vlm-device    # 512×512, 32 output tokens, 5 warm-ups + 10 measurements

A short smoke run is useful while iterating:

makers compare-vlm-device --warmups 0 --repetitions 1 --decode-tokens 3

Artifacts are written below target/bench/<model>/vlm-comparison/: the SVG chart, Markdown table, and raw bx/llama.cpp JSON. The chart shows input pixels, image tokens, each engine's prompt-token count, output tokens, device, and protocol. Every bench task accepts --model <name> (default qwen35-0.8b).

TTFT is vision plus LLM prefill through the first token; bx also includes its few milliseconds of image preprocessing. Decode starts after that token. Use makers plot-vlm to redraw existing standalone benchmark JSON.

bx-lm is consumed as a git dependency; the whole request protocol is the Vlm pipeline. Complete synthetic-image examples are named for their model and run against the fetched weights with one command:

makers example qwen35-0.8b
makers example lfm25-450m
makers example lfm25-1.6b

(Each wraps cargo run --release -p bx-lm --example <name> -- <model.gguf> <mmproj.gguf>, so the example sources stay copy-pasteable.)

The library API is shared by both Qwen3.5-VL and LFM2.5-VL:

use bx_lm::{DeviceMode, Event, Request, Vlm};

let vlm = Vlm::load("model.gguf", Some("mmproj.gguf"), DeviceMode::Hybrid)?;
let out = vlm.generate(
    &Request::new("Describe this image.").image_rgb(&rgb, width, height),
    |event| if let Event::Token(t) = event { print!("{}", vlm.tokenizer().decode(&[t])) },
)?;
eprintln!("{}", out.timings);

The core API takes raw RGB (a camera buffer); the optional image-io feature adds JPEG/PNG/PPM file decoding.

makers show              # task menu
makers ci                # format, lint, tests (model-free)
makers ci-full           # ci plus the recorded goldens
makers test              # host tests
makers golden            # recorded text and image outputs
makers golden-lfm        # both LFM2.5-VL sizes (makers setup lfm25-450m lfm25-1.6b)
makers bench-device      # text prefill/decode benchmark (backend arg, --model)
makers bench-vlm-device  # VLM phase benchmark (backend arg, --model)

Benchmark artifacts land under target/bench/<model>/{text,vlm}-<backend>/.

Useful runtime controls are BX_WORKERS=n, BX_PIN=0, BX_FORCE_SCALAR=1, BX_Q4_ROWS8=0, and BX_VK_SUBMIT_GROUPS=n (work groups a Vulkan submission accumulates before it is sent, the bound that keeps one submission short of the mobile GPU watchdog). Device tasks forward them.

Repository map:

• crates/bx/ — tensor graph, storage, CPU kernels, and Vulkan backend
• crates/bx-lm/ — GGUF, tokenization, models, vision, and the Vlm pipeline
• crates/bx-cli/ — the bx binary: generate, bench, vlm-bench, gguf-info
• crates/bx-model-tests/ — model-shaped acceptance tests
• crates/xtask/ — host-side task runner behind makers: device orchestration, benchmarks, reports
• docs/HANDOFF.md — current performance breakdown and remaining work
• docs/baseline/ — recorded regression baselines

Licensed under the MIT license.