Skip to content

llama.cpp: build every ggml CPU variant and pick one at run time - #1559

Draft
bryan-minimal wants to merge 2 commits into
mainfrom
llama-cpu-variants
Draft

bryan-minimal wants to merge 2 commits into
mainfrom
llama-cpu-variants

Conversation

@bryan-minimal

Copy link
Copy Markdown
Member

Fixes #975.

What

llama.cpp now builds with ggml's runtime CPU dispatch, as upstream's release builds do (.github/workflows/release.yml at b12359):

-DGGML_NATIVE=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_BACKEND_DIR=/usr/lib/ggml

ggml-cpu is built once per variant as libggml-cpu-<variant>.so. At startup ggml_backend_load_all() dlopens each one, calls its ggml_backend_score() (a feature probe built without the variant's flags), and keeps the highest-scoring one.

Variants at b12359 (ggml/src/CMakeLists.txt:494-533, Linux):

arch variants
arm64 (8) armv8.0_1, armv8.2_1 (+dotprod), armv8.2_2 (+fp16), armv8.2_3 (+sve), armv8.6_1 (+i8mm), armv8.6_2 (+sve2), armv9.2_1 (+sme), armv9.2_2 (+sve2+sme)
x86_64 (14) x64, sse42, sandybridge, ivybridge, piledriver, haswell, skylakex, cannonlake, cascadelake, icelake, cooperlake, zen4, alderlake, sapphirerapids

Choices

  • CFLAGS baseline. $MARCH is now gcc's default for each arch, written out: -march=armv8-a / -march=x86-64. The old -march=x86-64-v3 can't stay. CMake puts CFLAGS before each variant's flags, and on x86 the variant adds -m flags rather than setting its own -march. The x64/sse42/sandybridge variants would therefore still contain AVX2, and so would the feature probe (*-feats, built with CFLAGS only), which could SIGILL on the CPUs it is meant to reject. On arm64 each variant's own -march=armv8.x-a+… comes last and wins. libllama, ggml-base and the server now build at the baseline too, as upstream's do. The hot loops are in ggml-cpu.
  • amd64 drops the x86-64-v3 floor. The package now runs on pre-AVX2 x86_64 and gets the AVX-512/VNNI/BF16/AMX kernels where the CPU has them.
  • Install location: /usr/lib/ggml, baked in as GGML_BACKEND_DIR. The loader looks there first, then in the executable's directory (readlink /proc/self/exe, which isn't necessarily /usr/bin once a box links the package in), then in the cwd. The variants are plugins that nothing links against, so they get their own directory and output (cpu-backends, usr/lib/ggml/libggml-cpu-*.so*) rather than sitting in the linker's path. They NEED libggml-base.so.0 (in /usr/lib) and libgomp, both already shipped or in runtime_deps. The existing DT_NEEDED fail-closed loop now covers usr/lib/ggml too.
  • The variant set is pinned in build.sh. The build fails if the built libggml-cpu-*.so set differs from the per-arch VARIANTS list, or if a plain libggml-cpu.so appears. A bump that changes upstream's list fails loudly instead of quietly shipping a different set.
  • LLAMA_BUILD_TESTS=OFF, LLAMA_BUILD_EXAMPLES=OFF. Neither is shipped (no shared libs come from them; checked at b12359). This offsets part of the extra compile. build_cost_multiple goes from 3 to 5. That number is a guess: ggml-cpu is compiled 8x (arm64) or 14x (x86_64) instead of once.

Compiler support

Tests

  • llama-cli --version and llama-server --version stay as they are. They still catch a missing shared library (exit 127), but they never load a backend, so on their own they would pass with no CPU backend at all.
  • New: cpu-backend-cli / cpu-backend-server run <bin> -ot x=CPU --list-devices with no model. -ot runs ggml_backend_load_all() and throws unknown buffer type (exit 1) unless a CPU backend loaded and registered its CPU buffer type. --list-devices then exits 0. --list-devices alone isn't enough: it prints only non-CPU devices, so it shows (none) whether or not a CPU backend loaded.

Risks / to verify on build-bot

  • Reproducibility. The variant set comes from the CMake options, not the build host (no -march=native, and GGML_NATIVE is fatal with BACKEND_DL). Every variant gets the same -ffile-prefix-map/--build-id=none flags. Needs a prod-pair / double-build compare on both arches.
  • Size. About 8 to 14 copies of ggml-cpu in the output. Expect several MB per arch on top of today's size.
  • Loader paths. /usr/lib/ggml is absolute: it assumes the package's files appear at /usr/lib at run time, as the existing /usr/bin/llama-cli tests already assume. Upstream's loader also scans the current directory and keeps the highest score across all paths. A libggml-cpu-*.so planted in the cwd would be dlopened and could win. This PR doesn't change that upstream behavior. Raising it here because it is a library-planting surface.
  • Every candidate module is dlopened on each start (to call its score function), so startup does a little more work.

Not built locally. build-bot builds it.

🤖 Generated with Claude Code

arm64 was built for plain armv8-a, so ggml-cpu left out the DOTPROD, FP16,
SVE, i8mm, SVE2 and SME kernels; x86_64 had a hard x86-64-v3 floor that
crashed on pre-AVX2 CPUs and ignored AVX-512/VNNI/AMX. Build with
GGML_BACKEND_DL + GGML_CPU_ALL_VARIANTS instead, as upstream's release
builds do: one libggml-cpu-<variant>.so per microarchitecture (8 on arm64,
14 on x86_64), installed in /usr/lib/ggml (GGML_BACKEND_DIR), and the
loader keeps the highest-scoring one for the host.

CFLAGS drop to each arch's baseline because a higher -march leaks into
every x86 variant and into the feature probe. build.sh fails if the built
variant set differs from the list it expects. Tests/examples are no longer
built. New standalone tests fail unless a CPU backend loads.

Fixes #975

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

With GGML_BACKEND_DL, ggml_backend_load_best() also searches the
executable's directory and the current directory, and dlopen()s every
libggml-<name>-*.so it finds before scoring them (ggml-backend-reg.cpp
488-499, 516 at b12359), so their constructors run. Starting
llama-server or llama-cli in a writable directory (a downloaded model
folder, a cloned repo) would run a planted library. The static build
never did this.

ggml-backend-dir-only.patch makes a build with GGML_BACKEND_DIR set
search only that directory. GGML_BACKEND_PATH, an explicit opt-in, is
unchanged. The file's sha256 is checked before and after the patch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bryan-minimal

Copy link
Copy Markdown
Member Author

Added 50c1d71f: a carried patch so the dlopen()ed CPU backends load only from /usr/lib/ggml.

At b12359, ggml_backend_load_best() searches GGML_BACKEND_DIR, then the executable's directory, then the current directory, and dlopen()s every matching libggml-*.so before scoring (ggml-backend-reg.cpp 488–499 and 516). Library constructors run on load, so starting llama-server or llama-cli in a writable directory (a model folder, a cloned repo) would run a planted library. The static build never had this.

ggml-backend-dir-only.patch drops the executable and current directories when GGML_BACKEND_DIR is set. GGML_BACKEND_PATH, an explicit opt-in, is unchanged. The file is sha256-checked before and after the patch (0a23a627… → 607514a8…), following the stb patch's pattern. Worth upstreaming.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

llama.cpp: arm64 build leaves out dotprod/fp16/i8mm/SVE kernels (use ggml CPU variants)

1 participant