Repository navigation
llama.cpp: build every ggml CPU variant and pick one at run time - #1559
bryan-minimal wants to merge 2 commits into
Conversation
arm64 was built for plain armv8-a, so ggml-cpu left out the DOTPROD, FP16, SVE, i8mm, SVE2 and SME kernels; x86_64 had a hard x86-64-v3 floor that crashed on pre-AVX2 CPUs and ignored AVX-512/VNNI/AMX. Build with GGML_BACKEND_DL + GGML_CPU_ALL_VARIANTS instead, as upstream's release builds do: one libggml-cpu-<variant>.so per microarchitecture (8 on arm64, 14 on x86_64), installed in /usr/lib/ggml (GGML_BACKEND_DIR), and the loader keeps the highest-scoring one for the host. CFLAGS drop to each arch's baseline because a higher -march leaks into every x86 variant and into the feature probe. build.sh fails if the built variant set differs from the list it expects. Tests/examples are no longer built. New standalone tests fail unless a CPU backend loads. Fixes #975 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
|
With GGML_BACKEND_DL, ggml_backend_load_best() also searches the executable's directory and the current directory, and dlopen()s every libggml-<name>-*.so it finds before scoring them (ggml-backend-reg.cpp 488-499, 516 at b12359), so their constructors run. Starting llama-server or llama-cli in a writable directory (a downloaded model folder, a cloned repo) would run a planted library. The static build never did this. ggml-backend-dir-only.patch makes a build with GGML_BACKEND_DIR set search only that directory. GGML_BACKEND_PATH, an explicit opt-in, is unchanged. The file's sha256 is checked before and after the patch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Added At b12359,
🤖 Generated with Claude Code |
Fixes #975.
What
llama.cpp now builds with ggml's runtime CPU dispatch, as upstream's release builds do (
.github/workflows/release.ymlat b12359):ggml-cpu is built once per variant as
libggml-cpu-<variant>.so. At startupggml_backend_load_all()dlopens each one, calls itsggml_backend_score()(a feature probe built without the variant's flags), and keeps the highest-scoring one.Variants at b12359 (
ggml/src/CMakeLists.txt:494-533, Linux):armv8.0_1,armv8.2_1(+dotprod),armv8.2_2(+fp16),armv8.2_3(+sve),armv8.6_1(+i8mm),armv8.6_2(+sve2),armv9.2_1(+sme),armv9.2_2(+sve2+sme)x64,sse42,sandybridge,ivybridge,piledriver,haswell,skylakex,cannonlake,cascadelake,icelake,cooperlake,zen4,alderlake,sapphirerapidsChoices
$MARCHis now gcc's default for each arch, written out:-march=armv8-a/-march=x86-64. The old-march=x86-64-v3can't stay. CMake puts CFLAGS before each variant's flags, and on x86 the variant adds-mflags rather than setting its own-march. Thex64/sse42/sandybridgevariants would therefore still contain AVX2, and so would the feature probe (*-feats, built with CFLAGS only), which could SIGILL on the CPUs it is meant to reject. On arm64 each variant's own-march=armv8.x-a+…comes last and wins. libllama, ggml-base and the server now build at the baseline too, as upstream's do. The hot loops are in ggml-cpu./usr/lib/ggml, baked in asGGML_BACKEND_DIR. The loader looks there first, then in the executable's directory (readlink /proc/self/exe, which isn't necessarily/usr/binonce a box links the package in), then in the cwd. The variants are plugins that nothing links against, so they get their own directory and output (cpu-backends,usr/lib/ggml/libggml-cpu-*.so*) rather than sitting in the linker's path. They NEEDlibggml-base.so.0(in/usr/lib) and libgomp, both already shipped or in runtime_deps. The existing DT_NEEDED fail-closed loop now coversusr/lib/ggmltoo.libggml-cpu-*.soset differs from the per-archVARIANTSlist, or if a plainlibggml-cpu.soappears. A bump that changes upstream's list fails loudly instead of quietly shipping a different set.LLAMA_BUILD_TESTS=OFF,LLAMA_BUILD_EXAMPLES=OFF. Neither is shipped (no shared libs come from them; checked at b12359). This offsets part of the extra compile.build_cost_multiplegoes from 3 to 5. That number is a guess: ggml-cpu is compiled 8x (arm64) or 14x (x86_64) instead of once.Compiler support
armv9.2-a+smeneeds gcc ≥ 14. Upstream's release workflow installs gcc-14 for exactly this. pkgs has gcc 15.2.0 and binutils 2.47.-mavxvnni,-mavx512bf16,-mamx-tile,-mamx-int8) need gcc ≥ 11.Tests
llama-cli --versionandllama-server --versionstay as they are. They still catch a missing shared library (exit 127), but they never load a backend, so on their own they would pass with no CPU backend at all.cpu-backend-cli/cpu-backend-serverrun<bin> -ot x=CPU --list-deviceswith no model.-otrunsggml_backend_load_all()and throwsunknown buffer type(exit 1) unless a CPU backend loaded and registered itsCPUbuffer type.--list-devicesthen exits 0.--list-devicesalone isn't enough: it prints only non-CPU devices, so it shows(none)whether or not a CPU backend loaded.Risks / to verify on build-bot
-march=native, andGGML_NATIVEis fatal withBACKEND_DL). Every variant gets the same-ffile-prefix-map/--build-id=noneflags. Needs a prod-pair / double-build compare on both arches./usr/lib/ggmlis absolute: it assumes the package's files appear at/usr/libat run time, as the existing/usr/bin/llama-clitests already assume. Upstream's loader also scans the current directory and keeps the highest score across all paths. Alibggml-cpu-*.soplanted in the cwd would be dlopened and could win. This PR doesn't change that upstream behavior. Raising it here because it is a library-planting surface.Not built locally. build-bot builds it.
🤖 Generated with Claude Code