Repository navigation
Lemonade v10.3.0 is pulling Ubuntu executables #5
Description
Activity
Opened issue on lemonade-sdk repo: lemonade-sdk/lemonade#1791.
Also I am suspecting that the issue could arise from the new mechanism or the
*_binresolution, which "should" test for value being existing path.Important edit: only now I noticed, that the files fetched by lemonade are just the ROCm libraries when trying to run llama-server built with ROCm, but in the case of Vulkan, the lemonade pulls the llama-server.
It seems like if the lemonade fails to find ROCm in
/opt/rocm, or it finds a version different from on the the predefined "preview", "nightly", "stable", then it proceeds to install chosen version into~/.cache/lemonade/bin/and then it starts llama-server with modified LD_LIBRARY_PATH, however this manipulation with LD_LIBRARY_PATH breaks the llama-cpp from nixpkgs.Reacted by Noam Stolero- added a commit that references this issue
on May 5, 2026 @FrantaNautilus thanks for the detailed write-up — the env-var-migrated
*_binconfig really wasn't taking effect, and the diagnosis matches what's actually happening in v10.3.0. Three regressions, fixed in 3cd0ffd:BackendManager::install_backend(v10.3.0) added ano_fetch_executablesthrow and rocm-stable / TheRock runtime fetches beforeinstall_from_github's path-override short-circuit — so even whenllamacpp.rocm_bin/vulkan_binwere set, the runtime libs still downloaded into~/.cache/lemonade/bin/.LlamaCppServer::loadalways prepends TheRock's lib dir toLD_LIBRARY_PATHfor ROCm backends, shadowing the libs that the nix-builtllama-serveris RPATH-bound to.is_ggml_hip_plugin_available()only probes Debian/Ubuntu/Arch FHS paths, so thesystemllamacpp recipe was permanentlyunsupportedon NixOS.
Patches in this repo gate (1) and (2) on
find_external_backend_binary, and add aLEMONADE_GGML_HIP_PATHenv-var override for (3). The NixOS module wires the new env var to${pkgs.llama-cpp-rocm}/lib/libggml-hip.so.Verified end-to-end on a thinkpad-p14s-g6 (Strix Halo gfx1151):
lemonade backendsnow showsllamacpp:systemasinstalled(wasunsupported)lemonade load <model>no longer emits theInstalling llama-server (b8983)line, no Progress bytes, no~/.cache/lemonade/bin/writessystem_fingerprintin API responses reports the nix-built version (b8770-…) instead of upstream's bundled tag
Could you bump the input on your nix-config (
nix flake update nix-amd-aiagainstmainat 3cd0ffd) and confirm? If it works for you too, I'll close this.Upstream tracking: lemonade-sdk/lemonade#1791.
- added a commit that references this issue
on May 5, 2026 Thank you @noamsto for the impressive update and fix of the upstream problem. I have to admit it is slightly concerning that there is no way to disable the fetching behavior on the upstream Lemonade.
I have updated on my side and I confirm that:
- llamacpp works and does not download executables or libraries to cache directory
- the new integrations of stable diffusion models works (tested with Klein 9b)
- the integration whisper-cpp failed on my test with Whisper-large-v3 (details below)
- the update to the new version of nix-amd-ai required building of llama-cpp, whisper-cpp, stablediffusion-cpp from source, I am not sure whether this is to be expected or if it is expected to be available in binary cache of Nix/Cachix
Details of the whisper-large-v3 failure: Trying to process short audio file in MP3, I got the following output in the system log:
May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.260 [Info] (Server) Ensuring model loaded: Whisper-Large-v3 May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.260 [Info] (WhisperServer) Loading model: Whisper-Large-v3 May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.260 [Info] (WhisperServer) Per-model settings: whispercpp_backend=vulkan, whispercpp_args=(none) May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.260 [Info] (WhisperServer) Using model: /home/hudecvl199/.cache/huggingface/hub/models--ggerganov--whisper.cpp/snapshots/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-large-v3.bin May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.260 [Info] (WhisperServer) Using backend: vulkan May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.260 [Info] (WhisperServer) Starting server on port 8001 May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.260 [Info] (WhisperServer) Process started with PID: 46321 May 05 16:44:00 leng lemond[3813]: Waiting for whisper-server to be ready (timeout: 600s)... May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.963 [Error] (WrappedServer) whisper-server process has terminated with exit code: -1 May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.963 [Error] (WrappedServer) This usually means: May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.963 [Error] (WrappedServer) - Missing required drivers or dependencies May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.963 [Error] (WrappedServer) - Incompatible model file May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.963 [Error] (WrappedServer) - Try running the server manually to see the actual error May 05 16:44:00 leng lemond[3813]: 2026-05-05 16:44:00.963 [Info] (WhisperServer) Stopping server (PID: 46321) May 05 16:44:05 leng lemond[3813]: 2026-05-05 16:44:05.965 [Warn] (ProcessManager) Process did not respond to SIGTERM, using SIGKILL May 05 16:44:05 leng lemond[3813]: 2026-05-05 16:44:05.966 [Info] (ProcessManager) Process terminated, waiting for GPU driver cleanup... May 05 16:44:07 leng lemond[3813]: 2026-05-05 16:44:07.966 [Error] (Router) Backend load failed: whisper-server failed to start or become ready May 05 16:44:07 leng lemond[3813]: 2026-05-05 16:44:07.966 [Warn] (Router) Load failed with non-file-not-found error, evicting all models and retrying... May 05 16:44:07 leng lemond[3813]: 2026-05-05 16:44:07.966 [Info] (Router) Evicting all models (0 total) May 05 16:44:07 leng lemond[3813]: 2026-05-05 16:44:07.966 [Info] (Router) All models evictedSo I tried what the log suggested an ran whisper-server:
whisper-server ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32110 MiB): Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32110 MiB load_backend: loaded CUDA backend from /nix/store/5hpqk6sf9m46m8021wjw014vvpnk0hh6-whisper-cpp-1.8.4/lib/libggml-cuda.so load_backend: loaded CPU backend from /nix/store/5hpqk6sf9m46m8021wjw014vvpnk0hh6-whisper-cpp-1.8.4/lib/libggml-cpu-zen4.so whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.en.bin' whisper_init_from_file_with_params_no_state: failed to open 'models/ggml-base.en.bin' error: failed to initialize whisper contextThe Nvidia GPU is not an error, since I have an eGPU connected over USB4. However, it seems that the whisper-server is completely missing the vulkan devices, which is strange since I do not have any different than vulkan whisper installed.
nix why-depends /run/current-system nixpkgs#whisper-cpp-vulkan /nix/store/c9fd4g5j61cg2ivhszdk87f91g1vj2cl-nixos-system-leng-26.05.20260430.15f4ee4 └───/nix/store/0wqd253gdhgdw17dmpnznxh8bsd4rsgg-etc └───/nix/store/3ipjisb29ps3l5i777jc43dnk2yrkd02-set-environment └───/nix/store/4ic7jfnz90hjbn9klqbz3sa084kwll5j-whisper-cpp-1.8.4Interestingly the same issue seems to apply to the llama-server:
llama-server --list-devices ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32110 MiB): Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32110 MiB Available devices: CUDA0: NVIDIA GeForce RTX 5090 (32110 MiB, 31587 MiB free)I am currently rebuilding my system without cudaSupport=true, so that i can isolate the cause of this problem.
Glad llamacpp + sd-cpp are working — thanks for the detailed report.
Re: rebuilding from source. That shouldn't happen with the current setup.
pkgs.llama-cpp.override,pkgs.whisper-cpp.override, andpkgs.stable-diffusion-cpp.overrideare all Hydra-built for the nixpkgs commit this flake pins, so they substitute cleanly fromcache.nixos.org(verified on a fresh CI run — no rebuilds).If they're rebuilding for you, the most likely cause is
inputs.nix-amd-ai.inputs.nixpkgs.follows = "nixpkgs"in your flake. The overlay deliberately uses this flake's pinnednixpkgs(flake.nixpinned) so the input closure hash lines up with both Hydra and our Cachix. Following ournixpkgsre-hashes every override against your own commit, which Hydra hasn't built. Drop the.followsand the rebuilds should disappear. Added a README note: Binary cache.Re: whisper-server only seeing CUDA. Your standalone output is the smoking gun:
```
load_backend: loaded CUDA backend from /nix/store/…-whisper-cpp-1.8.4/lib/libggml-cuda.so
```That's a CUDA-flavored `whisper-cpp`, not our `whisper-cpp-vulkan` override — almost certainly `nixpkgs.config.cudaSupport = true` propagating through `.override` and shadowing the Vulkan build. Your in-progress rebuild without `cudaSupport` should resolve it. If you want to keep CUDA enabled elsewhere, an alternative is overriding `cudaSupport = false` on `llama-cpp-vulkan` / `whisper-cpp-vulkan` / `stable-diffusion-cpp-rocm` only, via a small consumer-side overlay.
Holding the issue open until you report back on the no-CUDA rebuild.
Are you in discord with the rest of lemonade devs https://discord.gg/5xXzkMu8Zk
We can work out a official way with you to make this work for Nix OS if you like to be a maintainer.@Geramy thanks for the invite — sounds good, I'll join the Discord. Happy to coordinate on the NixOS side: this flake is already shipping a handful of in-tree patches against v10.3.0 (lemonade-sdk/lemonade#1791 covers the main one — the v10.0 install_backend regression that breaks user-supplied
*_binpaths), and a few of those would be cleaner as proper upstream hooks (e.g. aLEMONADE_LLAMACPP_SYSTEM_BINenv var, a--no-fetch-executableshonoring path, theis_ggml_hip_plugin_available()env override). Glad to scope what a NixOS-maintainer role would look like — I can review NixOS-side PRs and keep the flake aligned with releases, but I'll want to keep the surface async/written-first rather than live-chat.What's the best channel on Discord to start in?
Again, thank you for the help @noamsto and I am sorry for taking long with my reply. I confirm that after the suggested change the whisper-cpp Vulkan now works in the Lemonade GUI and works on GPU (and does not fallback to CPU).
For some reason, llama-server and whisper-server fail to detect devices when invoked from CLI, they only list BLAS device. I am still searching for the reason for this in my configuration, nevertheless I believe that this issue can be closed.@Geramy thanks for the invite — sounds good, I'll join the Discord. Happy to coordinate on the NixOS side: this flake is already shipping a handful of in-tree patches against v10.3.0 (lemonade-sdk/lemonade#1791 covers the main one — the v10.0 install_backend regression that breaks user-supplied
*_binpaths), and a few of those would be cleaner as proper upstream hooks (e.g. aLEMONADE_LLAMACPP_SYSTEM_BINenv var, a--no-fetch-executableshonoring path, theis_ggml_hip_plugin_available()env override). Glad to scope what a NixOS-maintainer role would look like — I can review NixOS-side PRs and keep the flake aligned with releases, but I'll want to keep the surface async/written-first rather than live-chat.What's the best channel on Discord to start in?
General is fine just tag @Geramy and I'll tag the appropriate people, thanks!
Again, thank you for the help @noamsto and I am sorry for taking long with my reply. I confirm that after the suggested change the whisper-cpp Vulkan now works in the Lemonade GUI and works on GPU (and does not fallback to CPU). For some reason, llama-server and whisper-server fail to detect devices when invoked from CLI, they only list BLAS device. I am still searching for the reason for this in my configuration, nevertheless I believe that this issue can be closed.
Thanks!
closing,
re-open or create a new issue if need any help.- added a commit that references this issue
on Sep 3, 2026
Since the update to version 10.3.0, the lemonade seems to ignore all setting that should block it from downloading llamacpp executables, and proceeds to download executables to
~/.cache/lemonade/bin. After this download, lemonade uses these executables instead of those in the nix store. These executables understandably fail under NixOS.Before the update to v10.3.0, nix-amd-ai (10.2 0) worked without issues and the ROCm llamacpp backend was functional.
In the
~/.cache/lemonade/config.json, the paths to ROCm and Vulkan llamacpp binaries are set, and theprefer_systemis set to true. Yet, uponlemonade load <model>command, the llamacpp is pulled to~/.cache/lemonade/binand then proceeds to fail to initialize. After clearing the~/.cache/lemonadeand reboot, i tried to setofflinemode to true, and lemonade still continues to download the executables. Another attempt was made with the optionno_fetch_executablesto true. This, however, leads to the lemonade to just fail, with no llamacpp executable found.I am not sure if this problem lies on nix-amd-ai level or lemonade level. Thanks in advance for any help.