Setup to bring-up llama-server (via a systemd service) on a remote machine that is plain Debian/Ubuntu and has an NVIDIA GPU.
sudo apt install ansibleansible-playbook playbook.yml -vvv -e "target_ip=NODE_IP"-
By default, this setup will bring-up
llama-serverwith:- Gemma-4-12B-it-qat
- all layers loaded on the (NVIDIA) GPU
- MTP (Speculative decoding)
- flash attention
To adapt that, you need to adapt the
llama-serverservice. An easy way to do this is to set thellama_server_argsenvironment variable (and only play theservertags), e.g. to run Kimi K3 fully offloaded on CPU:ansible-playbook playbook.yml -vvv -e "target_ip=NODE_IP" -e 'llama_server_args="-hf unsloth/Kimi-K3-GGUF:UD-IQ1_S -ngl 0"' --tags server