Skip to content

About

Provision a machine from scratch and deploy llama-server on it

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

llama-ansible

Setup to bring-up llama-server (via a systemd service) on a remote machine that is plain Debian/Ubuntu and has an NVIDIA GPU.

Prerequisites

sudo apt install ansible

Run

ansible-playbook playbook.yml -vvv -e "target_ip=NODE_IP"

Tweaking

  • By default, this setup will bring-up llama-server with:

    • Gemma-4-12B-it-qat
    • all layers loaded on the (NVIDIA) GPU
    • MTP (Speculative decoding)
    • flash attention

    To adapt that, you need to adapt the llama-server service. An easy way to do this is to set the llama_server_args environment variable (and only play the server tags), e.g. to run Kimi K3 fully offloaded on CPU:

    ansible-playbook playbook.yml -vvv -e "target_ip=NODE_IP" -e 'llama_server_args="-hf unsloth/Kimi-K3-GGUF:UD-IQ1_S -ngl 0"' --tags server

About

Provision a machine from scratch and deploy llama-server on it

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors