Skip to content
owainkenwayuclPublic

About

Terraform/Ansible code to build the ARC LLM hosting

Resources

Stars

6 stars

Watchers

0 watching

Forks

Latest commit

 

History

108 Commits

Folders and files

Repository files navigation

arc-llms

Terraform/Ansible code to build the ARC LLM (and image generation) hosting

This is a slightly involved process.

You will need to get a tenancy with GPUs attached to it. Once you have done that, there is terraform code in inference-host which will build your VMs. Having created the VMs, you need to use attach_gpu.py to attach a GPU to each VM, RESTART NOT REBOOT the VMs so that they migrate to the right machine and then use ansible to install the base environment. Finally, you need to use the appropriate roles to install the right containers and set them to run as daemons.

Note

As we migrate towards the pilot this way of accessing the systems will cease to operate. Presently, for continuity, inf01 and inf02 are proxies to models hosted on Unified AI hardware which are running to smooth transition.

Note

If you are using the POC deployed on Condenser, I sometimes change the models. When I do I will update here what they are:

  • inf01 - Qwen - RedHatAI/Qwen3.8-27B-INT4
  • inf02 - Gemma - RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic
  • embed01 - ibm-granite/granite-embedding-278m-multilingual - ibm-granite/granite-embedding-278m-multilingual - note this is a CPU node
  • embed02 - Qwen/Qwen3-Embedding-0.6B - Qwen/Qwen3-Embedding-0.6B - note this is a CPU node

Walkthrough

1. Obtain tenancy and GPU IDs.

2. Modify inference-host/variables.tf to suit your environment.

Some things to note:

  1. It is assumed that the image is in your tenancy. This is because when I started there were no images in the general store on sl-g01.
  2. username is just a tag
  3. keyname sets which SSH key is injected into your VMs.

You may also want to modify the tags that configure ingress because by default the subdomains are inf01, inf02 etc.

3. Use Terraform to build your VMs.

cd inference-host 
terraform init
terraform apply

4. Attach GPUs to VMs

We don't have a standardised nice way of doing this so I have written a fairly shonky Python script which does it.

python3 attach_gpu.py <VM id> <GPU id>

e.g.

python3 attach_gpu.py uccaoke-inference-01-e3190ff38b sl-g01-10-000031000

This script will ask Condenser what type that GPU is and add it to the kubevirt config for the VM.

You then need to RESTART NOT REBOOT the VM, either through the Rancher UI or through kubectl virt/virtctl

5. Install the base software

As with other things I have built:

ansible-playbook -i generate_inventory.py full.yaml

This will:

  1. Install Docker
  2. Install Python3.12 (not actually necessary now development has been done)
  3. Install the Nvdia drivers and useful tools.
  4. Install and configure Ngnix as a transparent proxy

Step 4. is important for a number of reasons:

  1. VLLM only supports one API key for authentication. This is bad.
  2. Worse, for "reasons", the API key only protects /v1/ API calls, and not any of the other API calls, like the ones that let you shut down the server.

So what this does is configure Nginx in the following way:

  1. Use TLS so that traffic to ingress is encrypted.
  2. Block all requests except those that include a bearer token from /etc/nginx/nginx.auth which is auto-generated by ansible. By default it creates 3 API keys and puts copies in your ansible build machine in .apikey1, .apikey2 and .apikey3.
  3. Proxy requests to /v1 to port 5500 on localhost. This is where we will put our vllm/image server docker containers
  4. Configure logging so that /var/log/nginx/llm.log has a log of which key was used for each request.

It is worth changing the contents of /etc/nginx/nginx.auth so that keys are named so that they appear in the log. You can also add/delete new keys.

For example:

    map $http_authorization $api_user {
         default                     "__unauthenticated__";
         "Bearer HMSWarlockIsBest"   "keith.drummond";
         "Bearer ILoveHMSSaracen"    "richard.chesnaye";
    } 

This would result in requests using the bearer token "ILovedHMSSaracen" being able to access /v1/ on the Docker container and being logged by nginx as from richard.chesnaye and so on. The default item is the "fail" condition, passing "__unauthenticated__" as the api_user which means it has NO access to anything. DO NOT REMOVE THIS ENTRY

This is obviously not a "production ready" scalable setup as that would require some sort of service to manage and issue tokens.

6. Apply roles for different inference servers

LLMs

If you look at the role roles/vllm-qwen-coder you should see a role that can deploy the vllm docker container serving an OpenAPI compatible endpoint.

Image serving

NOTE: this is super not production ready! I wrote this with some help from Claude!

Image serving is done by building a docker container that uses FastAPI and the Huggingface diffusers library to serve a model - by default Stable Diffusion XL Turbo which is defined for fast inferences.

You can configure the model by either editing roles/diffusers-sdxl/files/imagesrv.ini before building the container, or by editing it on the VM and restarting the container as it is bind-mounted into the container. For models like Flux pay attention to the setting of variant - this should be empty.

Managing the containers

By default, the containers are run as daemons and set to restart unless explicitly stopped. The container is called inference so you can manage it knowing this fact.

For example, to watch vllm's logs on a vllm server:

docker logs inference

7. Connecting to your endpoints

Using the LLM:

There is a very simple chat client in examples/chat

Modify llm.ini appropriately, e.g. Keith Drummond in the authorisation example might make it look like:

[OPENAI]
endpoint = https://inf02.arc-llm.condenser.arc.ucl.ac.uk/v1
model = Gemma
api_key = HMSWarlockIsBest

Create a virtualenv, install requirements and run the client:

python3.12 -m venv runtime
source runtime/bin/activate
pip install -r requirements.txt
<...>
python3 chat.py
Starting up - LLM endpoint = https://inf02.arc-llm.condenser.arc.ucl.ac.uk/v1/Gemma
? hello
---
🤖 : Hello! How can I help you today?
--- [2.8368186950683594 seconds] 
Claude Code

If he wanted to use the LLM endpoint with Claude code he should set:

export ANTHROPIC_BASE_URL=https://inf02.arc-llm.condenser.arc.ucl.ac.uk/
export ANTHROPIC_MODEL=Gemma
export ANTHROPIC_AUTH_TOKEN=HMSWarlockIsBest
export ANTHROPIC_API_KEY=dummykey

The last variable (ANTHROPIC_API_KEY) is important the first time he runs claude because it bypasses the part of the claude setup that requires you to log in. He should unset it thereafter otherwise it will conflict with ANTHROPIC_AUTH_TOKEN.

There are additional environment variables you may want to set to make sure htings run smoothly:

export ANTHROPIC_DEFAULT_HAIKU_MODEL=$ANTHROPIC_MODEL
export ANTHROPIC_DEFAULT_SONNET_MODEL=$ANTHROPIC_MODEL
export ANTHROPIC_DEFAULT_OPUS_MODEL=$ANTHROPIC_MODEL
export ANTHROPIC_SMALL_FAST_MODEL=$ANTHROPIC_MODEL
export CLAUDE_CODE_SUBAGENT_MODEL=$ANTHROPIC_MODEL
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export CLAUDE_CODE_DISABLE_UNKNOWN_MODEL_WINDOW_ENFORCEMENT=1

The first five of these make sure that Claude Code also uses your configured model in various instances where it might call out to another model in a hard coded way. This is important if you have existing Anthropic authentication because those small calls cost (in one of our deployments where LiteLLM allows you to access local models, they end up going to Sonnet as that model is available there and run up the bills!). The last one makes sure that Claude Code trusts your model's self reported context window.

If you are using this in an environment where you can't control the environment variables your instance of Claude Code is running in (a good example being Code Server inside the Unified AI infrastructure), you can add these settings to an env block in ~/.claude/settings.local.json like this:

{
# ... other settings
  "env": {
    "ANTHROPIC_BASE_URL": "https://inf02.arc-llm.condenser.arc.ucl.ac.uk/",
    "ANTHROPIC_MODEL": "Gemma",
    "ANTHROPIC_AUTH_TOKEN": "<token>",
    "ANTHROPIC_API_KEY": "<token>",
    "CLAUDE_CODE_SKIP_AUTH_LOGIN": "1"
  },
}

Then restart Claude Code (in the case of Code Server restart the plugin) and it should pick up the settings.

You may also want to set CLAUDE_CODE_DISABLE_UNKNOWN_MODEL_WINDOW_ENFORCEMENT to 1, by either method as this will stop Claude Code defaulting to a short context window as it does not know these models.

Codex

Create two config files:

~/.codex/config.toml:

# Default local provider used with `codex --oss`
oss_provider = "arc_llms"

[model_providers.arc_llms]
name                  = "UCL ARC"
base_url              = "https://inf01.arc-llm.condenser.arc.ucl.ac.uk/v1"
env_key               = "ARC_LLMS_AUTH_TOKEN"
wire_api              = "responses"
requires_openai_auth  = false

~/.codex/arc_llms.config.toml:

model_provider = "arc_llms"
model = "Qwen"

Then set the environment variable ARC_LLMS_AUTH_TOKEN to your API key and run:

codex --oss --model Qwen
OpenCode

To use these LLMs with OpenCode, update your opencode.json configuration file on these lines:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ucl-arc-qwen": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "UCL ARC Qwen",
      "options": {
        "baseURL": "https://inf01.arc-llm.condenser.arc.ucl.ac.uk/v1",
        "apiKey": "{env:UCL_ARC_API_KEY}"
      },
      "models": {
        "Qwen": {
          "name": "UCL/Qwen/Qwen3.6-35B-A3B-FP8"
        }
      }
    },
    "ucl-arc-gemma": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "UCL ARC Gemma",
      "options": {
        "baseURL": "https://inf02.arc-llm.condenser.arc.ucl.ac.uk/v1",
        "apiKey": "{env:UCL_ARC_API_KEY}"
      },
      "models": {
        "Gemma": {
          "name": "UCL/Gemma4-26B-A4B-IT-FP8"
        }
      }
    }
  }
}

It will read your API key from the UCL_ARC_API_KEY environment variable.

You can also mention your key here, or you can provide it to the /connect command, in which case it will store it in ~/.local/share/opencode/auth.json (on Linux). Since these methods store your key as plain-text, they are insecure and not recommended.

Note that the ID you set when you run the /connect command should match the ID you set in the JSON configuration file (ucl-arc-qwen, for example). When you run /models in OpenCode, you should see the models listed.

Using the image service

You can also call the image service using the openai python library.

token = "HMSWarlockIsBest"
from openai import OpenAI
import base64
import io
from PIL import Image
endpoint = "https://inf01.arc-llm.condenser.arc.ucl.ac.uk/"
client = OpenAI(base_url=f"{endpoint}/v1", api_key=token)
response = client.images.generate(prompt="a white cup of pink tea", n=6, size="512x512")
images = []
for a in response.data:
    images.append(Image.open(io.BytesIO(base64.b64decode(a.b64_json))))

images will then be a list of PIL/Pillow images you can do whatever you want with.

About

Terraform/Ansible code to build the ARC LLM hosting

Resources

Stars

6 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages