k8s-ai-observability

Build and test GPU and LLM observability without a GPU. A simulated NVIDIA GPU stack, a simulated vLLM serving stack, and Prometheus + Grafana, on kind, EKS or GKE. No hardware, no quota, no drivers, no model weights.

Because the rig knows the true value behind every series, it catches the monitoring bugs that pass every other test. Three it finds in vLLM monitoring, each of which your real deployment has too, because the bucket boundaries and metric names are upstream's:

• An SLO alert that can never fire. vLLM's time-to-first-token buckets step 1.0, 2.5, 5.0. There is no 2.0, so an objective written as le="2" matches no series and its burn-rate alerts stay green forever (the error budget).
• A prefill p95 three times too high. The first bucket vLLM gives prefill ends at 0.3s, so every sub-300 ms prefill lands in it and histogram_quantile interpolates from zero: a 0.095s prefill reads as 0.285s (the measurement).
• Panels that go blank after an engine upgrade. vLLM's V1 engine renamed two metrics and replaced a third with two counters. Nothing errors; the panel just empties. --vllm-surface both shows which of your panels an upgrade breaks (which names moved).

cd compose && docker compose up -d     # both boards on localhost:3000. ~1 min, no Kubernetes.

That's the fastest way to see it. To exercise Kubernetes itself (scheduling on nvidia.com/gpu, the device plugin, ServiceMonitor discovery, the Grafana sidecar):

task local:up && task local:grafana    # deploy the kind cluster & full stack then open the grafana dashboards ~6 min.

Link straight here: https://github.com/ChrisAdkin8/k8s-ai-observability#watch-it-run

The install is fast-forwarded. verify.sh runs at the recording's own pace, because the checks are the point. GitHub scales the recording down to the column width, so click through for it at full size.

There's no cloud account, no credentials, no terraform.tfvars and no spend. local is not a reduced-fidelity preview: it runs the same manifests, the same pinned charts and the same acceptance checks as EKS and GKE, which exist only to prove it works on managed Kubernetes. The compose stack reads those same dashboards, rules, simulator and profiles. See compose/ for what it deliberately cannot cover.

• GPU simulation. run-ai/fake-gpu-operator advertises nvidia.com/gpu, injects a fake nvidia-smi, and emits DCGM-format metrics, so the standard NVIDIA observability stack works as-is.
• LLM serving simulation. scripts/llm-sim.py is one dependency-free Python file emitting real vLLM metric names, types and histogram buckets. It runs two tenants side by side: one healthy, one deliberately overloaded.
• Recording rules and alerts over both domains, unit-tested with promtool in about a second, with no cluster. The tests cover both sides of every threshold, including the ones the rig never drives. If you build alerting here, you can test it here.
• Two Grafana dashboards, one .json each and never clicked into place, so a re-install reproduces them exactly. Both are in the catalog. Import by id into a Grafana you already run: 25618 (GPU), 25620 (vLLM).
• An acceptance suite (scripts/verify.sh) that asserts metrics are flowing, both boards render, and the alerts actually reach firing.
• A weekly check against real vLLM, which is what makes the claim above checked rather than asserted. check-vllm-buckets.py fetches upstream's own metric definitions and fails if a name or a bucket boundary has moved. It exists because two releases shipped the wrong bucket layout with every test green. A drifted boundary returns a confident, plausible, wrong percentile (the full story).
• The simulator as a container image, so you can point your own dashboards at a realistic vLLM metric surface without cloning anything: docker run --rm -p 9401:9401 ghcr.io/chrisadkin8/vllm-metrics-sim:latest. Built from scripts/llm-sim.py rather than a copy of it, for amd64 and arm64 (details).
• A path for clusters that already run Prometheus, so panels blank for want of the llm:* recording rules are a two-minute fix rather than a reason to hand your monitoring stack to an install script. Either bring your own Prometheus or the Helm chart.

Two tenants sit either side of the 2s alert threshold, which is the point of running two simulators. sim-llama-3-8b-steady answers in ~120 ms; sim-llama-3-8b-saturated reports a p95 of 78s with 16 requests running and 160 queued behind it. Every panel aggregates by (model_name), so the overloaded tenant is never averaged into the healthy one.

The healthy tenant's p95 depends on the window you capture. This screenshot reads ~99 ms, close to the ~120 ms the profile arithmetic models: in these fifteen minutes its batch peaked at 14 of 16 slots, so no request waited. A capture from the same cluster minutes after install read ~440 ms, and the previous screenshot ~480 ms, with the queue gauge flat at zero in both. All three are right. The batch is two-thirds full on average and fills now and then, every request that arrives while it is full waits, and a 15s-sampled gauge misses what the TTFT histogram records in full. Either way it sits an order of magnitude or more under the 2s threshold, which is what the panel exists to show. The arithmetic.

These metrics are synthetic. That is the trade, and it is worth being precise about which half of your work it affects.

In short: build the pipeline here, tune the numbers on real hardware.

Needs kubectl, helm and task, plus kind + a container runtime for local. See Prerequisites. ⚠️ Size your container runtime first: colima's 2 CPU / 2 GiB default cannot run this. task tools checks the lot.

Three targets (local, eks, gke), each with two phases. Phase 1 creates the cluster with the GPU-sim node label (scripts/kind-up.sh locally, terraform apply on the clouds). Phase 2 is scripts/install.sh, identical everywhere. The phases stay separate so the Kubernetes tooling is never pointed at a cluster that does not exist yet.

task on its own lists everything. The task list is defined once in taskfiles/target.yml and included three times, so the targets cannot drift apart.

# EKS
cp terraform/eks/terraform.tfvars.example terraform/eks/terraform.tfvars   # set api_allowed_cidrs
task eks:up

# GKE is identical bar the prefix (Standard, not Autopilot)
echo 'GCP_PROJECT=my-project' > .env
task gke:up

Every command below takes any of the three prefixes ( local | eks | gke ):

Both load drivers target opt-in Deployments that install.sh does not apply, so apply the one you need first or the task fails its precondition:

kubectl apply -f manifests/workloads/extras/   # for task <prefix>:load
kubectl apply -f manifests/llm/extras/         # for task <prefix>:llm-load

scripts/ stays the source of truth for install ordering, the wrong-context guard and the drift assertions; Task only wraps it. To drive the phases as scripts instead, see docs/usage.md.

The default path installs kube-prometheus-stack. If you already run one, this repo does not need to touch it. Use either install.sh --skip-monitoring or the Helm chart:

./scripts/install.sh local --skip-monitoring
./scripts/verify.sh   local --byo

⚠️ Two labels decide whether that works, and both fail with no error at all: no scrape, no rule evaluation, empty boards, every object reporting itself as created. The release: selector on the rules and ServiceMonitors, and the Grafana sidecar's discovery label. docs/byo-prometheus.md has both, what to set them to, and why verify.sh --byo is what tells you.

No GPU instances or GPU quota are used on any target.

kind delete cluster throws away the node's image store, so every cold task local:up re-pulls the whole stack: ~915 MB across 14 images, grafana alone 352 MB. Optional pull-through caches sit on the kind network and outlive the cluster.

task cache:up          # once; they persist across kind delete
task cache:status      # what each one is holding
task cache:down        # stop them, keeping what they cached
task cache:purge       # stop them and discard it

The build that populates them is slower, not faster. Every cold build after it reads from local disk instead of from five registries.

Off by default, and safe to stop at any point: with no cache running kind-up.sh writes no mirror config, and containerd falls back to the real registry regardless.

⚠️ Unverified: the podman path. These have only ever been run against Docker.

Grafana stays ClusterIP, so there is no load balancer, no ingress and no cost. install.sh prints both URLs when it finishes, and task <prefix>:grafana holds one port-forward and opens both:

• http://localhost:3000/d/gpu-sim-dcgm for utilisation, memory, temperature and power
• http://localhost:3000/d/llm-sim-overview for first-token latency and its error budget, throughput, queue depth, KV cache and prefix-cache reuse

Viewing needs no login: anonymous access is Viewer-only. To edit, log in as admin; the script prints the generated password. Use GRAFANA_PORT=3001 if 3000 is taken, and --no-open to skip the browser.

⚠️ Anonymous access is safe only because the Service is ClusterIP, so the sole way in is a port-forward, which already requires cluster RBAC. If you switch grafana.service.type to LoadBalancer or NodePort, disable auth.anonymous in helm/kube-prometheus-stack/values.yaml first. Otherwise you publish the dashboards to anyone who can reach the address.

If a board 404s or its panels come up empty, see docs/troubleshooting.md.

task tools checks all of the above, and only fails on the target-agnostic ones.

Size your container runtime first. Colima's 2 CPU / 2 GiB default cannot run this: Prometheus alone requests 1Gi and limits 2Gi. scripts/kind-up.sh refuses below 5 GiB and prints the exact colima / Docker Desktop / podman machine command to fix it. Allow 8 GiB and 4 CPU (the floors and the recommendation are KIND_MIN_MEMORY_GIB / KIND_MIN_CPUS and KIND_WANT_MEMORY_GIB / KIND_WANT_CPUS, if you need to move them):

colima start --cpu 4 --memory 8 --disk 40

Or run the trimmed stack instead. LITE=1 drops Alertmanager, kube-state-metrics, node-exporter and the chart's ~100 default rules, and puts Prometheus on 256Mi/512Mi. Measured at 2.2 GiB in use, so it fits a small runtime. Allocate 4 GiB:

colima stop && colima start --cpu 4 --memory 4 --disk 40
LITE=1 task local:up

4 is the recommendation, not the floor. The floor is 3 and it now means what it says: ask for 3 and the preflight accepts it, where it used to refuse. What it has never been is measured at 3, which is why the command above still says 4. How the floors are read.

It keeps everything local exists to prove: ServiceMonitor discovery, PrometheusRule evaluation, the Grafana sidecar import, and scheduling on nvidia.com/gpu. Both dashboards render and every acceptance check in verify.sh still runs and still passes, because none of them touches what was removed. values-lite.yaml lists exactly what you give up.

• A panel is empty, or something 404s. Start with docs/troubleshooting.md. Nearly every failure here surfaces the same way, and the table tells the causes apart.
• Still stuck, or upstream has moved. Open an issue. There is a bug report form, and an upstream-drift form for a vLLM metric name or bucket boundary that has changed under the rig.
• A security report. Not a public issue: SECURITY.md has the private reporting link, and sets out which parts of a deliberately unauthenticated test rig are in scope and which are the point.
• A change. CONTRIBUTING.md covers the invariants that fail silently when broken, what to re-check when bumping a pinned version, and what is deliberately out of scope.
• How it is built. Every line of it by one method: a written specification, an adversarial review of that specification, a throwaway spike that settles what reading cannot, and only then the implementation. docs/development-method.md describes how, and why the checks matter more here than the code they check.
• Conduct. The Contributor Covenant applies to both.

MIT, © 2026 Chris Adkin.

Nothing third-party is vendored here. The Helm charts (fake-gpu-operator, kube-prometheus-stack) are pulled from their own repos at install time, under their own licences. Both dashboards in manifests/dashboards/ are original to this repo rather than derived from a grafana.com board, so the MIT grant above covers everything the repo actually ships.