kait

Self-hosted Buildkite agents with a batteries-included AI/ML execution environment. Choose the work capability and hardware; the selected Linux container already contains the Buildkite integration, pinned Python toolchain, hardware contract, diagnostics, and lightweight observability.

Kait is pronounced “kite.” Its spelling keeps ai at the center, reflecting its focus on AI and machine-learning execution. It also carries a small linguistic resonance: 愛 (ai) is the Japanese kanji for “love”; in Japanese, the vowels in ai are articulated together as /a.i/, which English speakers commonly perceive as close to “eye.” The connection is a resonance in the spelling, not a claim that Kait is a Japanese word; spoken aloud, it remains simply “kite.”

# Pull the latest data-science environment (stable alias)
docker pull ghcr.io/alexhraber/kait:cpu-slim

# Run as a Buildkite agent
BUILDKITE_AGENT_TOKEN=… \
KAIT_HARDWARE=cpu \
KAIT_VARIANT=slim \
KAIT_O11Y=prometheus \
KAIT_IMAGE=ghcr.io/alexhraber/kait:cpu-slim \
  ./deploy/docker/run.sh

For a training, orchestration, or serving job, choose the matching workload profile. The worker advertises the capabilities baked into that image; the pipeline does not install them at job start.

Apple is modeled as a native macOS/MPS hardware contract, but it is currently disabled like the other accelerator classes. On Apple Silicon, use the ordinary multi-architecture Linux CPU image through Docker or Apple Container and advertise kait.hardware=cpu.

Pin a release for production:

docker pull ghcr.io/alexhraber/kait:v0.2.1-cpu-slim

Package visibility: the container package must be public for anonymous pulls. If docker pull returns 401/404 for a public repo, open Package settings → Change visibility → Public.

Registry: ghcr.io/alexhraber/kait

slim is the compact data-science stack. full adds Hugging Face training, Ray/MLflow/W&B, and FastAPI/Gradio serving.

The public capability set is deliberately small:

Every official container records this set in /etc/kait/identity.json, exposes it through kait doctor, and advertises it to Buildkite as kait.capability.<name>=true. slim and full remain compatibility names; they describe package footprints, while capabilities describe usable work.

The native Apple MPS contract remains modeled for future re-enablement, but is inactive in CI and release matrices and is not a published execution surface.

Versioned container tags use the six profiles, for example v0.5.0-cpu-training. Apple-specific profiles are currently not released; Apple Silicon users should use the cpu-* multi-architecture images. Previously published apple-* tags remain Linux CPU compatibility artifacts and are not GPU surfaces. slim and full remain compatibility profiles, while workload-specific artifacts have their own identity and smoke proof.

NVIDIA / AMD / Intel bake targets exist for deliberate host testing but are inactive in automatic CI/release. Opt in with make build-all-accelerators or the image workflow’s accelerator input.

make build-plan   # print bake graph (Docker Buildx)
make build-slim   # Linux CPU slim compatibility profile
make build-full   # Linux CPU full compatibility profile
make build-profiles # all six active Linux container profiles

Kait is a small Go supervisor (cmd/kait) that validates the baked hardware/profile/capability contract, starts buildkite-agent start (or a diagnostic command), exposes health/metrics when enabled, and exits with the child status.

kait doctor    # JSON hardware probe
kait smoke     # framework + device check (used by CI)
kait hardware  # accelerator CLI output

kait doctor reports the baked identity, available capability checks, expected hardware, detected devices, and whether the host satisfies the hardware contract. kait smoke performs representative bounded programs for every declared capability and validates accelerator access when the profile includes the PyTorch data-science contract. A missing baked identity or runtime attempt to override it fails closed.

On Apple Silicon, run the Linux CPU image and use the CPU hardware selector. Apple-native doctor/smoke execution is inactive until a matching GPU surface is deliberately re-enabled.

Deeper layout: docs/architecture.md and the capability contract.

Read the Kait documentation site for the execution-substrate perspective, capability contract, and architecture.

• Docker: deploy/docker/run.sh — local agent launch. Use deploy/docker/smoke.sh for device/framework checks without a token.
• Kubernetes: deploy/kubernetes/kait-agent.yaml — one-shot Job. See deploy/kubernetes/README.md.
• Pipeline targeting: examples/pipeline.yml

agents:
  queue: ai
  kait.hardware: cpu
  kait.capability.data-science: "true"

Training and serving jobs request kait.capability.training: "true" or kait.capability.serving: "true" and can land on a full worker. The old kait.variant selector remains valid for compatibility, but it is not a workload promise. See examples/pipeline.yml for a complete set of selectors. Kait reserves kait.* agent tags so custom tags cannot silently contradict the image identity.

Apple Silicon CPU jobs use the same selector as any other CPU worker: kait.hardware: cpu. Apple GPU selectors remain unavailable until the native hardware contract is deliberately re-enabled.

Vendor-specific packages (DeepSpeed, bitsandbytes, FlashAttention, vLLM, s3fs, …) stay out of the defaults. Pass them through KAIT_EXTRA_PYTHON_PACKAGES at build time. Details: requirements/README.md.

Kait is also a base layer. Pin an immutable artifact, then add only the organization's delta:

FROM ghcr.io/alexhraber/kait:v0.2.1-cpu-full

COPY internal-requirements.txt /tmp/
RUN /opt/kait/venv/bin/pip install --no-cache-dir -r /tmp/internal-requirements.txt
COPY internal-tools/ /opt/acme-tools/

The inherited identity and supervisor remain the source of the advertised capability contract. If an extension replaces core frameworks or hardware packages, the organization should rerun kait doctor/kait smoke and own the resulting compatibility claim.

KAIT_O11Y=prometheus   # scrape :9090/metrics
KAIT_O11Y=datadog      # DogStatsD via KAIT_DD_* / DD_*
KAIT_O11Y=splunk       # same metrics + standard OTEL_* env

Structured JSON logs go to stderr. Child job output stays on the agent’s stdout/stderr. Vendor credentials belong on the collector or node agent, not in the image.

make test
make vet
make build          # bin/kait

Release Please opens a release PR from main. Merging it:

1. Creates the semver tag and GitHub release
2. Dispatches release-images.yml against that tag (actions: write required)
3. Publishes active Linux container profiles to GHCR with provenance and SBOM
4. Annotates the GitHub release with pull/install commands

When a merged change has no conventional release unit, the same workflow opens an automatic patch release PR so infrastructure, documentation, and identity changes still reach the versioned image publication path.

See CONTRIBUTING.md and SECURITY.md.

MIT © Alex H. Raber