AI Toolbox Cockpit

AI Toolbox Cockpit is a Textual terminal application for running a local AI workstation without memorising a different container workflow for every backend.

Choose the hardware platform once, then use one cockpit to:

• install, update, enter, and delete compatible Toolbx/Distrobox containers;
• manage llama.cpp, DS4, and Gufo GGUF files, R9V model packages, and Halogen HGN bundles;
• inspect vLLM Hugging Face repositories and cache state;
• open ComfyUI's workflow-aware model manager;
• configure and launch llama.cpp, DS4, Gufo, R9V, vLLM, ComfyUI, or Halogen Flash servers.

The cockpit does not pretend these backends are interchangeable. Each backend owns its model semantics, server form, validation, and command builder. Shared container behavior lives in one runtime layer.

Install the current GitHub version in an isolated environment:

# Debian/Ubuntu: sudo apt install pipx
# Fedora: sudo dnf install pipx
# Arch: sudo pacman -S python-pipx
pipx ensurepath

pipx install git+https://github.com/kyuz0/ai-toolbox-cockpit.git

Run it with:

ai-toolbox-cockpit

Upgrade an existing pipx installation with:

pipx upgrade ai-toolbox-cockpit

The cockpit checks GitHub tags in the background at startup. When a newer version is available, it shows the exact command in a persistent update strip and provides an Upgrade now button that runs the pipx upgrade after confirmation.

Every non-workflow push to main runs .github/workflows/auto-version.yml. It generates a UTC CalVer version (YYYY.M.D.HHMM), updates pyproject.toml, commits the bump as github-actions[bot], creates the matching v<version> tag, and pushes the commit and tag. Those tags are what the cockpit's update check compares against the installed pipx version.

For local development:

git clone https://github.com/kyuz0/ai-toolbox-cockpit.git
cd ai-toolbox-cockpit
pipx install --editable .

The current R9V channel uses ROCm 10 with a 67,840-token context, one sequence, MTP-2 and disk-backed PLE for a 64 GB host. Install it from Toolboxes, then Download / Repair the curated package and Prepare PLE in Models. The earlier R9V channel remains available with its existing context settings. Current qualification covers text serving.

GGZ14 has one toolbox entry with single-card and dual-card builds selected in Server Mode; Radiance and native FP8 provide dual-card profiles. Install their rolling channels from Toolboxes and choose the corresponding curated artifact in Models. MXFP4 preparation creates a separate checkpoint. DFlash2 uses a separate FP8 draft checkpoint and one sequence. Model directories are mounted read-only during serving.

AI Toolbox Cockpit supports either of these interactive-container combinations:

• Podman with Toolbx;
• Podman with Distrobox;
• Docker with Distrobox.

When both engines are installed, Distrobox can be pinned explicitly:

DBX_CONTAINER_MANAGER=docker ai-toolbox-cockpit

The server views launch native podman run or docker run commands. They mount host model/cache directories so data survives toolbox refreshes.

The Toolboxes view is the shared container control plane.

• Platform selection removes incompatible images from the list.
• Backend and channel filters keep a large catalog manageable.
• Row checkboxes are real selection state; moving the cursor does not select anything.
• Refresh inspects installed containers without starting them.
• Check Updates compares the registry tag timestamp with the local container creation time.
• Create / Update pulls selected images and recreates only confirmed update targets.
• Enter suspends the TUI while an interactive Toolbx/Distrobox shell is active.
• Delete always asks for confirmation.
• Model Manager opens ComfyUI's maintained in-toolbox model_manager.

The catalog covers AMD Strix Halo, Radeon AI PRO R9700, NVIDIA GB10, and Intel Arc B70. Server-only images are marked in the list: Create / Update pulls their image directly, Enter explains how to launch them in Server Mode, and Delete removes the cached image without deleting host model files. These images only require Podman or Docker; Toolbx/Distrobox is unnecessary.

Every server endpoint has its own source file and pure command builder under ai_toolbox_cockpit/backends/<backend>/.

The vLLM catalog imports the toolbox's model launch recipe rather than replacing it with generic defaults. Model-specific environment variables, parser flags, valid tensor-parallel sizes, eager mode, context, and locked attention implementations are applied by the command builder. DeepSeek V4, for example, keeps its model-specific sparse MLA path and does not receive a generic --attention-backend flag.

On R9700, install the rolling GGZ14, Radiance, native-vLLM FP8 or current R9V channel from Toolboxes. Use Update to pull a newer image. In Models, download the exact curated artifact; the AMD 27B MXFP4 target additionally needs Prepare MXFP4, which writes a separate checkpoint and converter receipt. The DFlash2 FP8 draft is a separate downloadable artifact. See GGZ14 and Radiance for engine-specific setup.

Server Mode → Inference engine and Models → Inference engine list llama.cpp, GGZ14, Radiance, vLLM TheRock, tcclaviger and R9V directly on R9700. Each named engine shows its compatible curated weights and preparation controls. llama.cpp Models defaults to the tested Unsloth Qwen3.8 27B UD-Q4_K_XL and Flash Next UD-Q2_K_XL repositories; the GGUF picker marks and prioritizes the measured quantization, including complete shard sets. Choose All curated repositories for other models. The Model Manager button on a selected toolbox opens its matching Models view.

Toolboxes lists engine installations. GGZ14 has one entry: use Server Mode → GPU build to choose one card (RX9) or two cards (RX6). The matching rolling image and launch settings are selected together. Use Download / Update build for that channel; a missing image also downloads on first server launch. For llama.cpp, choose ROCm 10.0 or Vulkan RADV, then use Server Mode → Model / GPU profile for the measured Qwen 27B single/dual-card, F16/Q8 KV, or Flash Next disk-PLE settings. These profiles share the installed engine image. Radiance has one toolbox entry; selecting the FP8 or MXFP4 model applies its matching launch settings. Existing containers and model files are preserved when upgrading.

Server Mode offers everyday C1 and explicit measured server allocations. The benchmark's client concurrency is a separate setting. Use Discover GPUs in the selected engine image and enter the intended R9700 indices; enumeration can differ between ROCm and Vulkan. Flash Next keeps PLE on disk, and R9V's 64 GB profile disables the older large-context buttons. Rolling builds can change performance; recorded benchmark results identify the actual measured build.

For Flash Next on two R9700s and a 64 GB host, select tcclaviger Flash Next · 2 R9700 cards · 64 GB RAM in Toolboxes. Install pulls the publisher's rolling docker.io/tcclaviger/vllm:latest server image directly. In vLLM Models, download Qwen3.8 Flash Next · tcclaviger MXFP4/FP8/GPTQ · dual R9700 (about 120 GB); this checkpoint needs no MXFP4 conversion. In vLLM Server, select that image and model, discover/select the two GPU indices, and set NVMe PLE/cache directory to persistent NVMe storage with about 42 GB free for PLE plus compiled caches. The first load creates the PLE file; later launches reuse it. This directory is separate from the compiled-cache reset controls.

The tcclaviger defaults reproduce the measured TP2/MTP-3 configuration: 36 GiB expert offload, disk PLE with a 4 GiB row cache, FP8 KV, 131072 maximum context, 16 server slots and a 56 GiB container limit with no container swap. The slot allocation is not a claim of C16 throughput: the measured single-user decode was 100.30 tok/s at zero history and 106.68 at 64K; short C2/C4 tests showed limited scaling. Minimum available host RAM was 2.40 GiB, so leave room for other applications. Host swap and kernel settings are unchanged. See the tested configuration and results. Only the dual-card text-serving profile is qualified; the image is a publisher distribution, not a Cockpit source build.

Toolbox-specific policy overrides keep hardware families separate. The GB10 vLLM image uses one GPU, vLLM's automatic CUDA attention selection, and no ROCm-only environment variables.

Purpose-built llama.cpp forks can declare a validated recommended_use profile in toolboxes.json. The server view uses it to explain the intended platform/model pairing, show operational notes, select an installed tested quant, apply fork-specific defaults, and warn before launch when the user deviates. The EngramHalo profile uses this mechanism for its SSD-backed engram, Q8 KV, and combined MTP/ngram recipe.

Server actions are enabled. Starting a server shows its generated command, suspends the TUI, runs the named container in the foreground, and removes that container after Ctrl+C.

models.json has seven backend-specific sections:

• llama_cpp.models: curated GGUF repositories, inference profiles, MTP metadata, vision projector patterns, and compatibility;
• ds4.models: exact filenames, sizes, repositories, family metadata, and server defaults;
• r9v.models: revision-pinned IQ4_XS/FP8-MTP/Q8-vision package with SHA256 hashes and derived PLE metadata;
• halogen.models: revision-pinned HGN bundles with checkpoint, precision overlay or n-gram table, tokenizer files, optional vision, and expected file sizes;
• gufo.models: revision-pinned GGUF target bundles with exact file sizes, MTP/DFlash2/DSpark sidecars, and optional BF16 Qwen vision projectors;
• vllm.models: Hugging Face repository IDs plus the launcher defaults imported from the vLLM toolbox;
• comfyui.bundles: workflow/model families, variant choices, and the toolbox downloader script used by model_manager.

The shipped catalog currently contains 31 llama.cpp repositories, 22 DS4 artifacts, three Gufo bundles, one R9V package, six Halogen bundles, 19 vLLM repositories, and 26 ComfyUI bundles.

llama.cpp, DS4, and Gufo downloads are explicit, confirmed Hugging Face CLI operations. A llama.cpp model can also declare auxiliary downloads, such as a fork-specific MTP sidecar repository, without presenting the sidecar as a standalone main model. The curated R9700 vLLM artifacts have explicit download/repair actions in Models; other vLLM repositories can download from Hub when vllm serve resolves them. ComfyUI downloads are delegated to the image's workflow-aware manager because one workflow may require several checkpoints, encoders, VAEs, and LoRAs.

Tested configuration: Qwen3.8 Flash Next, 2× R9700 32 GB, 64 GB host RAM and NVMe. The engine is offered only on the R9700 platform. It uses the ROCm 10 R9V toolbox.

1. Toolboxes: select R9700, backend R9V (R9700), then create/update the R9V toolbox. Its image is published through the source repository's manual build workflow.
2. Models → R9V: the tested package is preselected: Dyluhn/Qwen3.8-Flash-Next-R9V-IQ4_XS, revision bf836f0c20b6c92fcad4226ad3115eb8a19f7582. Download includes all 19 required files: three IQ4_XS shards, FP8 MTP, Q8 vision, tokenizer/configuration and expert placement. Download confirmation includes the model license; files are SHA256-verified afterward.
3. Set the package and PLE directories, then Prepare PLE. Allow 90.36 GiB for the package plus 26.82 GiB for extracted PLE. Existing files can be reused; Verify SHA256 checks the package and any existing PLE. Scans check sizes.
4. Server Mode → R9V: select the image and package. Defaults are GPU indices 0,1, TP2, MTP2, 67840 context, 1024-token prefill batches, one sequence, and http://127.0.0.1:8004/v1 / model qwen3.8-flash-next. Set paths, GPU indices, API address/port/name and optional API key as needed. Advanced controls expose KV memory, dynamic expert-cache slots, logical expert offload, sequences and extra vLLM arguments. TP2, MTP2, SSD PLE and synchronous scheduling remain fixed.

The current rolling channel uses the qualified 64 GB host profile: 1,748,799,488 KV bytes per GPU and 16 dynamic expert-cache slots. Its older 128K/256K preset buttons are disabled. The retained legacy channel keeps those presets; see the toolbox documentation for their supported setup. PLE stays disk-backed.

Startup takes minutes. The model and PLE mounts are read-only; caches use a separate directory. API keys are redacted from previews and not saved. Podman is the tested runtime; Docker command generation is supported but GPU validation is pending. More sequences or changed memory settings are outside the tested 64 GB profile. Temperature, output length and thinking are per-request settings. Vision recognized test shapes but did not always return strict unfenced JSON.

Tested configuration: DeepSeek V4.1 Flash Q2, AMD Strix Halo / gfx1151. The model is catalogued with the tested 262144 context, the matching DeepSeek-V4.1-Flash-Vision.gguf encoder, and SSD expert streaming with a 92GB routed-expert cache. Select Server Mode → DwarfStar (ds4) and the Q2 model; the compatible vision encoder is offered automatically and the API defaults to port 8000.

The Models panel also offers the revision-pinned 7.42 GiB native-MXFP4 DeepSeek-V4.1-Flash-DSpark-MXFP4.gguf support model. Once downloaded, Server Mode enables DSpark by default for V4.1 Q2, selects the matching sidecar and leaves confidence on Auto, reproducing the qualified command. DwarfStar currently resolves Auto to 0.7 on ROCm; this is its tested backend default, not a universal threshold prescribed by DeepSeek. Enter a value from 0 to 1 to override it. The support model fixes the draft cap at five tokens and ds4 tunes its adaptive scheduler automatically, so --mtp-draft does not control this DSpark path.

• Standalone: SSD streaming starts enabled with the 92GB cache. Clear it to keep the experts fully resident when the host has enough RAM.
• Coordinator/Worker: selecting a distributed role reveals the tensor-parallel controls. --tensor-parallel is enabled, transport defaults to TCP, and choosing RoCE enables the RDMA device, port and GID-index fields. Cluster experts stay resident; SSD streaming is not applied in distributed mode. The Worker role runs the ds4 CLI instead of the toolbox default ds4-server; it serves no HTTP API, so host and port are not passed and those fields are disabled while the worker is selected. The coordinator keeps ds4-server. DSpark remains available for the resident tensor-parallel path and passes its support model and confidence setting to both ranks. Bounded replay remains a separate opt-in setting.
• InfiniBand: every DwarfStar container gets the host's InfiniBand devices automatically when /dev/infiniband exists. Podman receives the device directory with the rdma group and an unlimited memlock ulimit; Docker receives each device node individually. Hosts without InfiniBand are unchanged and no flags are added.
• The coordinator link address defaults to port 9911; both hosts need matching builds, model and vision files, and context settings.

Halogen Flash is a third-party closed-source project and container for Qwen3.8-Flash-Next on gfx1151. Use at your own risk: its implementation is harder to inspect and audit than the open-source backends.

Cockpit creates the Halogen server container with --network=none, giving it no direct Internet, LAN, or host-network access. It mounts only the selected checkpoint, overlay, tokenizer files, and optional vision weights read-only, plus the GPU devices required by the runtime profile. It does not mount your home, the entire model directory, or engine sockets, and does not pass your Hugging Face token to the server. These restrictions reduce risk but do not remove it: the container still shares the host kernel and GPU driver, retains the profile's host IPC and unconfined seccomp settings, and can return anything it can read through API responses.

The API remains reachable through a Cockpit host listener, which forwards each incoming connection through podman exec -i or docker exec -i to the fixed API port on container loopback. No container ports are published and the relay provides no outbound route. It uses the image's python3, preserves streaming responses, and closes when serving stops. Host defaults to 127.0.0.1; selecting a non-loopback address exposes the API on that interface without adding authentication. Run Cockpit on the GPU host; the relay binds where Cockpit runs.

Cockpit follows ghcr.io/peonist-ai/halogen-flash-server:latest and uses --pull=always with Podman and Docker to fetch the current image before each server launch. The host engine still needs registry access; a failed pull stops the launch. Model downloads also happen separately on the host. Model bundles remain revision-pinned; GPU operation and effective isolation remain experimental until validated on the remote GPU host.

1. Select AMD Strix Halo, select Halogen in Toolboxes, and use Create / Update to pull the image.
2. In Models → Halogen Flash, keep the default v2 bundle or select v2 + vision. Downloads include qwen38-flash-next-v2.hgn, qwen38-flash-next-ngram.hgn, and the tokenizer from the upstream weights repository, pinned to revision 3648cf1e6a3143e8946d52301570f26003c0bdb0. The text bundle needs 109.8 GiB; vision adds 0.84 GiB. v2 requires Halogen server 0.15 or newer and carries its own precision, so it needs no W4B overlay. W4B quality/speed bundles remain available (about 118 GiB). Downloading v2 keeps existing W4B files; allow space for both if keeping them together.
3. The default folder is ~/halogen-models. Save Path creates and saves a different folder; Download / Repair also uses the edited folder, creates it after confirmation, and resumes interrupted transfers. Disk availability and local bundle readiness are shown. An optional Hugging Face token uses Cockpit's existing token handling.
4. In Server Mode → Halogen Flash, select the matching bundle and start it. New selections default to v2; an existing saved bundle choice is preserved. Required file sizes are checked before launch, including the n-gram table or W4B overlay and tokenizer. For v2, Cockpit sets HALOGEN_NGRAM_TABLE and HALOGEN_CK_OVERLAY=none. This detects missing/truncated files, but is not a checksum verification. Individual bundle files are mounted read-only and the image's own entrypoint starts both engine and API. The host API relay defaults to 127.0.0.1:8731.
5. Adjust native request context (up to 262144), KV pool positions, slots, or prompt-cache mode as needed. Defaults follow release 0.5.4; extended YaRN context is not exposed. Ctrl+C stops the server and returns to Cockpit.

For image input, download a + vision bundle and select it in Server Mode. It adds the optional qwen38-flash-next-vision.hgn sidecar (0.84 GiB), reusing the checkpoint, n-gram table or W4B overlay, and tokenizer. Cockpit checks the sidecar before launch and sets HALOGEN_VISION_TOWER to its mounted path. Text bundles leave vision off. Both /v1/chat/completions and /v1/responses accept images as data URLs or base64; upstream refuses remote HTTP(S) image URLs. The server's default image limit is 2560×1440 pixels before downscaling. See the upstream image flags.

The toolbox_compatible: false field in toolboxes.json selects the shared server-only image lifecycle. It defaults to true for existing toolboxes. A server-only record must declare interactive support unavailable and server support available or experimental.

Platform state is global and saved in ~/.config/ai-toolbox-cockpit/config.json. Backend model directories and compiler-cache paths live in the same version-independent configuration.

ai_toolbox_cockpit/
├── app.py                     # thin app shell, theme, platform state, update notice
├── assets/
│   ├── toolboxes.json         # platforms, full OCI refs, container names, capabilities
│   └── models.json            # seven backend-specific model/bundle schemas
├── catalog/                   # typed loading and cross-reference validation
├── runtime/                   # engines, Toolbx/Distrobox, registry, process lifecycle
├── views/                     # unified Toolboxes, Server Mode, and Models shells
└── backends/
    ├── llama_cpp/             # server and GGUF manager
    ├── ds4/                   # server and exact-artifact model manager
    ├── r9v/                   # R9700 server, pinned package and PLE preparation
    ├── halogen/               # Strix Halo server and HGN precision bundles
    ├── gufo/                  # experimental Strix Halo server and pinned GGUF bundles
    ├── vllm/                  # server and HF defaults/cache browser
    └── comfyui/               # server and workflow-bundle/model-manager bridge

Important boundaries:

• JSON is data only; it cannot provide arbitrary shell templates.
• Runtime container names and display names are separate fields.
• app.py does not construct backend commands.
• Interactive operations suspend Textual before taking over the terminal.
• API keys are entered at launch time and are not persisted.

The complete schema rules and backend addition sequence are in docs/ARCHITECTURE.md. The one-backend-at-a-time hardware test procedure is in docs/REMOTE_VALIDATION.md.

The complete schema and extension contract are in docs/ARCHITECTURE.md.

• Versioned catalogs and validation
• Platform/backend/channel navigation
• Podman/Docker and Toolbx/Distrobox discovery
• Full toolbox lifecycle and Docker Hub update checks
• Persistent settings and migration reads from the old cockpit files

• llama.cpp GGUF manager, server mode, vision, profiles, and MTP
• DS4 exact model manager, standalone/distributed server mode, disk KV cache, SSD streaming, and model defaults

• vLLM defaults-aware HF/cache browser and direct server launcher
• ComfyUI workflow catalog, existing model-manager bridge, and direct server launcher

The code and generated commands are covered locally. The GB10 llama.cpp, DS4, and vLLM paths have been exercised with real models; additional platform/backend combinations still need one-at-a-time validation on their intended hardware. See IMPLEMENTATION_NOTES.md. The executable checklist is in docs/REMOTE_VALIDATION.md.

• Extend the GB10 catalog beyond its llama.cpp, DS4, and vLLM toolboxes
• Add future backend packages without changing the app shell

The local suite does not pull images, create containers, download models, or run a GPU workload:

python -m pytest -q
python -m compileall -q ai_toolbox_cockpit
python -m pip wheel --no-deps --no-build-isolation . --wheel-dir dist

Command tests cover toolbox operations and representative llama.cpp, DS4, Gufo, vLLM, and ComfyUI launch construction. The Textual smoke test mocks container inspection and the update network call.