Add manually triggered model compatibility matrix - #440

run_gpu_ci.sh listed three GPU test modules inline, so the other modules in
the manifest never ran and any newly classified GPU test would be missed
until someone remembered to edit the script as well. ovg-project#443 had already added
one the script would not have picked up. Read the list through
check_test_classification.py --list-category gpu, the same way
run_cpu_tests.sh reads the CPU manifest.

test_ElasticMLAMemoryPool.py imports sglang.srt, so it cannot run in the
core environment the GPU profile uses. Move it to the integration manifest,
which already holds the engine-dependent modules.

test_prealloc_gil_deadlock.py imports only kvcached.vmm_ops and never
touches vLLM or SGLang, so the integration manifest was the wrong home for
it. Nothing runs that manifest, so the ovg-project#371 regression has never executed
in CI. It passes in the core GPU environment in about six seconds.

… trigger

A pull request could only ever run `core`: the `inputs` context is empty
outside workflow_dispatch, so the profile expression always fell through to
its default. That meant "does this change break the vLLM integration" could
not be answered until after the merge. Map five labels onto the five
profiles instead. The specific labels are matched before the plain `gpu-ci`
one so that carrying both does not downgrade the run.

`nixl` needs two devices and a label carries no input, so it falls back to a
new KVCACHED_GPU_NIXL_DEVICES repository variable.

Drop the push trigger on main. It re-ran `core` on the self-hosted runner
for a commit that a labelled pull request had already covered, and the
scheduled `engines` run exercises the default branch daily anyway.

…erges

Add an `all` profile that runs the GPU manifest, both engine smoke tests,
and the two-GPU NIXL prefill/decode smoke test in one pass, plus the
matching `gpu-ci-all` label. Like `nixl` it needs two devices.

Replace the unconditional daily run with a budget. The cron slot now only
asks whether the default branch has moved enough: a plan job counts commits
since the last successful scheduled run and declines the slot below
KVCACHED_GPU_SCHEDULED_MIN_MERGES, default five. Merges are squashed, so one
commit is one pull request. The run history is the state, so nothing extra
has to be stored.

Move profile and device resolution into that plan job. The six-way label
expression was already repeated on the job name and on two environment
variables, and the scheduled path now needs somewhere to say why it
declined.

Rename KVCACHED_GPU_NIXL_DEVICES to KVCACHED_GPU_DUAL_DEVICES since `all`
needs it too, and raise the job timeout to 180 minutes to fit `all`.

The plan lived as a shell block inside the workflow, where nothing could
execute it except GitHub firing the workflow for real. Every way it could be
wrong was silent: a mistyped label falls through to "no gpu-ci label" and the
job is skipped while looking like it ran, and a reordered branch downgrades
an engine request to core. run_gpu_ci.sh is a file for the same reason.

Move it to tools/resolve_gpu_ci_plan.sh, reading the trigger from the
environment, and add tests/test_gpu_ci_plan.py: every label maps to its
profile, a specific label beats the plain one, an unrelated or absent label
declines, a fork declines, dispatch inputs win, the two-GPU profiles take
the dual-device variable, and a missing dual-device variable fails before
the runner is taken.

The checkout was gated on the scheduled event, left over from when the plan
step was an inline shell block that needed no working tree. Once the step
became `bash tools/resolve_gpu_ci_plan.sh`, a pull request reached it with
nothing checked out:

    bash: tools/resolve_gpu_ci_plan.sh: No such file or directory
    Process completed with exit code 127

The existing plan tests could not see this: they run the script from a
checkout, so they exercise its logic and not the job wiring around it. Add
two tests that read the workflow instead, one asserting the checkout is
unconditional and ordered before the plan step, one asserting the script the
workflow names is actually in the repository.

Reviewing a contributor's change before merging it is the main thing the
GPU run is for, and every contributor here works from a fork, so the
same-repository check made the label useless for the case that matters.

Drop it and let the label be the only gate. GitHub withholds secrets from a
fork's pull_request and issues a read-only token, so what a labeller accepts
is code execution on the runner and nothing wider: no secret can be read and
nothing can be written back to the repository. Only write or triage
permission can apply a label, so that acceptance is always a maintainer's.
Settings > Actions > "Require approval for all outside collaborators" adds a
second, GitHub-native gate for anyone who wants one.

Also drop an unused import that ruff rewrote in CI.

The person applying a label is not necessarily the person whose machine
gets registered as the runner, so say out loud that the policy needs the
host owner's agreement rather than leaving it implied.

…ource tree

A fresh checkout could never have passed. The script installs kvcached into
site-packages, then runs `"${PYTHON}" -m pytest` from the repository root;
`python -m` puts the working directory first on sys.path, so `import
kvcached` resolved to the source tree, which carries no compiled extension.
Every GPU test died at collection:

    ModuleNotFoundError: No module named 'kvcached.vmm_ops'

Reproduced by cloning this branch into an empty directory, installing into a
freshly bootstrapped environment, and running the core profile: 3 collection
errors, exit 2. With PYTHONSAFEPATH set for the installing path, the same
clone runs 66 passed, exit 0.

Also print which kvcached and which extension the tests are about to import,
into gpu-ci-artifacts/kvcached-import.txt. One line naming the resolved paths
is worth more than a pile of collection errors when this goes wrong again.

Add GPU_CI_ENVS to the bootstrap so a host that only runs the core profile
can skip the two engine environments and the tens of gigabytes they pull,
and say in the BASE_PYTHON error that Ubuntu 24.04 has no python3.11 package.

…tory variables

A labelled pull request from a fork failed in the plan job:

    LABELS: gpu-ci
    SINGLE_DEVICES:
    no GPU device selection is configured for profile core

The label resolved fine; the device selection did not, because GitHub does
not pass repository variables to a workflow triggered by a pull request from
a fork. Every contributor here works from a fork, so that is precisely the
case the GPU run exists for.

Which GPUs a machine has, and where its interpreters live, are properties of
the machine anyway. Take them from the runner's own .env instead: it reaches
every job whatever triggered it, and a second runner with a different device
count needs no repository change. The plan job stops resolving devices
altogether, which also removes the only part of it that could not work from a
hosted runner. run_gpu_ci.sh still rejects a profile that got the wrong
number of devices, so nothing is unchecked, only checked on the machine that
knows.

Also give the bootstrap subset test stand-ins for nvidia-smi, nvcc and c++ so
it runs on the hosted CPU runner instead of only where a GPU toolchain
exists.

Device selection moved to the runner's .env, but three things still
described the old arrangement. The workflow_dispatch `devices` input was
read by nothing, so filling it in silently did nothing. The bootstrap
printed KVCACHED_GPU_PYTHON and friends, which no script reads, and a
contract test pinned those names in place.

Emit PYTHON, VLLM_PYTHON and SGLANG_PYTHON instead -- the names
run_gpu_ci.sh actually reads -- and say where they go.

Both documents still described the arrangement before the label trigger:
repository variables for the interpreters and the GPU selection, a push to
main running core, forks excluded on principle, and a daily engines run.
None of that is true, and following the setup instructions produced a
runner that failed on exactly the case the run is for.

Also correct two names the reader would have set in vain:
KVCACHED_GPU_REQUIRE_IDLE is read as GPU_CI_REQUIRE_IDLE, and the removed
`devices` input no longer exists.

The bootstrap pinned vllm==0.19.0 and sglang[all]==0.5.10 while the README
claims support up to v0.24.0 and v0.5.15, so a green engine profile said
nothing about the versions users are told to run.

`tests/test_utils.py` holds one config loader and `tests/test_traffic_monitor.py`
is a manual script whose class pytest never collects; both collect zero tests.
They were only in `integration.txt` because `check_test_classification.py`
requires every `tests/test_*.py` to be classified, so the manifest claimed five
integration tests where there were three.

Move them to `tests/helpers/`, which the classifier's non-recursive glob does
not see, and fix the config loader's path -- it walked up from `tests/` to find
`controller/` and now has one more level to climb.

Qwen2.5-1.5B is GQA only, so a green engine profile said nothing about MLA,
sliding-window or hybrid layouts -- which is where kvcached's recent KV-layout
bugs actually were. Qwen3.5-4B is a linear-attention hybrid (three
linear-attention layers per full-attention layer), ungated, and small enough to
stay a smoke test.

kvcached documents the non-contiguous layout for that family, so set it, and
leave both engines' hybrid KV-cache managers enabled -- the attention groups
have different specs and cannot be unified.

The workflow no longer hardcodes a model: the engine smoke and the NIXL P/D
smoke want different ones, and each already carries its own default.

`test_ElasticMLAMemoryPool.py` wraps its whole body in `except Exception`,
prints the traceback and neither re-raises nor exits non-zero, so the MLA pool
could break in any way and the script would still report success. Only the
module-level assert that SGLang's pool class got patched was live.

`test_offline_serving.py` is vLLM's own example with the prompts changed. It
has no assertion at all -- it prints four completions -- and it hardcodes the
gated `meta-llama/Llama-3.2-1B`.

Both would have joined the integration profile as permanent green, which is
worse than no coverage: it reads as tested.

kvcached patches both engines' KV pools, but only vLLM had an end-to-end check
that the mapped footprint grows under load and falls when requests drain. The
SGLang path had unit tests of the allocator and nothing above them -- and the
one test that touched SGLang's MLA pool could not fail.

Everything the check observes is engine agnostic: the /dev/shm segment, the
footprint samples, the limit cut. Only building the engine and issuing a
request differ. Move the phases and the verdict into helpers/elasticity.py and
leave each engine script with an adapter.

`disable_radix_cache` is SGLang's counterpart of vLLM's
`enable_prefix_caching=False`, and is required for the same reason: with prefix
reuse on, finished requests keep their KV resident and no shrink is observable.

Run both from the engine profiles. Until now a green `gpu-ci-vllm` meant only
that vLLM booted and said "Paris".

vLLM 0.24 lets FlashInfer JIT-compile its sampling kernels on the first
request, and FlashInfer shells out to `ninja`. Neither engine declares it, so
the engine core died at startup with a bare
`FileNotFoundError: ... 'ninja'` wrapped in "Engine core initialization
failed", which points nowhere near the cause.

Installing it is only half the fix. pip puts `ninja` in the environment's own
bin directory, and these scripts invoke the interpreter by absolute path, which
does not put that directory on PATH. Prepend it where an engine runs.

Three things had to change together for the smoke test to pass on a hybrid
reasoning model, and none of them showed up until it was run.

The script `cd`s to the repository root and launches the server with
`python -m`, which puts the working directory first on sys.path. The source
tree shadowed the installed kvcached, and it carries no compiled extension, so
both engines died at `ModuleNotFoundError: No module named 'kvcached.vmm_ops'`
-- the same failure run_gpu_ci.sh already guarded the pytest run against.

The correctness request asked a question. Qwen3.5-4B answers a question by
opening `<think>`, so both engines spent the whole 8-token budget reasoning and
never named the city. `/v1/completions` applies no chat template, so ask for a
continuation instead, which the same model answers in one token.

Startup took ~250s here with FlashInfer's JIT cache already warm. A cold cache
compiles first, and 360s left no room for it.

tools/run_gpu_ci.sh writes its logs into the checkout by default, and nothing
ignored them, so a `git add -A` on the runner would commit a run's worth of
engine logs.

…-sync

…-sync