plan graded CPU paths against available_ram_gb but GPU paths against
total VRAM, so a 24 GB card with 1.09 GB free (a resident vLLM engine)
reported 'Marginal' on the GPU path and picked it as the current run
mode; loading would OOM immediately (#835).
Scope as agreed in the issue: pooled available VRAM, total as fallback,
per-card placement left for a follow-up.
- GpuInfo gains free_vram_gb, summed over a model group's cards. NVIDIA
reads it from one more column in the nvidia-smi query already run
(no extra process); AMD from mem_info_vram_used next to the
mem_info_vram_total already read. The parser still accepts the old
three-column output.
- SystemSpecs.gpu_available_gb is populated on discrete GPUs as the pooled
sum. All-or-nothing: one card without a reading yields None, since a
partial sum would understate what is free. Hardware overrides already
reset it, so simulated hardware grades against capacity.
- SystemSpecs::gpu_fit_pool_gb() is what plan grades against: free VRAM
clamped to total when known, total otherwise. Unified memory keeps the
total pool; there the field is Metal's wiring cap, not a free reading.
- Upgrade deltas stay on total VRAM: what to buy is a capacity question.
- The GPU path notes when it was graded against free VRAM; system output
and /api/v1/system show the figure.
Fixes #835
Problem
#835:
plangrades CPU paths againstavailable_ram_gbbut GPU paths against total VRAM. On a 24 GB RTX 3090 with a vLLM engine holding 21.8 GB (1.09 GB free), a 23.85 GB model gradedMarginalon the GPU path andcurrentpicked it at ~103 tok/s. Loading it would OOM immediately. @Akciali reproduced it on a mixed RX 7900 XT + RTX 2080 Vulkan box, so it is not CUDA-specific.Scope
As settled in the thread: populate
gpu_available_gbfor NVIDIA/AMD, grade GPU paths against pooled available VRAM, fall back to total where detection is unavailable. Per-card placement is left for its own issue. Thanks to @marzmesas and @Akciali for the repro and for pointing out both readings are one field away, and to @SulimanAbdulrazzaq for framing the two options. Nobody had a branch up for this, so I picked it up; happy to fold in anything you had in progress.Change
Detection, no new process spawns
GpuInfo.free_vram_gb: free VRAM summed over the cards of a model group.memory.freeadded to thenvidia-smiquery that already runs (addressing_mode,memory.total,memory.free,name). The parser detects the column, so previously captured three-column output still parses.[N/A]and ATS unified parts yieldNone.mem_info_vram_usedread beside themem_info_vram_totalalready opened. The scanner's(String, Option<f64>)tuple became a smallAmdSysfsCardstruct rather than gaining a second same-typedOption<f64>.SystemSpecs.gpu_available_gbon discrete GPUs = the pooled sum, mirroring howtotal_gpu_vram_gbpools capacity. All-or-nothing: if any card lacks a reading the result isNone, because a partial sum would understate what is free and wrongly fail a fit.Grading
SystemSpecs::gpu_fit_pool_gb(): free VRAM clamped to total when known, total otherwise. Used byevaluate_currentandbuild_path_estimate.gpu_available_gbis Metal's wiring cap, not a free-memory reading, so the total pool is kept.gpu_available_gb, so--gpu-vram/profiles grade against capacity.Visibility: the GPU path adds a note when graded against free VRAM;
llmfit systemshows, X GB free;/api/v1/systemgainsgpus[].free_vram_gb(gpu_available_gbwas already there, previously alwaysnulloff macOS).API.mdupdated.Not covered
nvidia-smifallback for old drivers, androcm-smiall reportNoneand keep total-capacity grading. Each is additive later.fit(the ranking path) still scores against total VRAM. The issue and the agreed scope are aboutplan; changing ranking to depend on transient GPU load deserves its own discussion.Validation
New tests use the exact
nvidia-smiline from the issue (24576, 1112-> 1.09 GB free), grouped and partial multi-card sums, legacy three-column and ATS output, themixedsysfs fixture (16 of 24 GiB used -> 8.0 free, with the filtered iGPU's usage not leaking in), andplanend to end: occupied card -> GPU pathTooTightand not chosen ascurrent; no reading -> identical to before.Not verified on live discrete hardware: my machine is a unified-memory Strix Halo, where output is unchanged as intended. A check from anyone on the thread with an occupied NVIDIA or AMD card would be welcome.
Fixes #835