[Bug]: GPU fit grades against total VRAM while CPU paths grade against available RAM, so plan reports a fit on a full card #835

Bug description

plan's fit grading compares against a different kind of memory figure on the GPU path than on the CPU paths, and only the CPU side reflects what is actually free.

After #813, build_path_estimate grades each path against the pool it uses:

// GPU — total VRAM
let available_vram = system.total_gpu_vram_gb.or(system.gpu_vram_gb).unwrap_or(0.0);
let fit = fit_level_for(path, min_vram, available_vram, rec_vram);

// CPU paths — *available* RAM
let fit = fit_level_for(path, min_ram, system.available_ram_gb, rec_ram);

evaluate_current does the same. So the CPU verdicts move as system load changes while the GPU verdict is blind to anything already resident on the card. You flagged the flappiness half of this on #813; this is the other half.

The cause looks like a capability gap rather than a deliberate split. SystemSpecs::gpu_available_gb exists but is Metal-only — detect_gpu_available_gb() calls recommendedMaxWorkingSetSize() and returns None on every non-macOS target, and it is only consulted at all when unified_memory is true. On CUDA/ROCm there is simply no free-VRAM figure to grade against: try_nvidia_smi_* queries memory.total and addressing_mode, never memory.free.

Expected behavior

Either both pools describe total capacity ("could this machine ever run it") or both describe what is free right now ("will it fit at the moment") — and on a machine whose VRAM is largely occupied, the GPU path should not report that a model fits.

Actual behavior

On a 24 GB RTX 3090 with a vLLM engine already holding ~21.8 GB, so 1.09 GB actually free:

$ nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv,noheader
24576 MiB, 23010 MiB, 1112 MiB

$ llmfit --json plan "Qwen/Qwen3.6-35B-A3B" --context 32768 --quant Q4_K_M
needs VRAM      : 23.85 GB
gpu fit_level   : Marginal
current verdict : {'fit_level': 'Marginal', 'run_mode': 'Gpu', 'estimated_tps': 102.9}

Marginal reads as "tight but it fits", and current picks the GPU path at ~103 tok/s. Loading it would OOM immediately — it needs 23.85 GB against 1.09 GB free. Meanwhile a CPU-path verdict on the same machine would have accounted for the memory in use, because those arms use available_ram_gb.

This is not a regression from #813 — before it, the GPU arm compared the requirement against itself and was even less informative. It is the next layer down.

Steps to reproduce

1. Occupy most of VRAM (any long-lived process: vllm serve, a llama.cpp server, a training job).
2. llmfit --json plan "<model sized near total VRAM>" --context 32768 --quant Q4_K_M
3. The GPU path grades on total VRAM, so it reports Marginal/Perfect regardless of what is free.

Diagnostic report (llmfit doctor)

# llmfit doctor report

Paste this whole report into a GitHub issue at https://github.com/AlexsJones/llmfit/issues — the raw tool output below is what lets detection bugs become regression tests. It contains hardware model names and driver info only.

- llmfit version: 1.1.6
- OS: linux (x86_64)

## Detected by llmfit

SystemSpecs {
total_ram_gb: 31.265457153320313,
available_ram_gb: 25.987503051757813,
total_cpu_cores: 24,
cpu_name: "AMD Ryzen 9 3900X 12-Core Processor",
has_gpu: true,
gpu_vram_gb: Some(
24.0,
),
total_gpu_vram_gb: Some(
24.0,
),
gpu_available_gb: None,
gpu_name: Some(
"NVIDIA GeForce RTX 3090",
),
gpu_count: 1,
unified_memory: false,
backend: Cuda,
gpus: [
GpuInfo {
name: "NVIDIA GeForce RTX 3090",
vram_gb: Some(
24.0,
),
backend: Cuda,
count: 1,
unified_memory: false,
},
],
cluster_mode: false,
cluster_node_count: 0,
}

## nvidia-smi (extended query)

None, 24576, NVIDIA GeForce RTX 3090

## nvidia-smi (standard query)

24576, NVIDIA GeForce RTX 3090

## rocm-smi --showmeminfo vram

(not available: No such file or directory (os error 2))

## rocm-smi --showproductname

(not available: No such file or directory (os error 2))

## sysfs DRM cards

card1: vendor=0x10de device=0x2204 driver=nvidia mem_info_vram_total=-

## lspci (display controllers)

2d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)

## vulkaninfo --summary

(not available: No such file or directory (os error 2))

## npu-smi info

(not available: No such file or directory (os error 2))

## Provider app installs

LM Studio installed: false
Docker Desktop installed: false
ollama on PATH: false

Note gpu_available_gb: None — expected on CUDA, and precisely the gap.

llmfit version

llmfit 1.1.8

Operating system

Linux (Ubuntu 22.04.5, kernel 6.8.0)

Affected component

Model fit analysis (scoring, fit levels)

Mode used

Subcommand (plan) — fit/TUI share SystemSpecs, so the same asymmetry applies wherever VRAM headroom is graded.

GPU details

NVIDIA GeForce RTX 3090, 24576 MiB total, driver 580.173.02
AMD Ryzen 9 3900X (24 threads), 32 GB RAM (31.27 GiB visible to the OS)

Additional context

Three directions, and the choice is a semantics call like #806 turned out to be:

A — both use total. Switch the CPU arms to total_ram_gb. Smallest diff, and it makes verdicts reproducible between runs, which fixes the flappiness you raised. But it answers a weaker question, and the failure above stays: llmfit still says a model fits when the card is full.

B — both use available. Populate gpu_available_gb on discrete GPUs (nvidia-smi --query-gpu=memory.free, rocm-smi --showmeminfo vram gives used/total) and grade against it, keeping total as the fallback when the query fails. Most accurate, and it makes plan trustworthy on a busy machine — at the cost of the flappiness being real on both pools now, plus new detection code to maintain per backend.

C — grade on total, report the shortfall. Keep grading stable, but surface free memory when it is materially below total: a note on the path, or a memory_available_now_gb field. Consumers keep reproducible fit levels and gain the information needed to catch the OOM case.

My instinct is B for correctness with total as fallback, since "will this run right now" is what plan is asked, and nvidia-smi already exposes the number. But C is the conservative option and preserves the reproducibility argument, so I would rather have your call than guess as I did on #806.

Happy to implement whichever you prefer.