fix(models): read a discovered model's real capacity from the probe instead of hardcoding 131072 - #7817

Closes #7780.

What was wrong

merge_discovered_models (crates/librefang-runtime/src/model_catalog.rs) is the only writer for gateway- and locally-discovered catalog entries, and it hardcoded two literals:

context_window: 131_072,
max_output_tokens: 16_384,

The issue reads that as "no upstream can supply a real value", and that half is true — DiscoveredModelInfo had no field for either one. But the reason it had no field is one layer further down: parse_openai_models (crates/librefang-runtime/src/provider_health.rs) returned model_info: vec![] unconditionally, so every capacity field an OpenAI-compatible listing did carry was parsed away and discarded. Fixing the merge site alone would have papered over a probe that was throwing the answer on the floor.

Why a wrong context window is not cosmetic

The fabricated value is indistinguishable from a measured one at every consumer, and it lands directly in the turn's token budget:

• manifest_helpers::resolve_context_window (crates/librefang-kernel/src/kernel/manifest_helpers.rs:117) takes the catalog entry's context_window with .filter(|w| *w > 0). 131_072 > 0, so the guess is accepted as arm 2 of the precedence chain.
• agent_loop::mod.rs:878 turns that into ContextBudget::new(ctx_window). Compaction fires at 80% of the window.

So an 8K model behind a LiteLLM gateway got a 131K budget: compaction never fires, the daemon packs a prompt to ~104K tokens, and the provider rejects the request after the input tokens have been billed. That is precisely the failure the fallback comment three lines above already warns about ("a 200K assumption silently bills the user for prompts the provider then rejects with HTTP 400 (#3349)") — the discovered-entry literal reintroduced it by a different route.

The opposite direction is equally silent: a 1M-context model reached through a gateway is clamped to 131K, and ContextCompressor summarises away prompt content nobody asked to drop.

Neither failure is visible from either end, which is what makes this an area/security defect rather than a display bug: the operator reading 131072 in the Models page has every reason to believe LibreFang learned it from the gateway.

What changed

Probe layer — read what the endpoint reports.
DiscoveredModelInfo gains context_window: Option<u64> / max_output_tokens: Option<u64>, and parse_openai_models populates them through the key-priority parsers in model_metadata, which already encoded the per-server key zoo for the single-model endpoint and are now pub(crate) and shared: vLLM max_model_len, LM Studio / llama.cpp context_length, some proxies' context_window, LiteLLM max_input_tokens / max_output_tokens, OpenRouter context_length and top_provider.max_completion_tokens. A second parser, parse_openai_model_max_output, handles the output ceiling with a key list deliberately disjoint from the context-window list — max_tokens is already claimed there as a last-ditch reading of the full window, and counting it as both would silently assert an output ceiling equal to the whole context.

parse_ollama_tags reports None for both: /api/tags genuinely carries no token capacity (it lives in /api/show's model_info, which costs one extra request per model). Guessing from the family name would be the same defect wearing a different hat.

Merge site — never invent.
An entry now gets numbers only when the endpoint supplied them. When it did not, both fields stay at the catalog's documented 0 ("unknown") encoding, and every existing > 0 guard falls through to the conservative UNKNOWN_MODEL_CONTEXT_WINDOW (8192) whose warning already names agent.toml: model.context_window as the operator's fix. A conservative window plus a loud warning is recoverable; an invented one is not observable from either side.

Capacity also follows the same never-downgrade rule the capability flags already follow: a later probe fills in a limit that is still unknown, and a probe that stops reporting one (an older proxy in front of an upgraded gateway) cannot erase what an earlier probe learned.

limits_known on ModelCatalogEntry — adopting #7786's shape deliberately.
0 alone is ambiguous, and the repo's own doc comment says so: it means "unknown / not applicable". An image or audio entry legitimately has no token context. A discovered model with no reported capacity does have one and we do not know it. Those two behave differently for any consumer that wants to render a badge or clamp a request, and they were indistinguishable once written into the entry. limits_known records provenance: true when a registry entry, an operator override, or the endpoint supplied the numbers; false when nothing did. It mirrors the existing pricing_known field exactly — same #[serde(default = "default_true")], same "older entries predate this and carry real numbers" reasoning — so it is a convention the catalog already has rather than a new one.

It is threaded through model_metadata::synthesize_entry as well, because that pipeline's L5 substring table and its two provider-shaped defaults are guesses this crate invented, and leaving them marked as sourced would contradict the field's own contract. MetadataSource already drew that line; the flag puts it on the entry so a caller holding only the entry can see it too.

The three model routes emit limits_known next to pricing_known, which is what makes the fix assertable end to end.

Drive-by, forced by the new fields. The bare DiscoveredModelInfo { name, None, None, … } literal was triplicated across librefang-api (twice) and librefang-kernel (once) purely because parse_openai_models discarded its info. All three now call one DiscoveredModelInfo::bare(name) constructor, so a future field cannot be quietly given a fabricated value at one of several sites. Net deletion.

Collision with #7786 — and which side should rebase

Stated explicitly because it is a hard same-hunk conflict, not a merge hiccup. PR #7786 (feat/modelparams-tristate, @DaBlitzStein) inserts limits_known: false plus a matching field into the same ModelCatalogEntry hunk in crates/librefang-types/src/model_catalog.rs, and also touches crates/librefang-runtime/src/model_catalog.rs, crates/librefang-runtime/src/model_metadata.rs and crates/librefang-testing/src/mock_kernel.rs — every one of them a file this PR changes.

I did not touch that branch. The honest engineering answer is that the tri-state is the right model, so this PR adopts that shape rather than fighting it: limits_known: bool with default = "default_true", matching pricing_known. The two designs now agree on the field name and its semantics, which turns a design conflict into a textual one.

#7786 should rebase onto this, for one reason and one reason only: #7786 is already CONFLICTING against main independently of this PR, so it has to be rebased regardless, and after that rebase limits_known already exists and its insertion of the field simply drops out of the diff. Rebasing this PR onto #7786 instead would block a one-crate correctness fix behind a 44-file feature branch that is not landing imminently. If the maintainer prefers the other order, this branch is small and I will rebase it.

Interaction with #7816 (fix/7775-openai-compatible-discovery)

Flagging rather than acting, since it is not my branch. PR #7816 is MERGEABLE and adds a fourth copy of the bare DiscoveredModelInfo literal in crates/librefang-api/src/routes/providers.rs (in the new merge_probe_into_catalog). That copy will not compile against the two new fields, and it conflicts textually with the two copies this PR replaced with DiscoveredModelInfo::bare. Whichever lands second needs the one-line change to ::bare(name).

Worth noting for the reviewer: #7816's changelog says "being in the catalog is also what stops the runtime assuming an 8k context window for a model that handles far more". After this PR, a gateway model whose endpoint reports no capacity does get the conservative 8K — because that is the honest answer, and the issue asks for exactly that ("the compaction/budget math can treat it conservatively"). The two PRs are complementary rather than opposed: #7816 makes the model reachable, this one makes sure the number attached to it came from somewhere. The operator-facing remedy for the still-unknown case is agent.toml: model.context_window, which the fallback warning already names, and #7774 is the editing surface for it.

Verification

Commands and their actual output are in the collapsed section below. Summary:

• cargo check -p librefang-api -p librefang-cli -p librefang-kernel-metering -p librefang-testing --lib — clean.
• cargo test -p librefang-runtime --lib -- model_catalog::tests::test_merge_ provider_health::tests:: model_metadata::tests:: — 74 passed, 0 failed.
• cargo test -p librefang-api --test providers_routes_test -- list_models — 4 passed, 0 failed, including the new integration test.

New tests:

• provider_health::tests::test_parse_openai_models_reads_reported_capacity — one listing body covering the vLLM, LiteLLM and OpenRouter key shapes plus a silent entry; asserts each capacity reaches DiscoveredModelInfo and the silent one stays None.
• provider_health::tests::test_parse_openai_models_rejects_zero_capacity — a server reporting 0 is not recorded as having answered.
• model_catalog::tests::test_merge_records_capacity_reported_by_the_gateway — the reported-capacity requirement: gateway says 8192, entry carries 8192, limits_known is true.
• model_catalog::tests::test_merge_does_not_invent_capacity_when_the_gateway_reports_none — the unknown-capacity requirement: asserts != 131_072 explicitly as well as == 0, so the literal cannot come back by any route, and limits_known is false.
• model_catalog::tests::test_merge_accepts_a_partially_reported_capacity — a gateway reporting only the window.
• model_catalog::tests::test_merge_upgrades_unknown_capacity_but_never_erases_a_known_one — three probes: silent, reporting, silent again; the measured window survives.
• providers_routes_test::list_models_serves_probed_capacity_and_never_the_hardcoded_literal — the same two cases end to end through GET /api/models against the real router, asserting the JSON body.

Not run locally, and why: cargo clippy and any wider build. This machine's shared CARGO_TARGET_DIR is used concurrently by ~28 other worktrees; free disk fell from 45 GiB to 2.4 GiB during this session and I stopped building at that point rather than take the machine down. Clippy is left to CI. One clippy hazard was fixed by reading rather than running: the capacity back-fill in the in-place upgrade branch is written as two flat if field == 0 branches instead of a nested if let, since the enclosing block would otherwise contain only the inner if and trip collapsible_if.

A note on local test runs in this repo, since it cost me four wasted invocations: with a shared CARGO_TARGET_DIR across worktrees, cargo test reported Fresh librefang-runtime and then ran a test binary containing another worktree's tests (#7774's, by name), silently omitting mine while reporting ok. touching the edited sources before the run forces a real rebuild. Every result quoted above was taken after such a touch, and I verified by name that each new test actually appears in the run output.

$ cargo check -p librefang-api -p librefang-cli -p librefang-kernel-metering -p librefang-testing --lib
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 11.38s

$ cargo test -p librefang-runtime --lib -- model_catalog::tests::test_merge_ provider_health::tests:: model_metadata::tests::
test model_catalog::tests::test_merge_accepts_a_partially_reported_capacity ... ok
test model_catalog::tests::test_merge_records_capacity_reported_by_the_gateway ... ok
test model_catalog::tests::test_merge_upgrades_unknown_capacity_but_never_erases_a_known_one ... ok
test model_catalog::tests::test_merge_does_not_invent_capacity_when_the_gateway_reports_none ... ok
test provider_health::tests::test_parse_openai_models_reads_reported_capacity ... ok
test provider_health::tests::test_parse_openai_models_rejects_zero_capacity ... ok
test result: ok. 74 passed; 0 failed; 0 ignored; 0 measured; 2230 filtered out; finished in 0.39s

$ cargo test -p librefang-api --test providers_routes_test -- list_models
test list_models_surfaces_cli_profile_configured_model ... ok
test list_models_serves_probed_capacity_and_never_the_hardcoded_literal ... ok
test list_models_returns_well_formed_envelope ... ok
test list_models_filters_by_unknown_provider_yields_empty ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 56 filtered out; finished in 5.89s

Deliberately out of scope

• Dashboard. The investigation listed five locale files and a ModelsPage change; none are needed. formatCtx in ModelsPage.tsx:41 starts with if (!tokens) return "—", so an unknown capacity already renders as an em dash rather than as 0 or as a fabricated 131K. No new user-facing string exists, so no locale key does either. The limits_known flag is on the wire for whoever wants to render a badge — fix(models): agent inference settings win over the per-model override #7786's UI work, most likely.
• Option<u64> typing for api.ts. Left for the PR that actually consumes the flag in the UI, rather than adding an unread field to the TS model type here.
• Issue bug(models): a discovered model gets a hardcoded 131072-token context window that nothing can source or correct #7780's third ask ("let the operator correct it, once, permanently") is feat: model configuration, context window included, must be editable at any time from the WebUI, the TUI and the API #7774's editing surface plus override-survives-sync, and is not in this diff. What is in this diff is the half bug(models): a discovered model gets a hardcoded 131072-token context window that nothing can source or correct #7780 asks of the discovery path: do not overwrite, and do not pretend the default was measured. ModelOverrides is a separate map on the catalog and is untouched by merge_discovered_models, so an operator-set window already survives a re-probe.
• The other fabricators. parse_openrouter_model_entries (model_catalog.rs:2105) does the same .unwrap_or(131_072) for OpenRouter, and model_metadata's L5 substring table is a family-name guess. Both are now marked honestly via limits_known (L5 and the provider defaults are false), but their numbers are unchanged: zeroing them would change behaviour in paths this issue does not cover, and OpenRouter's is a curated live catalog rather than an operator's private gateway. Worth a separate issue if the maintainer wants the clamp bug(models): a discovered model gets a hardcoded 131072-token context window that nothing can source or correct #7780's comment thread contemplates, since a clamp is only safe against a limits_known: true ceiling.
• Ollama /api/show. Reading the real llama.context_length per discovered Ollama model means one extra HTTP request per model on every probe. That is a discovery-cost decision, not a correctness one, and I did not want to make it inside a fix for the "do not invent numbers" bug.