Skip to content

nearest_mem_tier has no tier below 8 GB, and the two copies have diverged #816

Description

@Akciali

nearest_mem_tier exists twice, with two different ladders, and neither covers GPUs
below 8 GB.

Surfaced by the Greptile review on #815, which flagged that a 4 GB GTX 1050 Ti was
reporting memTierGb: 8:
#815 (comment)

The value was not hand-written — it is what llmfit 1.1.3 emits. Digging into why
turned up two separate problems.

1. No tier below 8 GB

Both copies start the ladder at 8, so every sub-8 GB GPU is reported as an 8 GB part:

actual VRAM reported memTierGb
2.0 8
4.0 8
6.0 8
8.0 8

For a capacity field this errs in the dangerous direction — it overstates memory. A
4 GB card is grouped with 8 GB cards in any tier-based comparison, and in
benchmarks.rs it also queries community data recorded on 8 GB hardware, which is
precisely the data it should not be matched against.

This has stayed invisible so far because every existing entry in
llmfit-core/data/community/ is 8 GB or more, so nearest_mem_tier happens to return
their real VRAM. The 1050 Ti is the first submission below the lowest rung.

2. The two copies have diverged

llmfit-core/src/benchmarks.rs:305

const TIERS: [u32; 9] = [8, 12, 16, 24, 32, 48, 80, 96, 128];

llmfit-core/src/share.rs:116

const TIERS: [u32; 12] = [8, 12, 16, 24, 32, 48, 64, 80, 96, 128, 192, 256];

Same name, same algorithm, different constants — share.rs has 64, 192 and 256,
benchmarks.rs does not. The bodies differ only by a local variable name
(best_dist vs best_d), which suggests one was copied from the other and they
drifted.

The consequence is that a single machine can be classified into two different tiers
depending on the code path:

VRAM benchmarks.rs (lookup) share.rs (submission)
60 GB 48 64
150 GB 128 192
200 GB 128 256

The existing tesla-t4 entry carries memTierGb: 64, a value the benchmarks.rs
ladder cannot produce — so that submission can never match its own recorded tier when
the lookup path runs.

Call sites

  • benchmarks.rs:280 and :285 — builds the memTier query parameter for the
    community benchmark lookup
  • share.rs:139 and :142 — writes memTierGb into the submission payload

3. Which capacities actually exist below 8 GB

The gap is not hypothetical. Cards that shipped under 8 GB and are still in wide
circulation:

VRAM cards current tier error
2 GB GT 710, GTX 1050 8 +300%
3 GB GTX 1060 3 GB, GTX 780 8 +167%
4 GB GTX 1050 Ti, RX 570 4 GB 8 +100%
6 GB GTX 1060 6 GB, RTX 2060, RTX 3050 6 GB 8 +33%

The GTX 1060 3 GB in particular is one of the highest-volume discrete GPUs ever
shipped, and it is currently declared at nearly three times its real capacity.

4. A separate, milder gap at 10 and 11 GB

Worth flagging while you are in here, though it is a different problem and I am not
proposing to touch it:

VRAM cards tier error direction
10 GB RTX 3080 10 GB 8 -20% pessimistic
11 GB GTX 1080 Ti, RTX 2080 Ti 12 +9% optimistic

10 GB sits exactly between 8 and 12; the d < best_d comparison keeps the first
match, so it lands on 8. 11 GB is strictly closer to 12.

The directions matter because this value also drives the lookup in benchmarks.rs.
A 3080 owner is served results from 8 GB cards, which undersells the card but never
promises a model that will not fit. An 11 GB owner is served results from 12 GB
cards, which can promise a model that OOMs. That is the direction that actually
hurts, but the magnitude (9%) is within what a coarse bucket is presumably meant to
absorb — unlike the +100% to +300% at the bottom of the ladder.

Suggested direction, revised

The doc comment describes these as "coarse buckets so submissions group cleanly", and
coarse grouping is clearly the intent. So the fix I would suggest is narrow: give the
bottom of the ladder a floor, and leave everything at and above 8 alone.

For the submission path only (share.rs):

const TIERS: [u32; 16] = [2, 3, 4, 6, 8, 12, 16, 24, 32, 48, 64, 80, 96, 128, 192, 256];

Properties, deliberately:

  • No tier at or above 8 moves, so no previously submitted value changes.
  • Only inputs below about 7 GB are affected, and no shipping card sits in the
    6.5-7.0 band that shifts.
  • Exact ties keep resolving downward, which understates rather than overstates — the
    safe direction for a declared capacity.
  • The existing test mem_tier_rounds_to_nearest (7.5 -> 8) still passes.

I would not dedupe the two copies, despite that being the obvious reading of
section 2. nearest is the correct rule for a lookup key and the wrong one for a
declared capacity; merging them would freeze that conflation rather than resolve it.
If you want them unified, they probably want two names and two intents rather than
one function.

I have this written and tested locally against main (494 tests pass,
cargo fmt --check and cargo clippy --all-targets --all-features clean) but have
not opened a PR — happy to send it if the shape above is what you want, or to adjust
it if you would rather include 10 and 11, or handle the duplication differently.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions