nearest_mem_tier exists twice, with two different ladders, and neither covers GPUs
below 8 GB.
Surfaced by the Greptile review on #815, which flagged that a 4 GB GTX 1050 Ti was
reporting memTierGb: 8:
#815 (comment)
The value was not hand-written — it is what llmfit 1.1.3 emits. Digging into why
turned up two separate problems.
1. No tier below 8 GB
Both copies start the ladder at 8, so every sub-8 GB GPU is reported as an 8 GB part:
| actual VRAM |
reported memTierGb |
| 2.0 |
8 |
| 4.0 |
8 |
| 6.0 |
8 |
| 8.0 |
8 |
For a capacity field this errs in the dangerous direction — it overstates memory. A
4 GB card is grouped with 8 GB cards in any tier-based comparison, and in
benchmarks.rs it also queries community data recorded on 8 GB hardware, which is
precisely the data it should not be matched against.
This has stayed invisible so far because every existing entry in
llmfit-core/data/community/ is 8 GB or more, so nearest_mem_tier happens to return
their real VRAM. The 1050 Ti is the first submission below the lowest rung.
2. The two copies have diverged
llmfit-core/src/benchmarks.rs:305
const TIERS: [u32; 9] = [8, 12, 16, 24, 32, 48, 80, 96, 128];
llmfit-core/src/share.rs:116
const TIERS: [u32; 12] = [8, 12, 16, 24, 32, 48, 64, 80, 96, 128, 192, 256];
Same name, same algorithm, different constants — share.rs has 64, 192 and 256,
benchmarks.rs does not. The bodies differ only by a local variable name
(best_dist vs best_d), which suggests one was copied from the other and they
drifted.
The consequence is that a single machine can be classified into two different tiers
depending on the code path:
| VRAM |
benchmarks.rs (lookup) |
share.rs (submission) |
| 60 GB |
48 |
64 |
| 150 GB |
128 |
192 |
| 200 GB |
128 |
256 |
The existing tesla-t4 entry carries memTierGb: 64, a value the benchmarks.rs
ladder cannot produce — so that submission can never match its own recorded tier when
the lookup path runs.
Call sites
benchmarks.rs:280 and :285 — builds the memTier query parameter for the
community benchmark lookup
share.rs:139 and :142 — writes memTierGb into the submission payload
3. Which capacities actually exist below 8 GB
The gap is not hypothetical. Cards that shipped under 8 GB and are still in wide
circulation:
| VRAM |
cards |
current tier |
error |
| 2 GB |
GT 710, GTX 1050 |
8 |
+300% |
| 3 GB |
GTX 1060 3 GB, GTX 780 |
8 |
+167% |
| 4 GB |
GTX 1050 Ti, RX 570 4 GB |
8 |
+100% |
| 6 GB |
GTX 1060 6 GB, RTX 2060, RTX 3050 6 GB |
8 |
+33% |
The GTX 1060 3 GB in particular is one of the highest-volume discrete GPUs ever
shipped, and it is currently declared at nearly three times its real capacity.
4. A separate, milder gap at 10 and 11 GB
Worth flagging while you are in here, though it is a different problem and I am not
proposing to touch it:
| VRAM |
cards |
tier |
error |
direction |
| 10 GB |
RTX 3080 10 GB |
8 |
-20% |
pessimistic |
| 11 GB |
GTX 1080 Ti, RTX 2080 Ti |
12 |
+9% |
optimistic |
10 GB sits exactly between 8 and 12; the d < best_d comparison keeps the first
match, so it lands on 8. 11 GB is strictly closer to 12.
The directions matter because this value also drives the lookup in benchmarks.rs.
A 3080 owner is served results from 8 GB cards, which undersells the card but never
promises a model that will not fit. An 11 GB owner is served results from 12 GB
cards, which can promise a model that OOMs. That is the direction that actually
hurts, but the magnitude (9%) is within what a coarse bucket is presumably meant to
absorb — unlike the +100% to +300% at the bottom of the ladder.
Suggested direction, revised
The doc comment describes these as "coarse buckets so submissions group cleanly", and
coarse grouping is clearly the intent. So the fix I would suggest is narrow: give the
bottom of the ladder a floor, and leave everything at and above 8 alone.
For the submission path only (share.rs):
const TIERS: [u32; 16] = [2, 3, 4, 6, 8, 12, 16, 24, 32, 48, 64, 80, 96, 128, 192, 256];
Properties, deliberately:
- No tier at or above 8 moves, so no previously submitted value changes.
- Only inputs below about 7 GB are affected, and no shipping card sits in the
6.5-7.0 band that shifts.
- Exact ties keep resolving downward, which understates rather than overstates — the
safe direction for a declared capacity.
- The existing test
mem_tier_rounds_to_nearest (7.5 -> 8) still passes.
I would not dedupe the two copies, despite that being the obvious reading of
section 2. nearest is the correct rule for a lookup key and the wrong one for a
declared capacity; merging them would freeze that conflation rather than resolve it.
If you want them unified, they probably want two names and two intents rather than
one function.
I have this written and tested locally against main (494 tests pass,
cargo fmt --check and cargo clippy --all-targets --all-features clean) but have
not opened a PR — happy to send it if the shape above is what you want, or to adjust
it if you would rather include 10 and 11, or handle the duplication differently.
nearest_mem_tierexists twice, with two different ladders, and neither covers GPUsbelow 8 GB.
Surfaced by the Greptile review on #815, which flagged that a 4 GB GTX 1050 Ti was
reporting
memTierGb: 8:#815 (comment)
The value was not hand-written — it is what
llmfit 1.1.3emits. Digging into whyturned up two separate problems.
1. No tier below 8 GB
Both copies start the ladder at 8, so every sub-8 GB GPU is reported as an 8 GB part:
memTierGbFor a capacity field this errs in the dangerous direction — it overstates memory. A
4 GB card is grouped with 8 GB cards in any tier-based comparison, and in
benchmarks.rsit also queries community data recorded on 8 GB hardware, which isprecisely the data it should not be matched against.
This has stayed invisible so far because every existing entry in
llmfit-core/data/community/is 8 GB or more, sonearest_mem_tierhappens to returntheir real VRAM. The 1050 Ti is the first submission below the lowest rung.
2. The two copies have diverged
llmfit-core/src/benchmarks.rs:305llmfit-core/src/share.rs:116Same name, same algorithm, different constants —
share.rshas64,192and256,benchmarks.rsdoes not. The bodies differ only by a local variable name(
best_distvsbest_d), which suggests one was copied from the other and theydrifted.
The consequence is that a single machine can be classified into two different tiers
depending on the code path:
benchmarks.rs(lookup)share.rs(submission)The existing
tesla-t4entry carriesmemTierGb: 64, a value thebenchmarks.rsladder cannot produce — so that submission can never match its own recorded tier when
the lookup path runs.
Call sites
benchmarks.rs:280and:285— builds thememTierquery parameter for thecommunity benchmark lookup
share.rs:139and:142— writesmemTierGbinto the submission payload3. Which capacities actually exist below 8 GB
The gap is not hypothetical. Cards that shipped under 8 GB and are still in wide
circulation:
The GTX 1060 3 GB in particular is one of the highest-volume discrete GPUs ever
shipped, and it is currently declared at nearly three times its real capacity.
4. A separate, milder gap at 10 and 11 GB
Worth flagging while you are in here, though it is a different problem and I am not
proposing to touch it:
10 GB sits exactly between 8 and 12; the
d < best_dcomparison keeps the firstmatch, so it lands on 8. 11 GB is strictly closer to 12.
The directions matter because this value also drives the lookup in
benchmarks.rs.A 3080 owner is served results from 8 GB cards, which undersells the card but never
promises a model that will not fit. An 11 GB owner is served results from 12 GB
cards, which can promise a model that OOMs. That is the direction that actually
hurts, but the magnitude (9%) is within what a coarse bucket is presumably meant to
absorb — unlike the +100% to +300% at the bottom of the ladder.
Suggested direction, revised
The doc comment describes these as "coarse buckets so submissions group cleanly", and
coarse grouping is clearly the intent. So the fix I would suggest is narrow: give the
bottom of the ladder a floor, and leave everything at and above 8 alone.
For the submission path only (
share.rs):Properties, deliberately:
6.5-7.0 band that shifts.
safe direction for a declared capacity.
mem_tier_rounds_to_nearest(7.5 -> 8) still passes.I would not dedupe the two copies, despite that being the obvious reading of
section 2.
nearestis the correct rule for a lookup key and the wrong one for adeclared capacity; merging them would freeze that conflation rather than resolve it.
If you want them unified, they probably want two names and two intents rather than
one function.
I have this written and tested locally against
main(494 tests pass,cargo fmt --checkandcargo clippy --all-targets --all-featuresclean) but havenot opened a PR — happy to send it if the shape above is what you want, or to adjust
it if you would rather include 10 and 11, or handle the duplication differently.