fix(gate): catch fabricated delta-style counter claims - #1010
Open
wakqasahmed wants to merge 1 commit into
Open
wakqasahmed wants to merge 1 commit into
wakqasahmed wants to merge 1 commit into
Conversation
…d/CLAUDE.md derive_counters.py --check gated absolute totals in five files but never read CHANGELOG.md, and only scanned CLAUDE.md's two 'Current Scope'/ 'Status' lines rather than its release-note history — so a fabricated delta claim like 'commands 147 -> 150' for a one-command change passed CI green in both files (issue alirezarezvani#995). Add a one-directional check: since every tracked counter only grows over the repo's life, no delta claim's right-hand (new) value can legitimately exceed today's derived total. That's enforceable without knowing which changelog entry is 'current' and without touching any of the existing, correct historical entries -- unlike an equality check, which would fail on entirely legitimate past deltas. Wires scripts/test_derive_counters.py into CI so a future regression to the new delta-claim logic is caught, not just the absolute-total gate it already ran.
Author
|
Hi @alirezarezvani — noticed this PR doesn't have a reviewer assigned yet — it's been sitting for a while, CI is green and it's mergeable. Would you (or whoever's best placed) be able to take a look, or point me to who should? Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
derive_counters.py --checkgates absolute totals in five named files, but counter prose inCHANGELOG.mdandCLAUDE.md's release-note history sits entirely outside it — CHANGELOG.md was never read by this gate at all, and CLAUDE.md was only scanned on its two**Current Scope:**/**Status:**lines, not the release-note body where every pastCounters: skills N → Mentry actually lives. A fabricated delta (e.g.commands 147 → 150for a one-command change) passes CI green in both files, exactly as reported in #995.Fix
A one-directional invariant: since every one of these counters only grows over the repo's life, no delta claim's right-hand (new) value can legitimately exceed today's derived total. That's checkable without knowing which changelog entry is "current" — an equality check would fail on every legitimate historical entry describing a past, already-superseded state, which is most of what's in the file.
extract_delta_claims/check_delta_claimsimplement this, scanning bothCHANGELOG.mdand the full text ofCLAUDE.md(not just the two summary lines) for either arrow order ("skills 358 → 359" or "402 → 441 stdlib Python tools"), unicode or ASCII arrow.Testing
scripts/test_derive_counters.py(new, stdlib unittest): the exact repro from derive_counters.py --check gates absolute totals but not delta claims, and never reads CHANGELOG.md #995 (both arrow styles) is rejected; a genuinely correct delta passes; an absolute/unchanged entry passes; a claim in a neighboring clause on the same line isn't cross-attributed to the wrong counter.Counters: skills 999 → 1234; commands 12 → 999.line toCHANGELOG.mdnow correctly fails--check; reverting it back to the clean tree passes clean (no false positives against the repo's own legitimate multi-year changelog history).test_derive_counters.pyintoci-quality-gate.yml(gate G3) so a future regression to the delta-claim logic itself gets caught, not just the original absolute-total check.Closes #995.
Implemented with AI assistance (Claude Code).