feat(scripts): flag guest-kernel bumps that change vsock semantics - #893
Conversation
A guest-kernel point-release bump spans thousands of commits, so nobody
reads it. 6.12.43 -> 6.12.94 carried a4f0b001782b ("vsock/virtio: reset
connection on receiving queue overflow"), which turned a silently
dropped packet into a fatal connection reset and broke every sizeable
`min activate` for three weeks before anyone connected the two.
kernel-bump-review.sh lists the stable-tree commits in a version range
that touch the subsystems the guest cannot boot or be driven without:
net/vmw_vsock/, the virtio_vsock headers, drivers/virtio/, the hvc0
console and virtio_blk. Subjects matching a behavioural keyword (reset,
drop, overflow, credit, backpressure, queue, fatal) are marked for a
closer read, and the mainline sha each stable patch backports is
resolved from its "commit <sha> upstream." line. --wide adds the
host-side and non-control-plane paths kept out of the default set.
Versions come either from the command line or, with --pkgs, from
`let version` in packages/virtio-linux/build.ncl at two refs of a pkgs
checkout: the two sides of a pkgs pull request, or the two commits a
`locked_commit` bump moves between.
The data comes from gregkh/linux. torvalds/linux does not carry stable
tags and answers 422 for v6.12.94, which through curl is
indistinguishable from "no matching commits", so every request here is
checked and every failure is fatal. The size of the whole range is
reported next to the per-subtree counts, so "nothing changed" can never
be confused with "nothing was asked".
Refs: #869
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 30 seconds Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
The subtree scan walks the new tag's history through a committer-date window, which equals the diff between the two tags only while the old tag is an ancestor of the new one. For a cross-series pair the tags are on divergent branches -- compare v6.6.94...v6.12.94 reports status=diverged, behind_by 17883, merge base in 2023 -- so the window keeps only the few commits of the new series that fall inside it and drops everything the intervening series carried. Neither existing guard caught this: the tags both resolve, and the date-ordering check passes because v6.6.94 genuinely predates v6.12.94. The result was a short, plausible, wrong report, which is the failure mode this tool exists to prevent. Refuse the pair instead. Co-Authored-By: macroscopeapp[bot] <macroscopeapp[bot]@users.noreply.github.com>
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
@macroscope review |
|
Manual reviews triggered for commit All prior checks · these links stay valid even if you push more commits. |
|
Just FYI for future @mentions, I'm Code review is in progress. Results will be posted as check runs when complete. |
ApprovabilityVerdict: Would Approve Adds a standalone developer script for reviewing kernel bumps with no production runtime impact. The author owns all changed files and the changes are self-contained tooling. Macroscope would have approved this PR. Enable approvability here. |
The miss this prevents
gominimal/pkgsbumped the guest kernel 6.12.43 → 6.12.94 (gominimal/pkgs#311, 2026-07-01). That range crosses v6.12.92, which carriesa4f0b001782b— backported as06fa755325df— "vsock/virtio: reset connection on receiving queue overflow". It converted a previously-silent packet drop into a fatal connection reset and broke every sizeablemin activate(#869). Nobody noticed for three weeks.Nothing exotic was needed to catch it. The commit is public, its subject says what it does, and it lives in a subtree the guest depends on completely: the entire control plane rides AF_VSOCK. A mechanical diff of
net/vmw_vsock/between the two tags surfaces it at review time — no VM, no test run, no kernel checkout.scripts/kernel-bump-review.shis that diff, plus ajust kernel-reviewrecipe.Real output for 6.12.43 → 6.12.94
Run just now, unedited (
./scripts/kernel-bump-review.sh 6.12.43 6.12.94). The acceptance test is the!-marked line six rows down:35 commits out of 9025 in the range, 8 of them flagged. That is a five-minute read, and
06fa755325df/a4f0b001782bis in it, marked. Two of its neighbours are the same story:c0e42fb0e054("cap TX credit to local buffer size") andd05bc313788f("fix potential underflow invirtio_transport_get_credit()") are the credit-accounting changes that set up the overflow the reset commit then made fatal.The report is honest about its own coarseness:
0073c41d4b99("block: add aqueue_limits_commit_update_frozenhelper") is flagged only because "queue" appears in a function name. One false positive in 35 lines is the right side of the trade — the keyword list exists to make a reviewer stop, not to be right.Subtree selection
A wide net is worse than no net: a report that is mostly noise gets skimmed, and skimming is what let this through. The default set is only what the guest cannot boot or be driven without, and each path earns its place:
net/vmw_vsock/include/linux/virtio_vsock.hinclude/uapi/linux/virtio_vsock.hdrivers/virtio/drivers/char/virtio_console.cconsole=hvc0(crates/minvmd/src/vm.rs); all guest boot andminimaldlogs leave over it. Its loss/backpressure behaviour decides whether a guest-side hang is diagnosable at all — a live question tonight.drivers/block/virtio_blk.ckrun_add_disk2/krun_add_disk3); sparse/discard behaviour is load-bearing and was measured, not assumed.Deliberately out of the default set, with reasons:
drivers/vhost/vsock.c— this is the host-side vhost-vsock driver. libkrun implements virtio-vsock in userspace and minvmd bridges it to a host unix socket (krun_add_vsock_port2,crates/minvmd/src/sock.rs), so neither side of our stack ever loads it. In this very range it contributes 7 commits and 0 flags — pure noise for us. It is available under--wideas a cross-check, since vsock semantics changes are often mirrored there. Note that the genuinely relevant vhost change (65e808a6023f, nonlinear receive SKBs) reaches the default report anyway, through the shared header.drivers/net/virtio_net.c— guest egress rides it, but it is behind thenetworking-proxyfeature and off the control plane (7 commits, 2 flagged, none of them ours). Under--wide.fs/fuse/virtio_fs.c— not used at all: the rootfs is a raw ext4 disk, not virtiofs.arch/,mm/,sched/, … — real dependencies, but any point-release bump touches thousands of lines there. Including them is how a report becomes wallpaper.--wideon the same range gives 49 commits / 10 flagged instead of 35 / 8, and adds nothing that bears on the guest. That ratio is the argument for the default.Where the data comes from, and failing loudly
The stable tree is
gregkh/linux.torvalds/linuxdoes not carry stable tags:Through
curlthat is indistinguishable from "no matching commits", which is exactly the silent-empty-result failure this tool exists to avoid having. So:comparecall. "0 commits in these subtrees" can never be confused with "0 commits looked at", and a fully-empty scan prints a loud stderr note.Range membership is a committer-date window (
since= old tag,until= new tag) over the new tag's history. A stable branch is a linear sequence of cherry-picks with monotonically rising committer dates, so the window is exact; its one imprecision is benign (a commit sharing the old tag's exact second is reported once too often, never once too few). Verified on this range: the oldest commit reported,faf332a10372, is 80 commits after v6.12.43 and 8945 before v6.12.94, and the newest vsock commit reachable from v6.12.43 (680c7d9d9197, 16:30:40Z) falls 18 seconds outside the window and is correctly excluded.That reasoning holds only while the old tag is an ancestor of the new one, which is why the same-branch check above is enforced rather than merely documented. Across series the tags are not related that way:
behind_by: 17883is the problem — the window would run overv6.12.94's history, keep the few 6.12.y commits that happen to fall between the two timestamps, and drop everything the intervening series carried. Neither pre-existing guard catches it: both tags resolve, and v6.6.94 (2025-06-19) genuinely predates v6.12.94 (2026-06-19), so the date-ordering check passes. The output would be short, plausible and wrong — this tool's own failure mode. It now refuses:The series is major.minor, so a
.0release — taggedv6.12, notv6.12.0— still compares equal to6.12.94and runs. Reviewing a genuine series bump one branch at a time is the intended workflow; acompare-based fallback would mean a second range-computation path for a case the pin has never taken.How a human uses this on a pin bump
Two trigger points, one command.
Reviewing a pkgs kernel bump (the gominimal/pkgs#311 case). From a pkgs checkout on the PR branch:
It reads
let versionfrompackages/virtio-linux/build.nclat both refs and prints what it resolved before scanning:Reviewing a
locked_commitbump here — where the new kernel actually reaches this repo. Take the two pkgs commits from the diff of.minimal/minimal.tomland pass them as refs:Versions in hand, no checkout needed:
Then read every
!line. If a subject sounds like a behaviour change on a path the guest sits on, open it (https://github.com/gregkh/linux/commit/<sha>) and read the commit message — stable backports state their reasoning and name the mainline commit, which the report already prints for you.Requirements: an authenticated
gh(public data, any token) andgitfor--pkgs. No kernel checkout, no VM, nothing to build. The full run above takes a few seconds.CODEOWNER follow-up: wiring it into CI
Not done here —
.github/workflows/is frozen and CODEOWNER-gated, and this PR does not touch it. The script is CI-ready:--fail-on-flagexits 2 when anything is flagged (verified), 0 otherwise, and 1 on any error.For a CODEOWNER who wants this enforced, the smallest useful addition is a job on pull requests that touch
.minimal/minimal.toml:Two open questions that belong to whoever owns that file, not to this PR:
actions/checkoutofgominimal/pkgsat the two pinned commits and use--pkgs, or resolve the twovirtio-linuxversions in the workflow and pass them positionally.--fail-on-flagmakes it a hard gate, which will red-build on false positives like thequeue_limitsline above. A softer wiring — run without the flag and post the output as a job summary or PR comment — matches the intent (put it in front of a reviewer) without blocking on a keyword match. My recommendation is annotate first, gate later if it proves quiet.Until then it is a local command, and
just --listadvertises it.Verified vs assumed
Verified by running it:
06fa755325df/ upstreama4f0b001782b, flagged. This is the acceptance test and it passes.--wide(49 commits / 10 flagged) and--fail-on-flag(exit code 2).--pkgsagainst a realgominimal/pkgscheckout:HEAD~200 -> HEADresolves to6.12.43 -> 6.12.94, i.e. the real bump is discoverable from the checkout with no versions typed by hand.6.12.9999→ 422, with the torvalds hint); no/extra arguments; an unknown flag; a version containing shell metacharacters (rejected before it reaches the API); a cross-series pair (6.6.94 6.12.94), rejected before any API call.06fa755325df/a4f0b001782bstill marked[!],--fail-on-flagstill exit 2,6.12 -> 6.12.94still runs,shellcheckstill clean.torvalds/linuxreturning 422 forv6.12.94— the trap that motivated the hard-coded repo.shellcheck scripts/kernel-bump-review.sh— clean.just --listshows the recipe;just kernel-review --helpruns.Assumed (and why it is safe):
linux-6.12.yis. If it were ever violated the tool would over-report, not under-report.since/untilby committer date. Consistent with every boundary check above; if it used author date the report would be noisier, not emptier.commit <sha> upstream.is the stable backport convention. Where it is absent the column shows-(11 of 35 rows here, all of them genuine — direct stable commits or older-style backports); nothing depends on it being present.bash+gh+gitonly, with no platform-specific calls.Notes for merge
justfile(one recipe appended at the end of the run & inspect section). Another branch in flight addsscripts/bulk-upload-e2e.shand editsscripts/soak-session-e2e.shplus thejustfile; expect a possiblejustfileconflict at merge. It will be an add/add in different sections and should resolve by keeping both recipes.Refs: #869, gominimal/pkgs#311
Note
Add script to flag guest-kernel bumps that change vsock semantics
gh) to scan commits in a Linux stable kernel version range across specified subtrees, flagging subjects matching behavioral keywords.--pkgsto auto-derive versions from a pkgs checkout,--widefor broader subtree coverage,--fail-on-flagto exit non-zero when flagged commits are found, and--help.kernel-reviewrecipe in justfile that invokes the script with passthrough arguments.Macroscope summarized 5bb6e65.