Skip to content

feat(scripts): flag guest-kernel bumps that change vsock semantics - #893

Merged
norrietaylor merged 2 commits into
mainfrom
feat/kernel-bump-tripwire
Jul 22, 2026
Merged

feat(scripts): flag guest-kernel bumps that change vsock semantics#893
norrietaylor merged 2 commits into
mainfrom
feat/kernel-bump-tripwire

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Jul 22, 2026

Copy link
Copy Markdown
Member

The miss this prevents

gominimal/pkgs bumped the guest kernel 6.12.43 → 6.12.94 (gominimal/pkgs#311, 2026-07-01). That range crosses v6.12.92, which carries a4f0b001782b — backported as 06fa755325df"vsock/virtio: reset connection on receiving queue overflow". It converted a previously-silent packet drop into a fatal connection reset and broke every sizeable min activate (#869). Nobody noticed for three weeks.

Nothing exotic was needed to catch it. The commit is public, its subject says what it does, and it lives in a subtree the guest depends on completely: the entire control plane rides AF_VSOCK. A mechanical diff of net/vmw_vsock/ between the two tags surfaces it at review time — no VM, no test run, no kernel checkout.

scripts/kernel-bump-review.sh is that diff, plus a just kernel-review recipe.

Real output for 6.12.43 → 6.12.94

Run just now, unedited (./scripts/kernel-bump-review.sh 6.12.43 6.12.94). The acceptance test is the !-marked line six rows down:

== kernel bump review: gregkh/linux v6.12.43 -> v6.12.94 ==

  old  v6.12.43   9becd7c25c61  2025-08-20T16:30:58Z
  new  v6.12.94   0b8f247169e4  2026-06-19T11:42:39Z
  range spans 9025 commits; scanning the core subtree set
  columns: [!] <stable-sha> <upstream-sha> <date> <subject>
  browse:  https://github.com/gregkh/linux/commit/<stable-sha>

-- net/vmw_vsock (26 commits, 7 flagged) --
  !  149205a18bcf  4157501b9a8f  2026-06-19  vsock/virtio: fix skb overhead overflow on 32-bit builds
     f3bf0f3b8d5c  c6087c5aaad6  2026-06-19  vsock/virtio: fix skb overhead accounting to preserve full buf_alloc
  !  1eca304f97a3  059b7dbd20a6  2026-06-19  vsock/virtio: fix potential unbounded skb queue
     bcb275626055  c05fa14db43e  2026-06-19  vsock/vmci: fix sk_ack_backlog leak on failed handshake
     243277ae7cf1  -             2026-06-09  vsock: keep poll shutdown state consistent
  !  06fa755325df  a4f0b001782b  2026-06-01  vsock/virtio: reset connection on receiving queue overflow
  !  47e63077605c  99e22ddf4edb  2026-06-01  vsock/vmci: fix UAF when peer resets connection during handshake
  !  29371f3cc83e  52bcb57a4e8a  2026-05-17  vsock/virtio: fix accept queue count leak on transport mismatch
     06747f52ab15  3a3e3d90cbc7  2026-05-17  vsock/virtio: fix empty payload in tap skb for non-linear buffers
     d70b04a00e69  5f344d809e01  2026-05-17  vsock/virtio: fix length and offset in tap skb for split packets
     310da27932dd  d114bfdc9b76  2026-05-17  vsock: fix buffer size clamping order
     0fee717b1d92  080f22f5d302  2026-05-17  vsock/virtio: fix MSG_PEEK ignoring skb offset when calculating bytes to copy
     15eb24645948  b31681206e3f  2026-05-14  hv_sock: fix ARM64 support
     7ebd51c3f032  -             2026-03-04  vmw_vsock: bypass false-positive Wnonnull warning with gcc-16
     e6cee5d4a122  -             2026-01-30  vsock/virtio: Fix message iterator handling on transmit path
     69c5bf306115  -             2026-01-30  vsock/virtio: Allocate nonlinear SKBs for handling large transmit buffers
     2d651c3c035e  -             2026-01-30  vsock/virtio: Rename virtio_vsock_skb_rx_put()
     74ea6184df7f  -             2026-01-30  vsock/virtio: Rename virtio_vsock_alloc_skb()
     ca82ab9fd920  -             2026-01-30  vsock/virtio: Move length check to callers of virtio_vsock_skb_rx_put()
  !  c0e42fb0e054  -             2026-01-30  vsock/virtio: cap TX credit to local buffer size
  !  d05bc313788f  -             2026-01-30  vsock/virtio: fix potential underflow in virtio_transport_get_credit()
     568e9cd8ed7c  -             2026-01-30  vsock/virtio: Coalesce only linear skb
     6762937a8b45  -             2026-01-17  vsock: Make accept()ed sockets use custom setsockopt()
     f1c170cae285  -             2025-12-01  vsock: Ignore signal/timeout on connect() if already established
     251caee792a2  f7c877e75352  2025-10-29  vsock: fix lock inversion in vsock_assign_transport()
     faf332a10372  0dab92484474  2025-08-28  vsock/virtio: Validate length in packet header before skb_put()

-- include/linux/virtio_vsock.h (5 commits, 0 flagged) --
     65e808a6023f  -             2026-01-30  vhost/vsock: Allocate nonlinear SKBs for handling large receive buffers
     2d651c3c035e  -             2026-01-30  vsock/virtio: Rename virtio_vsock_skb_rx_put()
     bdea2c39fa06  -             2026-01-30  vsock/virtio: Move SKB allocation lower-bound check to callers
     74ea6184df7f  -             2026-01-30  vsock/virtio: Rename virtio_vsock_alloc_skb()
     ca82ab9fd920  -             2026-01-30  vsock/virtio: Move length check to callers of virtio_vsock_skb_rx_put()

-- include/uapi/linux/virtio_vsock.h (0 commits, 0 flagged) --
  (no changes in range)

-- drivers/virtio (2 commits, 0 flagged) --
     1d71d509b413  -             2026-01-08  mm/balloon_compaction: convert balloon_page_delete() to balloon_page_finalize()
     5f5cb2c99ae9  -             2025-12-18  virtio_vdpa: fix misleading return in void function

-- drivers/char/virtio_console.c (1 commits, 0 flagged) --
     828b59fdf8ef  5326ab737a47  2026-01-11  virtio_console: fix order of fields cols and rows

-- drivers/block/virtio_blk.c (1 commits, 1 flagged) --
  !  0073c41d4b99  -             2025-09-09  block: add a queue_limits_commit_update_frozen helper

== summary ==

  net/vmw_vsock                        26 commits    7 flagged
  include/linux/virtio_vsock.h          5 commits    0 flagged
  include/uapi/linux/virtio_vsock.h     0 commits    0 flagged
  drivers/virtio                        2 commits    0 flagged
  drivers/char/virtio_console.c         1 commits    0 flagged
  drivers/block/virtio_blk.c            1 commits    1 flagged
  TOTAL                                35 commits    8 flagged

  8 commit(s) marked [!]: a behavioural keyword (reset|drop|overflow|credit|backpressure|queue|fatal)
  in the subject. Read those before landing the bump.

35 commits out of 9025 in the range, 8 of them flagged. That is a five-minute read, and 06fa755325df / a4f0b001782b is in it, marked. Two of its neighbours are the same story: c0e42fb0e054 ("cap TX credit to local buffer size") and d05bc313788f ("fix potential underflow in virtio_transport_get_credit()") are the credit-accounting changes that set up the overflow the reset commit then made fatal.

The report is honest about its own coarseness: 0073c41d4b99 ("block: add a queue_limits_commit_update_frozen helper") is flagged only because "queue" appears in a function name. One false positive in 35 lines is the right side of the trade — the keyword list exists to make a reviewer stop, not to be right.

Subtree selection

A wide net is worse than no net: a report that is mostly noise gets skimmed, and skimming is what let this through. The default set is only what the guest cannot boot or be driven without, and each path earns its place:

Path Why it is in
net/vmw_vsock/ Every RPC, attach/PTY stream and file upload rides AF_VSOCK; the virtio transport lives here. Tonight's root cause landed here.
include/linux/virtio_vsock.h Credit accounting, buffer/packet limits, structs shared with the host device. Five commits in ten months.
include/uapi/linux/virtio_vsock.h The wire protocol constants. Zero commits here in this range — which is itself the answer to "did the protocol change?"
drivers/virtio/ virtio core: virtqueue/vring, feature negotiation, DMA. Underneath every device below.
drivers/char/virtio_console.c The guest boots console=hvc0 (crates/minvmd/src/vm.rs); all guest boot and minimald logs leave over it. Its loss/backpressure behaviour decides whether a guest-side hang is diagnosable at all — a live question tonight.
drivers/block/virtio_blk.c The ext4 root disk and the persistent data volume are virtio-blk (krun_add_disk2/krun_add_disk3); sparse/discard behaviour is load-bearing and was measured, not assumed.

Deliberately out of the default set, with reasons:

  • drivers/vhost/vsock.c — this is the host-side vhost-vsock driver. libkrun implements virtio-vsock in userspace and minvmd bridges it to a host unix socket (krun_add_vsock_port2, crates/minvmd/src/sock.rs), so neither side of our stack ever loads it. In this very range it contributes 7 commits and 0 flags — pure noise for us. It is available under --wide as a cross-check, since vsock semantics changes are often mirrored there. Note that the genuinely relevant vhost change (65e808a6023f, nonlinear receive SKBs) reaches the default report anyway, through the shared header.
  • drivers/net/virtio_net.c — guest egress rides it, but it is behind the networking-proxy feature and off the control plane (7 commits, 2 flagged, none of them ours). Under --wide.
  • fs/fuse/virtio_fs.c — not used at all: the rootfs is a raw ext4 disk, not virtiofs.
  • arch/, mm/, sched/, … — real dependencies, but any point-release bump touches thousands of lines there. Including them is how a report becomes wallpaper.

--wide on the same range gives 49 commits / 10 flagged instead of 35 / 8, and adds nothing that bears on the guest. That ratio is the argument for the default.

Where the data comes from, and failing loudly

The stable tree is gregkh/linux. torvalds/linux does not carry stable tags:

$ gh api repos/torvalds/linux/commits/v6.12.94
{"message":"No commit found for SHA: v6.12.94", ... "status":"422"}

Through curl that is indistinguishable from "no matching commits", which is exactly the silent-empty-result failure this tool exists to avoid having. So:

  • Both tags are resolved with a checked request before anything else; a failure names the trap in the error text.
  • Every per-subtree query is checked; a failed query is fatal, never an empty section.
  • The total size of the range (9025 commits) is reported next to the per-subtree counts, from an independent compare call. "0 commits in these subtrees" can never be confused with "0 commits looked at", and a fully-empty scan prints a loud stderr note.
  • Argument order is checked by committer date — reversed versions die instead of reporting an empty range.
  • Versions are validated as digits-and-dots before being pasted into an API path.
  • Both versions must name the same stable branch. A cross-series pair is refused, not reported on (see below).

Range membership is a committer-date window (since = old tag, until = new tag) over the new tag's history. A stable branch is a linear sequence of cherry-picks with monotonically rising committer dates, so the window is exact; its one imprecision is benign (a commit sharing the old tag's exact second is reported once too often, never once too few). Verified on this range: the oldest commit reported, faf332a10372, is 80 commits after v6.12.43 and 8945 before v6.12.94, and the newest vsock commit reachable from v6.12.43 (680c7d9d9197, 16:30:40Z) falls 18 seconds outside the window and is correctly excluded.

That reasoning holds only while the old tag is an ancestor of the new one, which is why the same-branch check above is enforced rather than merely documented. Across series the tags are not related that way:

$ gh api repos/gregkh/linux/compare/v6.6.94...v6.12.94?per_page=1
{"status":"diverged","ahead_by":112743,"behind_by":17883,"base":"ffc253263a13","base_date":"2023-10-30T02:31:08Z"}

behind_by: 17883 is the problem — the window would run over v6.12.94's history, keep the few 6.12.y commits that happen to fall between the two timestamps, and drop everything the intervening series carried. Neither pre-existing guard catches it: both tags resolve, and v6.6.94 (2025-06-19) genuinely predates v6.12.94 (2026-06-19), so the date-ordering check passes. The output would be short, plausible and wrong — this tool's own failure mode. It now refuses:

$ ./scripts/kernel-bump-review.sh 6.6.94 6.12.94 ; echo $?
kernel-bump-review: v6.6.94 and v6.12.94 are on different stable branches (6.6.y vs 6.12.y).
  This tool reviews one stable branch at a time: the range is a committer-date
  window over the new tag's history, which is only the true diff when the old
  tag is an ancestor of the new one. A cross-series report would understate the
  change. Review the bump one series at a time.
1

The series is major.minor, so a .0 release — tagged v6.12, not v6.12.0 — still compares equal to 6.12.94 and runs. Reviewing a genuine series bump one branch at a time is the intended workflow; a compare-based fallback would mean a second range-computation path for a case the pin has never taken.

How a human uses this on a pin bump

Two trigger points, one command.

Reviewing a pkgs kernel bump (the gominimal/pkgs#311 case). From a pkgs checkout on the PR branch:

just kernel-review --pkgs ~/code/pkgs          # defaults to origin/main -> HEAD

It reads let version from packages/virtio-linux/build.ncl at both refs and prints what it resolved before scanning:

kernel-bump-review: packages/virtio-linux/build.ncl in ~/code/pkgs: HEAD~200 -> HEAD (6.12.43 -> 6.12.94)

Reviewing a locked_commit bump here — where the new kernel actually reaches this repo. Take the two pkgs commits from the diff of .minimal/minimal.toml and pass them as refs:

just kernel-review --pkgs ~/code/pkgs <old-locked-commit> <new-locked-commit>

Versions in hand, no checkout needed:

just kernel-review 6.12.43 6.12.94
just kernel-review --wide 6.12.43 6.12.94     # add host-side / non-control-plane paths

Then read every ! line. If a subject sounds like a behaviour change on a path the guest sits on, open it (https://github.com/gregkh/linux/commit/<sha>) and read the commit message — stable backports state their reasoning and name the mainline commit, which the report already prints for you.

Requirements: an authenticated gh (public data, any token) and git for --pkgs. No kernel checkout, no VM, nothing to build. The full run above takes a few seconds.

CODEOWNER follow-up: wiring it into CI

Not done here — .github/workflows/ is frozen and CODEOWNER-gated, and this PR does not touch it. The script is CI-ready: --fail-on-flag exits 2 when anything is flagged (verified), 0 otherwise, and 1 on any error.

For a CODEOWNER who wants this enforced, the smallest useful addition is a job on pull requests that touch .minimal/minimal.toml:

# .github/workflows/<existing-pr-workflow>.yml
  kernel-bump-review:
    if: contains(github.event.pull_request.changed_files, '.minimal/minimal.toml')  # or a paths: filter
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}   # public data; default token is enough
        run: scripts/kernel-bump-review.sh <old> <new> --fail-on-flag

Two open questions that belong to whoever owns that file, not to this PR:

  1. Where the versions come from in CI. The job has no pkgs checkout, so either add an actions/checkout of gominimal/pkgs at the two pinned commits and use --pkgs, or resolve the two virtio-linux versions in the workflow and pass them positionally.
  2. Gate or annotate. --fail-on-flag makes it a hard gate, which will red-build on false positives like the queue_limits line above. A softer wiring — run without the flag and post the output as a job summary or PR comment — matches the intent (put it in front of a reviewer) without blocking on a keyword match. My recommendation is annotate first, gate later if it proves quiet.

Until then it is a local command, and just --list advertises it.

Verified vs assumed

Verified by running it:

  • The full 6.12.43 → 6.12.94 run above, surfacing 06fa755325df / upstream a4f0b001782b, flagged. This is the acceptance test and it passes.
  • --wide (49 commits / 10 flagged) and --fail-on-flag (exit code 2).
  • --pkgs against a real gominimal/pkgs checkout: HEAD~200 -> HEAD resolves to 6.12.43 -> 6.12.94, i.e. the real bump is discoverable from the checkout with no versions typed by hand.
  • Failure modes, each loud and non-zero: reversed arguments; a nonexistent tag (6.12.9999 → 422, with the torvalds hint); no/extra arguments; an unknown flag; a version containing shell metacharacters (rejected before it reaches the API); a cross-series pair (6.6.94 6.12.94), rejected before any API call.
  • After the cross-series guard landed, the acceptance test re-run unchanged: 35 commits / 8 flagged, 06fa755325df / a4f0b001782b still marked [!], --fail-on-flag still exit 2, 6.12 -> 6.12.94 still runs, shellcheck still clean.
  • torvalds/linux returning 422 for v6.12.94 — the trap that motivated the hard-coded repo.
  • Range-boundary correctness, by comparing the oldest reported commit against both tags and checking the newest vsock commit inside v6.12.43 is excluded.
  • shellcheck scripts/kernel-bump-review.sh — clean. just --list shows the recipe; just kernel-review --help runs.

Assumed (and why it is safe):

  • Committer dates rise monotonically along a stable branch. True for linear cherry-pick queues, which is what linux-6.12.y is. If it were ever violated the tool would over-report, not under-report.
  • The GitHub commits endpoint filters since/until by committer date. Consistent with every boundary check above; if it used author date the report would be noisier, not emptier.
  • commit <sha> upstream. is the stable backport convention. Where it is absent the column shows - (11 of 35 rows here, all of them genuine — direct stable commits or older-style backports); nothing depends on it being present.
  • Not run in CI (deliberately — see above), and not run on Linux; it is bash + gh + git only, with no platform-specific calls.

Notes for merge

  • Touches justfile (one recipe appended at the end of the run & inspect section). Another branch in flight adds scripts/bulk-upload-e2e.sh and edits scripts/soak-session-e2e.sh plus the justfile; expect a possible justfile conflict at merge. It will be an add/add in different sections and should resolve by keeping both recipes.
  • No Rust, no CI, no VM, no state touched.

Refs: #869, gominimal/pkgs#311

Note

Add script to flag guest-kernel bumps that change vsock semantics

  • Adds scripts/kernel-bump-review.sh, a Bash script that uses the GitHub API (gh) to scan commits in a Linux stable kernel version range across specified subtrees, flagging subjects matching behavioral keywords.
  • Accepts --pkgs to auto-derive versions from a pkgs checkout, --wide for broader subtree coverage, --fail-on-flag to exit non-zero when flagged commits are found, and --help.
  • Adds a kernel-review recipe in justfile that invokes the script with passthrough arguments.

Macroscope summarized 5bb6e65.

A guest-kernel point-release bump spans thousands of commits, so nobody
reads it. 6.12.43 -> 6.12.94 carried a4f0b001782b ("vsock/virtio: reset
connection on receiving queue overflow"), which turned a silently
dropped packet into a fatal connection reset and broke every sizeable
`min activate` for three weeks before anyone connected the two.

kernel-bump-review.sh lists the stable-tree commits in a version range
that touch the subsystems the guest cannot boot or be driven without:
net/vmw_vsock/, the virtio_vsock headers, drivers/virtio/, the hvc0
console and virtio_blk. Subjects matching a behavioural keyword (reset,
drop, overflow, credit, backpressure, queue, fatal) are marked for a
closer read, and the mainline sha each stable patch backports is
resolved from its "commit <sha> upstream." line. --wide adds the
host-side and non-control-plane paths kept out of the default set.

Versions come either from the command line or, with --pkgs, from
`let version` in packages/virtio-linux/build.ncl at two refs of a pkgs
checkout: the two sides of a pkgs pull request, or the two commits a
`locked_commit` bump moves between.

The data comes from gregkh/linux. torvalds/linux does not carry stable
tags and answers 422 for v6.12.94, which through curl is
indistinguishable from "no matching commits", so every request here is
checked and every failure is fatal. The size of the whole range is
reported next to the per-subtree counts, so "nothing changed" can never
be confused with "nothing was asked".

Refs: #869

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 30 seconds

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f4b76651-5d4f-4597-9b25-cd087db582bb

📥 Commits

Reviewing files that changed from the base of the PR and between f03ced1 and 5bb6e65.

📒 Files selected for processing (2)
  • justfile
  • scripts/kernel-bump-review.sh

Comment @coderabbitai help to get the list of available commands.

Comment thread scripts/kernel-bump-review.sh
@norrietaylor

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

The subtree scan walks the new tag's history through a committer-date
window, which equals the diff between the two tags only while the old tag
is an ancestor of the new one. For a cross-series pair the tags are on
divergent branches -- compare v6.6.94...v6.12.94 reports status=diverged,
behind_by 17883, merge base in 2023 -- so the window keeps only the few
commits of the new series that fall inside it and drops everything the
intervening series carried.

Neither existing guard caught this: the tags both resolve, and the
date-ordering check passes because v6.6.94 genuinely predates v6.12.94.
The result was a short, plausible, wrong report, which is the failure
mode this tool exists to prevent. Refuse the pair instead.

Co-Authored-By: macroscopeapp[bot] <macroscopeapp[bot]@users.noreply.github.com>
@norrietaylor

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@norrietaylor

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@norrietaylor

Copy link
Copy Markdown
Member Author

@macroscope review

@macroscopeapp

macroscopeapp Bot commented Jul 22, 2026

Copy link
Copy Markdown

Manual reviews triggered for commit 5bb6e65:

All prior checks · these links stay valid even if you push more commits.

@macroscopeapp

macroscopeapp Bot commented Jul 22, 2026

Copy link
Copy Markdown

Just FYI for future @mentions, I'm Macroscope-App, not Macroscope.

Code review is in progress. Results will be posted as check runs when complete.

@macroscopeapp

macroscopeapp Bot commented Jul 22, 2026

Copy link
Copy Markdown

Approvability

Verdict: Would Approve

Adds a standalone developer script for reviewing kernel bumps with no production runtime impact. The author owns all changed files and the changes are self-contained tooling.

Macroscope would have approved this PR. Enable approvability here.

@norrietaylor
norrietaylor merged commit 1298a0f into main Jul 22, 2026
57 checks passed
@norrietaylor
norrietaylor deleted the feat/kernel-bump-tripwire branch July 22, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants