fix(minvmd): coalesce libkrun vsock reads instead of emitting sub-skb packets - #921

Carries a single libkrun patch on the macOS source build: fill the RX descriptor and, when the socket drains below a packet worth carrying, wait briefly on EAGAIN rather than emit a vsock packet too small to carry its own receive-queue accounting.

macOS only. Linux needs the equivalent through gominimal/pkgs (pkgs#506, unmerged).

Why the shipped fix is not enough

Current unstable (159f8839) carries the fill loop from #884 and still fails. Confirmed on Tom's machine and reproduced here — including on the real repo, a 141 MB compressed upload.

The fill loop stops the stream fragmenting when libkrun outruns the writer inside one wakeup. It does nothing when the writer is slower across wakeups: each notification finds a few hundred bytes buffered, the loop drains them, hits EAGAIN, and ships a packet a fraction of the descriptor's size.

Linux charges every queued packet SKB_TRUESIZE(0) against buf_alloc regardless of payload — 576 bytes on arm64 — so a stream whose mean packet falls below that exhausts the receiver's queue budget long before its byte budget. At the 256 KiB default window the queue caps at 455 packets, and since 6.12.92 ("vsock/virtio: reset connection on receiving queue overflow") the receiver answers the 456th with a reset (ENOBUFS) instead of a drop.

Where the small packets come from — measured in the VMM

An instrumented recv_to_pkt counted, per emitted packet, the descriptor size, credit, and emitted length across 86,016 packets on the failing stack:

• descriptor size is a constant 3732 B and is never the cause of a small packet;
• the credit clamp bounds a small packet only 14% of the time, and only in the final bytes of the window;
• 86% of sub-576-byte packets come from the fill loop exiting on EAGAIN with descriptor space AND credit both free — the writer was simply behind.

That is the single lever, and it is exactly where this patch inserts a bounded poll() wait. It also explains why anything that slows the host path makes the bug vanish (a client sleep, RUST_LOG=debug, MINVMD_KRUN_LOG=debug) while tokio::task::yield_now() does not: it needs wall-clock time for bytes to accumulate, not scheduler fairness.

The guest-kernel side agrees: every reset was the skb-depth branch, never the credit branch, rx_qlen exactly 455, buf_alloc 262144.

VSOCKDBG reject A_credit=0 B_skbdepth=1 buf_used=254632 len=308
  buf_alloc=262144 rx_qlen=455 skb_overhead=262656 SKB_TRUESIZE_0=576

Measurement

libkrun the only variable — same host, same VMM binary, same installed guest, same fixture.

This patch, built in isolation (the exact single patch this PR carries, nothing else) against the current shipped unpatched dylib:

Earlier three-way run (before #923 removed the old patches from the tree), fill loop as the baseline:

The credit floor is refuted twice — once against stock, once on top of the fill loop. It should not be proposed again.

What changed since approval

#923 removed the old 0001/0002 libkrun patches from the tree (to test a client-side buffer in isolation). So this patch is now the first and only carried patch, and has been consolidated from a delta-on-the-fill-loop into a single self-contained patch (0001-vsock-coalesce-sub-skb.patch) that applies directly to the pinned libkrun commit 728df812. Verified: it git applys cleanly to stock and produces a tree byte-identical to the 16/0 build above. No behavioural change from the approved version — same emitted code.

It does not re-carry the separate signal-used-queue credit-request hang fix that #923 also removed; that is independent and worth its own PR.

On the client-side buffer (#923)

#923's 4 KiB BufWriter around the upload does not help — measured 0/6, with a socket write histogram identical to unbuffered. russh already batches at the transport, our writes average ~2.5 KB, and the fragmentation is created inside libkrun downstream of anything the client does. The fix has to live in the VMM.

Cost

poll() returns as soon as more data lands, so a fast writer pays nothing. A slow one costs at most COALESCE_ROUNDS × 1 ms = 4 ms per descriptor, and data is always delivered rather than held, so no stream can stall on this.

Upstreaming

Coalescing changes latency for every unix-backed vsock port, not just bulk uploads. Tom's read is that this "basically implements nagle's algorithm", which would break any consumer expecting send() to be 1:1 with recv() — not us (our protocol is stream-oriented), but worth putting behind an option if we upstream to containers/libkrun. Carrying it locally needs no such decision.

Companion PRs

• revert(minimald): remove the 8 MiB guest vsock receive window #925 — removes the 8 MiB guest window (fix(minimald): raise the guest vsock receive window to 8 MiB #885/revert(minimald): restore the 8 MiB guest vsock receive window #922) as redundant once this lands. The local stack does not carry that window regardless (installed build is 159f8839, pre-revert(minimald): restore the 8 MiB guest vsock receive window #922; measured buf_alloc = 256 KiB), so all the failures above are against the true default baseline.

macOS CI exercises this patch — #910 folded vendor/libkrun/patches/** into the setup-libkrun-macos cache key, so adding or changing a patch invalidates the cache and forces a rebuild.

Refs: #869