microvm-rootfs: build from Alpine, drop the glibc closure - #534
Conversation
The image was assembled by snapshotting this package's own runtime closure out of the build sandbox (base + git + iproute2 + e2fsprogs + util-linux). Those packages carry glibc, and that closure was the bulk of the ~180 MB image — enough to show up as download lag on the install path and as slower VM boots. Assemble it from pinned upstream Alpine artifacts instead: the minirootfs tarball (musl libc + busybox + baselayout) overlaid with a sha256-pinned closure of 33 .apk files, resolved against the v3.24 main APKINDEX. Six of those are the tools busybox does not cover — bash, git, e2fsprogs (mkfs.ext4), iproute2-minimal (ip), util-linux-misc (nsenter) and fstrim — and the rest is their shared-library closure. runtime_deps is now empty, so no package from this repo, and no glibc, reaches the image; base and e2fsprogs stay as build-only deps for build.sh and mke2fs. The staged tree is 33.7 MiB on arm64 and 28.1 MiB on x86_64, with ~30 MiB of blocks used in the packed image on both. Two side effects of dropping our packages: the libblkid/libuuid soname canonicalization is gone (Alpine has a single provider for each rather than two forks), and so is the /bin/sh fallback (the minirootfs ships it). build.sh now asserts each guest tool is present and that the only ELF interpreter in the tree is musl's, so a leaked glibc dep or an upstream path move fails the build rather than the guest boot. Closes #531 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 11 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe microVM rootfs build now assembles the guest filesystem from pinned Alpine minirootfs and APK artifacts instead of repository-built runtime dependencies. The build script overlays the APK closure, validates required musl-based contents, and produces the existing ext4 rootfs output. ChangesAlpine rootfs assembly
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant BuildSpec
participant AlpineArtifacts
participant BuildScript
participant RootfsImage
BuildSpec->>AlpineArtifacts: fetch pinned minirootfs and APK closure
BuildScript->>AlpineArtifacts: unpack and extract payloads
BuildScript->>RootfsImage: build ext4 rootfs
RootfsImage-->>BuildScript: report image and staged-tree sizes
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/microvm-rootfs/build.ncl (1)
102-187: 🗄️ Data Integrity & Integration | 🔵 TrivialMirror these Alpine APKs or expect branch drift.
packages/microvm-rootfs/build.ncl:102-187usesv3.24/main/...URLs, which are current-branch artifacts rather than an archive. When Alpine rebuilds a package, older filenames can disappear and later fetches can 404; a local mirror/cache or documented re-resolution path would make this less brittle.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/microvm-rootfs/build.ncl` around lines 102 - 187, Make the Alpine APK inputs in the shared-library closure reproducible without relying on mutable v3.24/main URLs. Update the apk declarations around the listed packages to use a local mirror/cache or an archived, immutable repository source, and document the re-resolution process required when the Alpine version changes. Preserve the package versions and checksums as the validated closure.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/microvm-rootfs/build.sh`:
- Around line 55-63: Update the archive merge command in the APK loop to avoid
applying the scratch directory’s own metadata to $STAGE, while still extracting
all payload contents and preserving existing destination-directory metadata. Use
tar’s no-overwrite-directory behavior for this extraction, keeping the
surrounding cleanup and per-APK merge flow unchanged.
---
Nitpick comments:
In `@packages/microvm-rootfs/build.ncl`:
- Around line 102-187: Make the Alpine APK inputs in the shared-library closure
reproducible without relying on mutable v3.24/main URLs. Update the apk
declarations around the listed packages to use a local mirror/cache or an
archived, immutable repository source, and document the re-resolution process
required when the Alpine version changes. Preserve the package versions and
checksums as the validated closure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: fc2fac4d-e2dd-4200-8333-37030e2e61c2
📒 Files selected for processing (2)
packages/microvm-rootfs/build.nclpackages/microvm-rootfs/build.sh
Two CI checkers failed on the previous commit: - build script disallowed-patterns: build.sh:62 used `tar -xf` for the .apk payload merge; the repo requires `tar -xof`. Every entry in all 33 .apk archives is uid 0 / gid 0, so --no-same-owner is inert on the payload, and the minirootfs extraction is untouched, so /etc/shadow keeps root:shadow. Rebuilt both arches: the image is byte-for-byte what it was before. - fmt: build.ncl reformatted with the repo's Nickel formatter. Whitespace only.
|
@coderabbitai review |
✅ Action performedReview finished.
|
A branch's /main/ directory on dl-cdn.alpinelinux.org only ever holds each package's current build, so the moment Alpine rebuilds one of these (bash-5.3.9-r1 -> -r2) the pinned filename 404s and this package stops building. 245 packages in this repo already point at gs://minimal-staging-archives for the same reason. Mirror the 68 objects (33 .apk files x 2 arches, plus both minirootfs tarballs) under alpine/<branch>/, preserving the upstream path layout so every object's mirror key is also its provenance. The bump procedure in the header comment now names the mirroring step.
|
@coderabbitai review |
✅ Action performedReview finished.
|
The guest minimald points DNS at the switch's server by writing
/run/resolv.conf and bind-mounting it over /etc/resolv.conf
(minimald guest.rs, install_resolv_conf). A bind only rewrites the
mount tree, so it works on the read-only root — but the target path
has to exist first.
The old sandbox-snapshot image inherited a resolv.conf from the build
root. The Alpine minirootfs ships none, so on the assembled image the
mount fails:
WARN installing /etc/resolv.conf for guest egress (DNS may fail)
error=No such file or directory (os error 2)
leaving the guest with no resolver at all — anything that resolves a
name, including the in-guest `pkgs` clone during session mint, dies
with "Could not resolve host".
Stage the path as an empty file; the contents are written at runtime.
Verified on aarch64/KVM: booting the image built from this commit
emits READY and brings egress up with the warning gone, matching the
pre-Alpine image. Booting the parent commit's image reproduces it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
) The pin was 9 days and 62 commits stale (c854d6b1, 2026-07-20). The motivating change is gominimal/pkgs#534, which rebuilds microvm-rootfs from Alpine and drops the glibc closure — but the pin is linear, so this necessarily carries everything before it too. Materially in range: - microvm-rootfs from Alpine (#534) — 186 MB -> 45 MB - two libkrun vsock fixes: RX descriptor fill (#506) and packet-count backpressure (#512), both on the path minvmd depends on - rust 1.97.1 (#502) and glibc 2.44 (#529) in the build stack The rest is routine package version bumps. Pinned at the branch tip rather than at #534's commit: pinning mid-history buys nothing here, and the three commits after it are a bottom bump, a graphviz bump, and a license-metadata fix. Verified on this host: `mip materialize --arch aarch64 minvmd-rootfs` resolves from cache and yields a 47,212,544-byte ext4 image whose only interpreter is /lib/ld-musl-aarch64.so.1 — no glibc. The same image has already been booted and driven through the session e2e (cold activate 4296 ms, warm ls 16 ms, sandbox proof 8030 ms), and A/B'd for cold-boot latency against the outgoing rootfs: median 129 ms vs 154 ms, n=10 each, non-overlapping. This changes the guest for macOS as well, which already ships the payload; the VM lanes are the gate. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Closes #531.
What changed
microvm-rootfsbuilt its image by snapshotting its own runtime closure out of the build sandbox —base+git+iproute2+e2fsprogs+util-linux, all glibc-linked. That closure was the weight.It now assembles from pinned upstream Alpine artifacts:
alpine-minirootfs-3.24.1tarball (musl libc, busybox,alpine-baselayout's/etcand mountpoints), sha256-pinned per arch..apkfiles, resolved as a closure against thev3.24mainAPKINDEX. Six are the tools busybox does not cover —bash(the in-guest minimald chroots in and runs/bin/bash),git(session bring-up shells out to it),e2fsprogs(mkfs.ext4for the per-VM/dev/vdb),iproute2-minimal(ip),util-linux-misc(nsenter) andfstrim. The other 27 are their shared-library closure.runtime_deps = []. Runtime deps get hardlinked into the build sandbox, which is what would drag glibc back in.baseande2fsprogsstay as build-only deps —bash/coreutils/tar/findutilsforbuild.sh,mke2fsto pack the image.The
.apkfiles are unpacked withtarrather thanapk: the guest root is read-only and nothing in the closure has an install script to run (the busybox applet symlinks and CA bundle that do need one are already applied in the minirootfs).Two chunks of
build.shdeleted themselves as a result:libblkid/libuuidsoname canonicalization — that existed because oure2fsprogsandutil-linuxeach shipped a fork with the same soname. Alpine has one provider for each./bin/shfallback — the minirootfs ships it.New in its place:
build.shasserts every guest tool is present at its expected path and that the only ELF interpreter in the tree is musl's, so a leaked glibc dep or an upstream path move fails the build instead of the guest boot.Size
Against ~180 MB before. The gap between allocated and used is the pre-existing
tree + 10% + 8 MiBheadroom formula, left untouched — it is empty space that compresses away in the distributed artifact, and tightening it risksmke2fs -dfailing to fit the tree.Only real prune left is
usr/lib/bash(~2.7 MB of loadable builtins reachable only viaenable -f). Alpine splits headers, static libs, man pages and docs into-dev/-docsubpackages that are not in the closure at all, so the old prune list had nothing else to remove.How this was verified
CI is green —
minimal-checkandminimal buildboth pass on the current head. Everything below was done in addition, outside the sandbox against the real artifacts, on Linux with GNU tar 1.35 + e2fsprogs 1.47.4 (the version this repo pins):.apkdigests and both minirootfs digests re-derived from freshly downloaded artifacts and diffed against the file — no transcription drift.build.shrun end to end for both arches. Both produce a byte-identical image across two runs separated in time, and bothe2fsck -fnclean./lib/ld-musl-<arch>.so.1. Zerolibc.so.6orGLIBC_2.*references.chrooted into the staged tree — with no host libraries reachable — and ran each one:bash 5.3.9 (aarch64-alpine-linux-musl),git 2.54.0(realgit init+ commit),iproute2-v7.0.0,nsenter 2.42.1,mke2fs 1.47.4,fstrim 2.42.1,/bin/sh. Same on x86_64.tar -xof: required byminimal-check. Every entry in all 33 archives is uid 0 / gid 0, so--no-same-owneris inert on the payload; the minirootfs extraction is untouched, so/etc/shadowkeepsroot:shadow. The image is byte-for-byte what it was before the flag.Risk
Per the issue: land it if integ tests are clean, defer if not. Deliberate behavioural deltas to watch, none of which a unit check would catch:
execs inside the chroot must be musl-linked or static. This is the reason the issue calls for minimald's static musl build if it ever moves into the rootfs; it stays in the initramfs here, so nothing was added for it./etc/resolv.conf. The minirootfs does not ship one and the root is read-only, so guest DNS has to come from the initramfs (bind-mount or equivalent). Whether the old sandbox-snapshot image happened to carry one is not something I could determine from this repo.coreutils/grep/awk/sedare now busybox applets, not GNU.ip,nsenterandfstrimare the real util-linux/iproute2 binaries precisely because they are load-bearing, but anything shelling out with GNU-only flags elsewhere would surface here.Source mirroring
The pinned artifacts are fetched from
gs://minimal-staging-archives/alpine/<branch>/..., notdl-cdn.alpinelinux.org. CodeRabbit flagged the reason and it is right: a branch's/main/<arch>/directory holds only each package's current build, so the moment Alpine rebuilds one of these (bash-5.3.9-r1->-r2) the pinned filename 404s and this package stops building. 245 packages in this repo already fetch from the bucket for the same reason (see #511 for the sqlite case).All 68 objects (33
.apkfiles x 2 arches, plus both minirootfs tarballs) are mirrored underalpine/v3.24/, preserving the upstream path layout one-to-one — so every object's mirror key is also its provenance, and an Alpine bump only changes the branch segment. Each was re-downloaded fromdl-cdn, checked against the sha256 inbuild.nclbefore upload, and then re-verified by reading it back out of the bucket: 68/68 match.Follow-up
Bumping Alpine means re-resolving the whole closure against the new branch's APKINDEX, not editing entries one at a time. Both
build.nclcomments say so, but a small resolver script in the repo would make that a one-command job — happy to add it here or separately.🤖 Generated with Claude Code
Summary by CodeRabbit
Summary by CodeRabbit