Skip to content

chore: add tracing instrumentation to execution-path async fns - #221

Merged
twitchyliquid64 merged 1 commit into
mainfrom
chore/217-otel-instrumentation
May 24, 2026
Merged

chore: add tracing instrumentation to execution-path async fns#221
twitchyliquid64 merged 1 commit into
mainfrom
chore/217-otel-instrumentation

Conversation

@norrietaylor

Copy link
Copy Markdown
Member

Summary — Adds #[tracing::instrument] to high-value async Result-returning functions on the execution path and propagates span context into spawned gRPC streaming tasks.

Instrumented

  • crates/remote-client/src/lib.rs
    • Client::make_env#[tracing::instrument(skip_all)]
    • Client::exec#[tracing::instrument(skip_all)]
    • tokio::spawn graph-streaming task in make_env → wrapped with .instrument(tracing::Span::current())
    • tokio::spawn graph-streaming task in build → wrapped with .instrument(tracing::Span::current())
    • tokio::spawn check-response streaming task in check → wrapped with .instrument(tracing::Span::current())
  • crates/check/src/lib.rs
    • run_checks#[tracing::instrument(skip_all, err)]
    • package_check_futures#[tracing::instrument(skip_all, err)]
    • profile_check_futures#[tracing::instrument(skip_all, err)]
    • stack_check_futures#[tracing::instrument(skip_all, err)]
    • check_package#[tracing::instrument(skip_all, fields(pkg = %pkg), err)]
    • tokio::task::spawn_blocking in check_package → span propagated by capturing tracing::Span::current() and span.enter() inside the closure (closures can't use .instrument())
  • crates/rcache/src/remote.rs
    • RemoteCache::new_over_https#[tracing::instrument(skip_all, err)]
    • RemoteCache::new_with_gcs_bucket#[tracing::instrument(skip_all, fields(bucket_id = %bucket_id), err)]
    • RemoteCache::into_writer#[tracing::instrument(skip_all, err)]
    • RemoteCache::materialize#[tracing::instrument(skip_all, fields(span_name = %span_name), err)]
  • crates/rcache/src/remote_writer.rs
    • RemoteCacheWriter::upload#[tracing::instrument(skip_all, err)]
    • RemoteCacheWriter::finish_uploads#[tracing::instrument(skip_all, err)]
  • crates/op/src/oci_image.rs
    • OciImageCreate::run (the public Runnable entry point covering the whole OCI path) → #[tracing::instrument(skip_all, err)]
    • create_base_layer#[tracing::instrument(err)]
    • create_layer_from_cache#[tracing::instrument(skip_all, fields(package_name = %package_name), err)]

Also added tracing.workspace = true to crates/remote-client/Cargo.toml and crates/check/Cargo.toml (the only two target crates that did not already depend on tracing).

Skipped

  • remote-client make_env / exec use #[instrument(skip_all)] without err: the crate's remote_client::Error enum implements Debug but not Display, and the err directive requires Display (confirmed by a failing compile). Dropped err on these two per the issue's guidance.
  • commit_index free fn (named in the issue for remote_writer.rs): does not exist in the codebase. The index-commit logic lives inline inside RemoteCacheWriter::finish_uploads, which is instrumented instead.
  • build_oci_image public wrapper (named in the issue for oci_image.rs): does not exist. The public OCI entry point is OciImageCreate::run (impl of Runnable::run), which is instrumented and covers the whole OCI path.

Verification

All run in a Linux container (rust:1-bookworm); the target crates transitively depend on sandbox2, which is Linux-only (hakoniwa/nix), so it cannot build on the macOS dev host.

  • cargo fmt --all --check — passed
  • cargo build -p remote-client -p check -p rcache -p op — passed
  • cargo build --workspace — passed
  • cargo clippy -p remote-client -p check -p rcache -p op --all-targets -- -D warnings — passed
  • cargo test --workspace — passed

Closes #217

🤖 Generated with Claude Code

Adds #[tracing::instrument] to high-value async Result-returning
functions across remote-client, check, rcache, and op, and propagates
span context into spawned gRPC streaming tasks. Addresses #217.

Closes #217

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented May 22, 2026

Copy link
Copy Markdown

Warning

Rate limit exceeded

@norrietaylor has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 11 minutes and 58 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a582c400-6ea4-47ba-b7d5-a3da01ea871b

📥 Commits

Reviewing files that changed from the base of the PR and between 304c886 and 557d750.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (7)
  • crates/check/Cargo.toml
  • crates/check/src/lib.rs
  • crates/op/src/oci_image.rs
  • crates/rcache/src/remote.rs
  • crates/rcache/src/remote_writer.rs
  • crates/remote-client/Cargo.toml
  • crates/remote-client/src/lib.rs

Comment @coderabbitai help to get the list of available commands and usage tips.

@norrietaylor

Copy link
Copy Markdown
Member Author

@codereabbit review

@norrietaylor
norrietaylor requested review from bryan-minimal and twitchyliquid64 and removed request for twitchyliquid64 May 22, 2026 16:28
@norrietaylor

Copy link
Copy Markdown
Member Author

@codereabbit review

@twitchyliquid64
twitchyliquid64 merged commit e43d12f into main May 24, 2026
8 checks passed
@twitchyliquid64
twitchyliquid64 deleted the chore/217-otel-instrumentation branch May 24, 2026 19:07
norrietaylor added a commit that referenced this pull request Jun 4, 2026
The host awaited the boot READY marker on vsock 9799, but no guest emits
on that port: the guest rootfs manifest documents 7350
(etc/minvmd/manifest: vsock_port_ready=7350) and the min-ctl reference
impl boots to a TCP connection on 127.0.0.1:7350. With 9799 the marker
never arrived. Align the host to 7350.

Necessary but not sufficient for boot: krun_start_enter still fails
EINVAL because the external-kernel path (krun_set_kernel) is paired with
krun_set_root (virtiofs directory) instead of a root disk
(krun_add_disk / set_root_disk). See PR comment.

Refs: #221

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
norrietaylor added a commit that referenced this pull request Jun 4, 2026
The host awaited the boot READY marker on vsock 9799, but no guest emits
on that port: the guest rootfs manifest documents 7350
(etc/minvmd/manifest: vsock_port_ready=7350). With 9799 the marker never
arrived. Align the host to 7350.

Necessary but not sufficient for boot: krun_start_enter still fails
EINVAL because the external-kernel path (krun_set_kernel) is paired with
krun_set_root (virtiofs directory) instead of a root disk
(krun_add_disk / set_root_disk). See PR comment.

Refs: #221

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
norrietaylor added a commit that referenced this pull request Jun 4, 2026
The host awaited the boot READY marker on vsock 9799, but no guest emits
on that port: the guest rootfs manifest documents 7350
(etc/minvmd/manifest: vsock_port_ready=7350). With 9799 the marker never
arrived. Align the host to 7350.

Necessary but not sufficient: krun_start_enter still returns EINVAL; root
cause under investigation (see PR comment) — not a kernel-format or
rootfs-format issue.

Refs: #221

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
norrietaylor added a commit that referenced this pull request Jun 4, 2026
#326) (#344)

* feat(minvmd): add boot command and VMM child subcommand

Implements R2.3 and R2.4 from the minvmd spec:

- cmd/mod.rs: declares the cmd module, VSOCK_MARKER_PORT (9799) and
  MARKER_SOCK_ENV constants shared between parent and child.

- cmd/boot.rs: the `minvmd boot [--foreground]` subcommand.
  On macOS: validates MINVMD_KERNEL_PATH and MINVMD_ROOTFS_PATH,
  creates a UNIX socket listener for the READY marker, fork-execs
  `minvmd __krun-vmm` with MINVMD_MARKER_SOCK set to the socket
  path, writes the child PID to vmm.pid, then waits up to 5 s for
  the guest to connect and write READY\n (R2.4). On success prints
  vm-up. With --foreground, blocks until the VMM child exits.
  On Linux: bails immediately (no-op stub).

- cmd/vmm_child.rs: the hidden `minvmd __krun-vmm` subcommand.
  On macOS: creates a libkrun context, applies VmConfig (kernel,
  rootfs, 2 vcpus, 512 MiB), registers VSOCK_MARKER_PORT pointing
  to the host UNIX socket, then calls krun_start_enter (R2.3).
  On Linux: bails immediately (no-op stub).

- main.rs: wires Boot and KrunVmm subcommands to the CLI.
- lib.rs: exports pub mod cmd.
- tests/boot_e2e.rs: READY-marker round-trip E2E test (gated on
  MINVMD_E2E=1 and #[ignore], macOS only) (R2.4).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(minvmd): reap child and clean up pid/socket on boot failure

On the error and timeout paths in boot.rs, terminate and wait the VMM
child and remove both vmm.pid and the marker socket before returning,
so a failed boot does not leave stale state behind.

In boot_e2e.rs, set XDG_STATE_HOME to an isolated tempdir for the
spawned child so the E2E test does not clobber the developer's real
state directory.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(minvmd): serialize boot E2E test and add serial_test dep

Add #[serial] to boot_e2e_ready_marker_round_trip so it cannot race
other stateful E2E tests that mutate XDG_STATE_HOME. The test already
isolates its state directory via XDG_STATE_HOME=<tempdir>, so no
additional isolation is needed; serialization ensures only one such
test runs at a time.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(minvmd): harden marker socket path with random nonce

Use PID + 4 random bytes from /dev/urandom for the READY-marker socket
path instead of PID alone, making the path unpredictable and closing
the TOCTOU race a local attacker could exploit to inject a spoofed
READY.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(minvmd): align READY marker vsock port to canonical 7350

The host awaited the boot READY marker on vsock 9799, but no guest emits
on that port: the guest rootfs manifest documents 7350
(etc/minvmd/manifest: vsock_port_ready=7350). With 9799 the marker never
arrived. Align the host to 7350.

Necessary but not sufficient: krun_start_enter still returns EINVAL; root
cause under investigation (see PR comment) — not a kernel-format or
rootfs-format issue.

Refs: #221

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minvmd): load aarch64 Image.gz with PE_GZ format

The aarch64 libkrun loader implements only RAW and PE_GZ; IMAGE_GZ (=4)
is x86_64-only and returns KernelFormatUnsupported, so krun_set_kernel
failed before the VM could boot. Select KRUN_KERNEL_FORMAT_PE_GZ (=2)
for the aarch64 Image.gz (the loader scans for the gzip magic and
decompresses) and add the constant to the FFI surface.

Verified against /opt/homebrew/include/libkrun.h (libkrun 1.18.1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(minvmd): set guest workload, capture console, raise boot resources

The VMM child configured a kernel + rootfs but never set a workload, so
libkrun's /init.krun (pid-1) fell back to /bin/sh and never emitted the
READY marker. Set the guest workload via krun_set_exec (default
/sbin/minvmd-stub-init, MINVMD_EXEC overrides) with an explicit minimal
envp — passing None would inherit the full host env and can overflow the
~2 KiB aarch64 kernel cmdline.

Add opt-in early-boot console capture (MINVMD_BOOT_LOG) for diagnosing a
stuck boot, and raise the VM from 512 MiB to 2 vCPU / 1024 MiB for cheap
headroom under Hypervisor.framework.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(minvmd): add VM image build scripts with guest READY writer

Stage the non-EFI VM image production on main:

- fetch-alpine.sh: pinned, sha256-verified Alpine 3.21.7 minirootfs.
- build-rootfs.sh: overlay socat + the /sbin/minvmd-stub-init workload
  and the guest manifest onto the rootfs directory (consumed by
  krun_set_root as virtio-fs; no disk image).
- fetch-virtio-kernel.sh: pull the prebuilt virtio-linux vmlinuz
  (Image.gz) from the public minimal build cache (carried from #341).

The stub's READY writer connects OUT to the host (CID 2, port 7350)
rather than listening: the host registers the marker with the plain
krun_add_vsock_port (== listen=false), so the direction is guest->host.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(minvmd): boot E2E on the real Alpine rootfs + virtio kernel

Stage the guest rootfs via fetch-alpine.sh + build-rootfs.sh (was a
vanilla extract with no READY writer), capture the guest console as an
artifact for debugging, and scope the workflow to scripts/**. The boot
E2E stays non-gating (continue-on-error) until first-green on the
self-hosted runner, which also validates the virtio-linux kernel config.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(minvmd): correct kernel format, init model, and marker direction

Align the spec with libkrun's verified behaviour: aarch64 loads Image.gz
via PE_GZ (not IMAGE_GZ); libkrun's /init.krun is pid-1 and execs the
krun_set_exec workload (no init system in the rootfs); the kernel cmdline
stays unset; boot uses 2 vCPU / 1024 MiB; and the READY marker is
guest-initiated (krun_add_vsock_port, listen=false), the opposite of the
R3 ssh.sock bridge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minvmd): make image scripts run under macOS bash 3.2

The self-hosted runner's `/usr/bin/env bash` is the macOS system bash
3.2, which lacks associative arrays; `declare -A` made `[aarch64]=` parse
as an unbound arithmetic index under `set -u`, failing build-rootfs.sh
with "aarch64: unbound variable". Replace the per-arch sha256 arrays with
plain vars + a case lookup. Verified under /bin/bash 3.2.57.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minvmd): add socat's runtime closure to the guest rootfs

The READY-writer stub runs socat, which dynamically links libreadline.so.8
(needs libncursesw.so.6) — neither is in the Alpine minirootfs, so socat
aborted with "Error loading shared library libreadline.so.8" and never
wrote READY. Overlay the readline and libncursesw apks (sha256-pinned)
alongside socat via a fetch_apk helper. (libssl/libcrypto/libc are already
in the base.) NOTE: the library ships in `libncursesw`, not the
payload-less `ncurses-libs` metapackage.

Verified locally on Apple Silicon: a codesigned `minvmd boot` against the
prebuilt virtio-linux kernel boots Alpine and prints `vm-up`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(minvmd): run the boot E2E binary directly so codesign survives

`cargo test` relinks target/debug/minvmd under the test profile, discarding
the hypervisor entitlement applied by codesign — so krun_start_enter failed
with EINVAL even though a prior step signed the binary. Build with
`--no-run`, codesign, then execute the prebuilt boot_e2e test binary
directly (no further cargo invocation), which preserves the signature.
Stage the real rootfs via build-rootfs.sh and upload the guest console log.

Verified locally: build --no-run -> codesign -> run test binary == 1 passed,
binary still signed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(minvmd): make the boot E2E a required gate

The READY round-trip is green on the self-hosted Apple Silicon runner
(test result: ok. 1 passed). Drop continue-on-error so a boot regression
fails the workflow. This is the acceptance gate for #311 / #326.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(minvmd): expand R2.2 for the rootfs overlay and socat closure

Name build-rootfs.sh (overlays /sbin/minvmd-stub-init + socat + the
readline/libncursesw closure), not just fetch-alpine.sh, and state the
result is a virtio-fs directory (no disk image).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minvmd): propagate /dev/urandom read failure for the marker nonce

A failed open/read of /dev/urandom silently fell back to a zero nonce,
making the marker socket path predictable and defeating the TOCTOU
hardening the comment promises. Propagate the error with context instead
of discarding it via `let _ =` (which the repo's standards ban).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: gominimal-aw-bot[bot] <281738952+gominimal-aw-bot[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

agent:otel:rust 15 instrumentation gaps in 5 files (advisory)

2 participants