Skip to content

feat(minimald): support min materialize within a session - #1054

Merged
twitchyliquid64 merged 1 commit into
mainfrom
tom/check-sop
Jul 29, 2026
Merged

feat(minimald): support min materialize within a session#1054
twitchyliquid64 merged 1 commit into
mainfrom
tom/check-sop

Conversation

@twitchyliquid64

@twitchyliquid64 twitchyliquid64 commented Jul 29, 2026

Copy link
Copy Markdown
Member
  • Move materialize implementation into type in op crate
  • Implement side session-op for materialize
  • Make callable via min helper in a session

Note

Add materialize subcommand to minimald sessions to stream build artifacts

  • Adds op::Materialize in crates/op/src/materialize.rs that produces OCI image tarballs or raw files from a build graph, emitting structured MaterializeEvent progress and returning a Report with byte count and file mode.
  • Adds SideOp::new_materialize in crates/minimald/src/session_sop.rs to run materialize as a background side-op, streaming MaterializeUpdate (events, chunks, and a terminal Finished) to subscribers with backpressure; the pump stops assembly immediately when all subscribers disconnect.
  • Exposes SessionHandle::start_materialize in crates/minimald/src/session.rs so callers can trigger a materialize run and receive the update stream.
  • Adds a materialize RPC in crates/minimald/src/env.rs that resolves the --output path relative to the sandbox working directory, writes streamed bytes into the session workspace or home directory with symlink traversal protection, and sets file permissions from the reported mode.
  • Adds the materialize subcommand to crates/minimald/src/env_min_helper.sh, forwarding the sandbox PWD and arguments to the daemon.
  • Refactors mip's cmd_materialize in crates/mip/src/cmd_materialize.rs to use the new unified op::Materialize.
  • Risk: create_artifact opens destination files with O_NOFOLLOW and refuses paths outside the session workspace or home; symlink escapes at any path component are denied with an IO error.

Macroscope summarized d6dd623.

Summary by CodeRabbit

  • New Features
    • Added min materialize to materialize minimal.toml outputs as OCI image archives or extracted raw-file artifacts.
    • Supports --output and --arch overrides, streams progress/events, reports completion, and applies executable permissions when produced by the build.
    • Outputs are validated and mapped safely to the session’s sandbox locations.
  • Bug Fixes
    • Hardened handling for unknown outputs/architectures, missing raw artifacts, and attempts to write outside allowed sandbox roots.
    • Ensures deterministic raw-file selection and preserves file modes across conflicting paths.
  • Documentation
    • Updated sandbox operations reference with materialize syntax and flags.

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Adds a unified materialization operation for OCI and raw-file outputs, streams progress and artifact bytes through sessions, resolves secure sandbox destinations, and centralizes target architecture and graph top-level selection.

Changes

Materialize output pipeline

Layer / File(s) Summary
Output target and graph contracts
crates/mfile/src/lib.rs
Adds target architecture precedence, Linux target construction, graph top-level selection, and tests.
Materialization operation and OCI integration
crates/op/src/materialize.rs, crates/op/src/oci_image.rs, crates/op/src/lib.rs
Adds unified OCI/raw-file materialization, sink handling, event reporting, byte counts, mode propagation, and deterministic raw-file extraction.
Session materialization side-op
crates/minimald/src/session_sop.rs, crates/minimald/src/session.rs, crates/minimald/src/sessions.rs, crates/minimald/Cargo.toml
Streams chunks and events through a cancellable materialization side-op and exposes it through the session actor API.
Sandbox command and artifact landing
crates/minimald/src/env.rs, crates/minimald/src/env_min_helper.sh, crates/sandbox2/src/lib.rs
Adds min materialize, maps /workbench and /home paths, securely creates artifacts, and applies output modes.
CLI materialize orchestration
crates/mip/src/cmd_materialize.rs
Replaces separate OCI/raw handling with the unified operation and event logger.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Helper as min materialize
  participant Channel as SessionChannel
  participant Session as SessionHandle
  participant SideOp as Materialize side-op
  participant FS as Sandbox filesystem
  Helper->>Channel: send materialize request with cwd and args
  Channel->>Session: start_materialize(output_name, arch)
  Session->>SideOp: spawn materialization
  SideOp-->>Channel: stream chunks and events
  Channel->>FS: create and write output artifact
  Channel-->>Helper: render progress and completion
Loading

Possibly related PRs

Suggested reviewers: norrietaylor

Poem

A rabbit packs bytes in a burrow so bright,
OCI layers hop into tarballs at night.
Raw files find paths, modes neatly preserved,
Session streams flow as helpers are served.
“Materialize!” thumps softly—success in the air.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description lacks the required Summary, Testing, and Checklist sections from the template. Add ## Summary, ## Testing with evidence, and ## Checklist with docs/BREAKING CHANGE notes as applicable.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise, conventional, and accurately reflects the main session materialize change.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands.

@norrietaylor

Copy link
Copy Markdown
Member

This is going to need documentation

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (2)
crates/minimald/src/session_sop.rs (1)

797-834: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Consider a larger byte pipeline for multi-hundred-MiB images.

4 KiB BufWriter × a depth-3 channel means ~12 KiB in flight and one blocking_send + Vec allocation per 4 KiB, i.e. tens of thousands of hand-offs for a sizeable OCI tar. Bumping the buffer to 64–256 KiB (channel depth unchanged) cuts the cross-thread traffic without weakening the backpressure property this design relies on.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/minimald/src/session_sop.rs` around lines 797 - 834, Increase the
BufWriter capacity used in the Materialize sink within the spawn_blocking
assembly flow from 4 KiB to a substantially larger buffer, such as 64–256 KiB,
while keeping the byte channel depth at 3 to preserve backpressure.
crates/op/src/materialize.rs (1)

220-234: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Consider atomic writes for Sink::Path destinations.

std::fs::File::create (OCI path) and std::fs::copy (raw-file path) write directly to the final destination path. If materialization fails partway (I/O error, disk full, process killed), the destination is left holding a partial/corrupt artifact rather than either the old file or nothing — a caller polling for the output path's existence could be misled.

♻️ Suggested direction (write-to-temp + rename)
-Sink::Path(path) => {
-    let file = std::fs::File::create(&path).map_err(|e| {
-        Error::Other(anyhow!("creating {}: {e}", path.display()))
-    })?;
-    let mut w = image.write(opts, Counted::new(file), &events).await?;
-    w.flush()?;
-    w.bytes
-}
+Sink::Path(path) => {
+    let tmp = tmp_path_for(&path);
+    let file = std::fs::File::create(&tmp).map_err(|e| {
+        Error::Other(anyhow!("creating {}: {e}", tmp.display()))
+    })?;
+    let mut w = image.write(opts, Counted::new(file), &events).await?;
+    w.flush()?;
+    std::fs::rename(&tmp, &path)
+        .map_err(|e| Error::Other(anyhow!("finalizing {}: {e}", path.display())))?;
+    w.bytes
+}

A similar temp+rename pattern would apply to deliver_file's Sink::Path arm.

Also applies to: 295-320

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/op/src/materialize.rs` around lines 220 - 234, Make `Sink::Path`
writes atomic in both the materialization match block and `deliver_file`: write
the OCI or raw-file output to a temporary file in the destination directory,
then rename it to the final path only after the write or copy succeeds. Preserve
the existing error context and ensure temporary artifacts are cleaned up when
the operation fails.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/minimald/src/env.rs`:
- Around line 1013-1042: The materialization path must remove the destination
artifact when writing fails or MaterializeOutcome::summarize returns a non-zero
status. Update the write_err and status-error branches after the updates loop to
delete output before returning, while preserving existing error reporting and
successful completion behavior.
- Around line 1253-1269: Move the containment validation in the artifact path
handling before tokio::fs::create_dir_all(parent), so no directories are created
through a symlinked parent outside the session. Resolve the deepest existing
ancestor of parent and verify its canonical path remains under the canonicalized
base, while preserving the existing invalid-path and permission-denied errors;
only create missing directories after this check.
- Around line 1053-1062: Update the permission mask in the
MaterializeOutcome::Completed handling to use only 0o777 when calling
set_permissions, preventing setuid, setgid, and sticky bits from being
propagated while preserving ordinary read, write, and executable permissions.

In `@crates/minimald/src/session_sop.rs`:
- Around line 333-348: Update the chunk fan-out handling around the byte_rx
receiver to snapshot or clone the current sinks while holding inner.lock(), then
release the lock before awaiting any sink.send calls. Preserve delivery tracking
and the no-subscribers early return, while ensuring bounded subscriber
backpressure cannot block finish(), registration, or shutdown on the Inner
mutex.

In `@crates/op/src/materialize.rs`:
- Around line 179-320: Update Materialize::run so blocking materialization work,
including filesystem operations and rayon-backed image writing, runs through
Tokio’s spawn_blocking rather than on the async caller thread. Move the
synchronous logic currently in run, including extract_raw_file and deliver_file
usage, into a synchronous helper invoked by spawn_blocking, while preserving
event emission, error propagation, and the existing Report result.

---

Nitpick comments:
In `@crates/minimald/src/session_sop.rs`:
- Around line 797-834: Increase the BufWriter capacity used in the Materialize
sink within the spawn_blocking assembly flow from 4 KiB to a substantially
larger buffer, such as 64–256 KiB, while keeping the byte channel depth at 3 to
preserve backpressure.

In `@crates/op/src/materialize.rs`:
- Around line 220-234: Make `Sink::Path` writes atomic in both the
materialization match block and `deliver_file`: write the OCI or raw-file output
to a temporary file in the destination directory, then rename it to the final
path only after the write or copy succeeds. Preserve the existing error context
and ensure temporary artifacts are cleaned up when the operation fails.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 21c0006c-7df5-43c6-8fe6-d7e519e2689b

📥 Commits

Reviewing files that changed from the base of the PR and between c1d466c and 8c8ba85.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (12)
  • crates/mfile/src/lib.rs
  • crates/minimald/Cargo.toml
  • crates/minimald/src/env.rs
  • crates/minimald/src/env_min_helper.sh
  • crates/minimald/src/session.rs
  • crates/minimald/src/session_sop.rs
  • crates/minimald/src/sessions.rs
  • crates/mip/src/cmd_materialize.rs
  • crates/op/src/lib.rs
  • crates/op/src/materialize.rs
  • crates/op/src/oci_image.rs
  • crates/sandbox2/src/lib.rs

Comment thread crates/minimald/src/env.rs
Comment thread crates/minimald/src/env.rs
Comment thread crates/minimald/src/env.rs Outdated
Comment thread crates/minimald/src/session_sop.rs
Comment thread crates/op/src/materialize.rs
@twitchyliquid64

Copy link
Copy Markdown
Member Author

Added docs to sandbox operations doc. Will do a follow-up to clean that up as well.

@twitchyliquid64
twitchyliquid64 enabled auto-merge (rebase) July 29, 2026 18:16
@twitchyliquid64
twitchyliquid64 merged commit b1a4f1f into main Jul 29, 2026
29 checks passed
@twitchyliquid64
twitchyliquid64 deleted the tom/check-sop branch July 29, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants