Skip to content

feat(minimal): upload project files to session on activate - #748

Merged
0chroma merged 10 commits into
mainfrom
0chroma/feat-file-upload-263
Jul 15, 2026
Merged

feat(minimal): upload project files to session on activate#748
0chroma merged 10 commits into
mainfrom
0chroma/feat-file-upload-263

Conversation

@0chroma

@0chroma 0chroma commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Summary

After CreateSession succeeds, the client now tar+zstd's the project directory and streams it to the daemon's WorkspaceFilesTarZst subsystem, which unpacks it into the session workspace. This is the file-sync-INTO direction from #767.

Background

The daemon-side receiver already existed (rpc.rs:404WorkspaceFilesTarZst subsystem), but the minimal CLI had no client-side sender. cmd_activate sent only SessionConfig + WireContribution over RPC — no file payload. The daemon read project_path directly on the host (only works when co-located).

Changes

  • client.rs — new upload_workspace_files method: opens a channel, sets MINIMAL_SESSION_ID, requests the WorkspaceFilesTarZst subsystem, streams a zstd-compressed tarball, and drains the response
  • file_upload.rs — new module: tar_directory recursively walks a directory and builds an in-memory tar archive preserving relative paths, symlinks, and empty directories
  • lib.rscmd_activate calls upload_workspace_files after session creation, before the attach prompt
  • Cargo.toml — added async-tar, async-compression, constcat deps

Open questions

  • Ignore/exclude files: The upload currently sends the entire project directory including .git/, node_modules/, build artifacts, etc. Should we support .gitignore parsing, a hardcoded default exclude list (.git, node_modules, target, __pycache__, .venv), a .minimalignore file, or some combination? Tracked as an open question on feature: File sync INTO a Session #767.

Closes #767 (file sync INTO a session).

Summary by CodeRabbit

  • New Features
    • When activating, the full project workspace is now uploaded to the daemon as a zstd-compressed tar archive.
    • Upload preserves directories (including empty ones), regular files, and symbolic links, while excluding common heavy defaults like .git, target, and node_modules.
  • Bug Fixes
    • Workspace upload failures now report clearer error details.
    • Unreadable or permission-restricted paths are skipped to prevent activation from failing.
  • Tests
    • Added an integration test that activates a project and verifies uploaded contents (including nested files) via SFTP.

After CreateSession succeeds, the client now tar+zstd's the project
directory and streams it to the daemon's WorkspaceFilesTarZst subsystem,
which unpacks it into the session workspace. This is the file-sync-INTO
direction from #263.

The daemon-side receiver already existed (rpc.rs:404); this adds the
client-side sender. The upload happens after session creation and
before the attach prompt.
@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@0chroma, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 46 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Free

Run ID: 19b5f8d5-144f-463f-b871-ada96ea80931

📥 Commits

Reviewing files that changed from the base of the PR and between 7fb34c8 and a8c4f13.

📒 Files selected for processing (1)
  • crates/minimal/src/file_upload.rs
📝 Walkthrough

Walkthrough

Changes

The activation flow now streams the project workspace as a zstd-compressed tar archive to the daemon and verifies the resulting session files through integration testing.

Workspace upload flow

Layer / File(s) Summary
Streaming workspace archive
crates/minimal/Cargo.toml, crates/minimal/src/file_upload.rs, crates/minimal/src/lib.rs
Adds archive dependencies and streams directories, symlinks, and regular files into a zstd-compressed tar archive, with exclusions and unit tests for supported and skipped entries.
SSH workspace upload
crates/minimal/src/client.rs
Adds the client method that sets the session ID, requests the WorkspaceFilesTarZst subsystem, and streams the archive to the daemon.
Activation integration
crates/minimal/src/lib.rs, crates/minimal/tests/cli.rs, scripts/session-e2e.sh
Uploads the activated project directory after session creation, validates uploaded files through SFTP, and prepares a minimal VM E2E workspace.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant Client
  participant stream_tar_zstd
  participant minimald
  participant SFTP
  CLI->>Client: activate project
  Client->>minimald: request workspace subsystem for session
  Client->>stream_tar_zstd: stream project directory
  stream_tar_zstd->>minimald: send zstd-compressed tar stream
  minimald-->>Client: complete upload
  CLI->>SFTP: read uploaded files
  SFTP-->>CLI: return workspace contents
Loading

Poem

A rabbit packs the files with care,
In tarred-up bundles through the air.
Zstd hops from client to host,
The daemon stores each leafy post.
SFTP checks the burrow bright—
Workspace uploaded just right!


Note

🎁 Summarized by CodeRabbit Free

Your organization has reached its limit of developer seats under the Pro Plan. For new users, CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please add seats to your subscription by visiting https://app.coderabbit.ai/login.If you believe this is a mistake and have available seats, please assign one to the pull request author through the subscription management page using the link above.

Comment @coderabbitai help to get the list of available commands.

@0chroma
0chroma force-pushed the 0chroma/feat-file-upload-263 branch from 8b917de to bad5c1f Compare July 14, 2026 17:43
End-to-end test that creates a session with a temp project dir
containing known files, then reads them back over SFTP to confirm
the tar+zstd upload landed in the session workspace.
Comment thread crates/minimal/src/file_upload.rs Outdated
0chroma added 6 commits July 14, 2026 15:52
async-tar panics if its Builder is dropped without calling into_inner.
The previous code would panic if add_dir_entries returned an error
mid-walk (e.g. permission denied on a file). Now the builder is
finalized before propagating the error. Also switched back to a
borrowed &mut Builder with Box::pin for the recursive async call.
The tar walker fell through to the regular-file branch for sockets,
FIFOs, and device files, causing an open error on Unix sockets (e.g.
ssh.sock inside the state dir). Now only regular files are archived;
sockets, FIFOs, and device files are silently skipped.
The tar walker now gracefully skips directories and files that return
EACCES (Permission denied) instead of failing the entire upload. This
fixes the e2e test failing on Linux KVM where /tmp contains systemd
private directories unreadable by the test user.
Tests for Unix sockets (skipped), permission-denied directories
(skipped), symlinks (stored as symlinks, not followed), and regular
files/dirs/symlinks (included with correct paths). These cover the
CI failures we hit: socket files in /tmp on macOS and permission-
denied systemd directories on Linux KVM.
The in-memory tar was trying to allocate 32GB on the KVM CI runner
because it was uploading the entire repo including target/ build
artifacts. Add a minimal default exclude list (.git, target,
node_modules) to avoid OOM. Full .gitignore support is tracked as
a follow-up on #263.
VM lanes pass E2E_PROJECT_DIR=/tmp because the guest image doesn't have
the host repo. But now that activate uploads the project dir, uploading
all of /tmp (systemd dirs, sockets, large files) causes OOM and timeouts.
Create a minimal project subdir with a minimal.toml under /tmp instead.
@0chroma

0chroma commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

sorry, I should have probably marked this as a draft, I'm glad I got the early feedback in though! helpful seeing things that maybe I didn't originally consider

The previous implementation buffered the entire tar archive and
zstd-compressed output in memory before sending, causing OOM on repos
with large target/ or .git/ directories.

Now the tar builder writes into a zstd encoder which writes to a
duplex pipe; a background task produces the tar while the caller
copies from the pipe to the SSH channel. Memory usage is bounded by
the 256KB pipe buffer.

Per @twitchyliquid64's review feedback on #748.
Comment thread crates/minimal/src/file_upload.rs Outdated
.git is project metadata users need (git status/diff/log), not a
rebuildable artifact. Only target/ and node_modules/ are excluded.
@0chroma
0chroma merged commit 8f2c47c into main Jul 15, 2026
27 checks passed
@0chroma
0chroma deleted the 0chroma/feat-file-upload-263 branch July 15, 2026 05:55
norrietaylor added a commit that referenced this pull request Jul 15, 2026
The shared session e2e only proved the lifecycle; it never forked a
session sandbox. Deepen it, on every lane, to attach an interactive shell
— which forks a real hakoniwa sandbox (on a VM lane, inside the guest over
the vsock bridge) — and prove the in-sandbox `min` helper: record a
genuinely-absent tool (jq — the shell stack composes base + curl, and jq
is pulled in by neither) as absent, `min add` it, then run it; its version
banner round-tripping proves `min add` reached the daemon over the
/run/minenv_sock relay and hardlinked the package into the live rootfs.

A session is interactive by design, so the proof drives it like a real
user through a REAL pty (scripts/e2e-attach-pty.py): pump the command
stream, then answer the session-exit Detach/Delete prompt with keystrokes
(Down + Enter => Delete). A pipe is not a tty and cannot answer that
prompt. Selecting Delete tears the session down, so it must be delisted.
Output is matched with a bash glob, not `printf | grep -q`, so `min add`'s
progress-bar flood cannot SIGPIPE a pipefail false-negative. A host-side
minimal.toml is seeded (uploaded to the session since #748) so the client
pre-flight neither prompts nor bails (#758); every seed is removed on
teardown.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
norrietaylor added a commit that referenced this pull request Jul 15, 2026
The shared session e2e only proved the lifecycle; it never forked a
session sandbox. Deepen it, on every lane, to attach an interactive shell
— which forks a real hakoniwa sandbox (on a VM lane, inside the guest over
the vsock bridge) — and prove the in-sandbox `min` helper: record a
genuinely-absent tool (jq — the shell stack composes base + curl, and jq
is pulled in by neither) as absent, `min add` it, then run it; its version
banner round-tripping proves `min add` reached the daemon over the
/run/minenv_sock relay and hardlinked the package into the live rootfs.

A session is interactive by design, so the proof drives it like a real
user through a REAL pty (scripts/e2e-attach-pty.py): pump the command
stream, then answer the session-exit Detach/Delete prompt with keystrokes
(Down + Enter => Delete). A pipe is not a tty and cannot answer that
prompt. Selecting Delete tears the session down, so it must be delisted.
Output is matched with a bash glob, not `printf | grep -q`, so `min add`'s
progress-bar flood cannot SIGPIPE a pipefail false-negative. A host-side
minimal.toml is seeded (uploaded to the session since #748) so the client
pre-flight neither prompts nor bails (#758); every seed is removed on
teardown.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
norrietaylor added a commit that referenced this pull request Jul 15, 2026
The shared session e2e only proved the lifecycle; it never forked a
session sandbox. Deepen it, on every lane, to attach an interactive shell
— which forks a real hakoniwa sandbox (on a VM lane, inside the guest over
the vsock bridge) — and prove the in-sandbox `min` helper: record a
genuinely-absent tool (jq — the shell stack composes base + curl, and jq
is pulled in by neither) as absent, `min add` it, then run it; its version
banner round-tripping proves `min add` reached the daemon over the
/run/minenv_sock relay and hardlinked the package into the live rootfs.

A session is interactive by design, so the proof drives it like a real
user through a REAL pty (scripts/e2e-attach-pty.py): pump the command
stream, then answer the session-exit Detach/Delete prompt with keystrokes
(Down + Enter => Delete). A pipe is not a tty and cannot answer that
prompt. Selecting Delete tears the session down, so it must be delisted.
Output is matched with a bash glob, not `printf | grep -q`, so `min add`'s
progress-bar flood cannot SIGPIPE a pipefail false-negative. A host-side
minimal.toml is seeded (uploaded to the session since #748) so the client
pre-flight neither prompts nor bails (#758); every seed is removed on
teardown.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
norrietaylor added a commit that referenced this pull request Jul 15, 2026
… every lane (#760)

* fix(checkouts): report exit status on git command failure

A failed git command surfaced only its stderr. When git is killed by a
signal (e.g. OOM mid-fetch) stderr holds only whatever it wrote before
dying — which can be a misleadingly benign line such as a "templates not
found" warning, masking that the process was killed rather than failing
cleanly.

Include the rendered ExitStatus (exit code or signal) in GitCommandFailed
and its mctx wrapper, so a signal kill is distinguishable from a
git-reported error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: exercise the session sandbox via interactive attach

The shared session e2e only proved the lifecycle; it never forked a
session sandbox. Deepen it, on every lane, to attach an interactive shell
— which forks a real hakoniwa sandbox (on a VM lane, inside the guest over
the vsock bridge) — and prove the in-sandbox `min` helper: record a
genuinely-absent tool (jq — the shell stack composes base + curl, and jq
is pulled in by neither) as absent, `min add` it, then run it; its version
banner round-tripping proves `min add` reached the daemon over the
/run/minenv_sock relay and hardlinked the package into the live rootfs.

A session is interactive by design, so the proof drives it like a real
user through a REAL pty (scripts/e2e-attach-pty.py): pump the command
stream, then answer the session-exit Detach/Delete prompt with keystrokes
(Down + Enter => Delete). A pipe is not a tty and cannot answer that
prompt. Selecting Delete tears the session down, so it must be delisted.
Output is matched with a bash glob, not `printf | grep -q`, so `min add`'s
progress-bar flood cannot SIGPIPE a pipefail false-negative. A host-side
minimal.toml is seeded (uploaded to the session since #748) so the client
pre-flight neither prompts nor bails (#758); every seed is removed on
teardown.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci: provision gvproxy for the KVM/macOS session-e2e lanes

The KVM and macOS session-e2e lanes boot a microVM whose guest daemon
clones the upstream `pkgs` repo during session mint. That clone needs the
guest's egress — NAT + DNS — which minvmd provides by spawning gvproxy as
the per-host-VM switch. Neither lane provisioned the gvproxy binary, and
its resolution is best-effort (never errors), so minvmd booted the VM
switchless: the guest's overlay had no other end and every egress —
including the clone — died with "Could not resolve host: github.com".

Fetch the pinned gvproxy (scripts/fetch-gvproxy.sh, already used by the
native minimald-root-integration job) and point minvmd at it via
MINVMD_GVPROXY_BIN on each lane's session-e2e step. The macOS lane's
fetch auto-selects the gvproxy-darwin asset.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature: File sync INTO a Session

3 participants