feat(minimal): upload project files to session on activate - #748
Conversation
After CreateSession succeeds, the client now tar+zstd's the project directory and streams it to the daemon's WorkspaceFilesTarZst subsystem, which unpacks it into the session workspace. This is the file-sync-INTO direction from #263. The daemon-side receiver already existed (rpc.rs:404); this adds the client-side sender. The upload happens after session creation and before the attach prompt.
|
Warning Review limit reached
Next review available in: 46 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughChangesThe activation flow now streams the project workspace as a zstd-compressed tar archive to the daemon and verifies the resulting session files through integration testing. Workspace upload flow
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant Client
participant stream_tar_zstd
participant minimald
participant SFTP
CLI->>Client: activate project
Client->>minimald: request workspace subsystem for session
Client->>stream_tar_zstd: stream project directory
stream_tar_zstd->>minimald: send zstd-compressed tar stream
minimald-->>Client: complete upload
CLI->>SFTP: read uploaded files
SFTP-->>CLI: return workspace contents
Poem
Note 🎁 Summarized by CodeRabbit FreeYour organization has reached its limit of developer seats under the Pro Plan. For new users, CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please add seats to your subscription by visiting https://app.coderabbit.ai/login.If you believe this is a mistake and have available seats, please assign one to the pull request author through the subscription management page using the link above. Comment |
8b917de to
bad5c1f
Compare
End-to-end test that creates a session with a temp project dir containing known files, then reads them back over SFTP to confirm the tar+zstd upload landed in the session workspace.
async-tar panics if its Builder is dropped without calling into_inner. The previous code would panic if add_dir_entries returned an error mid-walk (e.g. permission denied on a file). Now the builder is finalized before propagating the error. Also switched back to a borrowed &mut Builder with Box::pin for the recursive async call.
The tar walker fell through to the regular-file branch for sockets, FIFOs, and device files, causing an open error on Unix sockets (e.g. ssh.sock inside the state dir). Now only regular files are archived; sockets, FIFOs, and device files are silently skipped.
The tar walker now gracefully skips directories and files that return EACCES (Permission denied) instead of failing the entire upload. This fixes the e2e test failing on Linux KVM where /tmp contains systemd private directories unreadable by the test user.
Tests for Unix sockets (skipped), permission-denied directories (skipped), symlinks (stored as symlinks, not followed), and regular files/dirs/symlinks (included with correct paths). These cover the CI failures we hit: socket files in /tmp on macOS and permission- denied systemd directories on Linux KVM.
The in-memory tar was trying to allocate 32GB on the KVM CI runner because it was uploading the entire repo including target/ build artifacts. Add a minimal default exclude list (.git, target, node_modules) to avoid OOM. Full .gitignore support is tracked as a follow-up on #263.
VM lanes pass E2E_PROJECT_DIR=/tmp because the guest image doesn't have the host repo. But now that activate uploads the project dir, uploading all of /tmp (systemd dirs, sockets, large files) causes OOM and timeouts. Create a minimal project subdir with a minimal.toml under /tmp instead.
|
sorry, I should have probably marked this as a draft, I'm glad I got the early feedback in though! helpful seeing things that maybe I didn't originally consider |
The previous implementation buffered the entire tar archive and zstd-compressed output in memory before sending, causing OOM on repos with large target/ or .git/ directories. Now the tar builder writes into a zstd encoder which writes to a duplex pipe; a background task produces the tar while the caller copies from the pipe to the SSH channel. Memory usage is bounded by the 256KB pipe buffer. Per @twitchyliquid64's review feedback on #748.
.git is project metadata users need (git status/diff/log), not a rebuildable artifact. Only target/ and node_modules/ are excluded.
The shared session e2e only proved the lifecycle; it never forked a session sandbox. Deepen it, on every lane, to attach an interactive shell — which forks a real hakoniwa sandbox (on a VM lane, inside the guest over the vsock bridge) — and prove the in-sandbox `min` helper: record a genuinely-absent tool (jq — the shell stack composes base + curl, and jq is pulled in by neither) as absent, `min add` it, then run it; its version banner round-tripping proves `min add` reached the daemon over the /run/minenv_sock relay and hardlinked the package into the live rootfs. A session is interactive by design, so the proof drives it like a real user through a REAL pty (scripts/e2e-attach-pty.py): pump the command stream, then answer the session-exit Detach/Delete prompt with keystrokes (Down + Enter => Delete). A pipe is not a tty and cannot answer that prompt. Selecting Delete tears the session down, so it must be delisted. Output is matched with a bash glob, not `printf | grep -q`, so `min add`'s progress-bar flood cannot SIGPIPE a pipefail false-negative. A host-side minimal.toml is seeded (uploaded to the session since #748) so the client pre-flight neither prompts nor bails (#758); every seed is removed on teardown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared session e2e only proved the lifecycle; it never forked a session sandbox. Deepen it, on every lane, to attach an interactive shell — which forks a real hakoniwa sandbox (on a VM lane, inside the guest over the vsock bridge) — and prove the in-sandbox `min` helper: record a genuinely-absent tool (jq — the shell stack composes base + curl, and jq is pulled in by neither) as absent, `min add` it, then run it; its version banner round-tripping proves `min add` reached the daemon over the /run/minenv_sock relay and hardlinked the package into the live rootfs. A session is interactive by design, so the proof drives it like a real user through a REAL pty (scripts/e2e-attach-pty.py): pump the command stream, then answer the session-exit Detach/Delete prompt with keystrokes (Down + Enter => Delete). A pipe is not a tty and cannot answer that prompt. Selecting Delete tears the session down, so it must be delisted. Output is matched with a bash glob, not `printf | grep -q`, so `min add`'s progress-bar flood cannot SIGPIPE a pipefail false-negative. A host-side minimal.toml is seeded (uploaded to the session since #748) so the client pre-flight neither prompts nor bails (#758); every seed is removed on teardown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared session e2e only proved the lifecycle; it never forked a session sandbox. Deepen it, on every lane, to attach an interactive shell — which forks a real hakoniwa sandbox (on a VM lane, inside the guest over the vsock bridge) — and prove the in-sandbox `min` helper: record a genuinely-absent tool (jq — the shell stack composes base + curl, and jq is pulled in by neither) as absent, `min add` it, then run it; its version banner round-tripping proves `min add` reached the daemon over the /run/minenv_sock relay and hardlinked the package into the live rootfs. A session is interactive by design, so the proof drives it like a real user through a REAL pty (scripts/e2e-attach-pty.py): pump the command stream, then answer the session-exit Detach/Delete prompt with keystrokes (Down + Enter => Delete). A pipe is not a tty and cannot answer that prompt. Selecting Delete tears the session down, so it must be delisted. Output is matched with a bash glob, not `printf | grep -q`, so `min add`'s progress-bar flood cannot SIGPIPE a pipefail false-negative. A host-side minimal.toml is seeded (uploaded to the session since #748) so the client pre-flight neither prompts nor bails (#758); every seed is removed on teardown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… every lane (#760) * fix(checkouts): report exit status on git command failure A failed git command surfaced only its stderr. When git is killed by a signal (e.g. OOM mid-fetch) stderr holds only whatever it wrote before dying — which can be a misleadingly benign line such as a "templates not found" warning, masking that the process was killed rather than failing cleanly. Include the rendered ExitStatus (exit code or signal) in GitCommandFailed and its mctx wrapper, so a signal kill is distinguishable from a git-reported error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: exercise the session sandbox via interactive attach The shared session e2e only proved the lifecycle; it never forked a session sandbox. Deepen it, on every lane, to attach an interactive shell — which forks a real hakoniwa sandbox (on a VM lane, inside the guest over the vsock bridge) — and prove the in-sandbox `min` helper: record a genuinely-absent tool (jq — the shell stack composes base + curl, and jq is pulled in by neither) as absent, `min add` it, then run it; its version banner round-tripping proves `min add` reached the daemon over the /run/minenv_sock relay and hardlinked the package into the live rootfs. A session is interactive by design, so the proof drives it like a real user through a REAL pty (scripts/e2e-attach-pty.py): pump the command stream, then answer the session-exit Detach/Delete prompt with keystrokes (Down + Enter => Delete). A pipe is not a tty and cannot answer that prompt. Selecting Delete tears the session down, so it must be delisted. Output is matched with a bash glob, not `printf | grep -q`, so `min add`'s progress-bar flood cannot SIGPIPE a pipefail false-negative. A host-side minimal.toml is seeded (uploaded to the session since #748) so the client pre-flight neither prompts nor bails (#758); every seed is removed on teardown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: provision gvproxy for the KVM/macOS session-e2e lanes The KVM and macOS session-e2e lanes boot a microVM whose guest daemon clones the upstream `pkgs` repo during session mint. That clone needs the guest's egress — NAT + DNS — which minvmd provides by spawning gvproxy as the per-host-VM switch. Neither lane provisioned the gvproxy binary, and its resolution is best-effort (never errors), so minvmd booted the VM switchless: the guest's overlay had no other end and every egress — including the clone — died with "Could not resolve host: github.com". Fetch the pinned gvproxy (scripts/fetch-gvproxy.sh, already used by the native minimald-root-integration job) and point minvmd at it via MINVMD_GVPROXY_BIN on each lane's session-e2e step. The macOS lane's fetch auto-selects the gvproxy-darwin asset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary
After
CreateSessionsucceeds, the client now tar+zstd's the project directory and streams it to the daemon'sWorkspaceFilesTarZstsubsystem, which unpacks it into the session workspace. This is the file-sync-INTO direction from #767.Background
The daemon-side receiver already existed (
rpc.rs:404—WorkspaceFilesTarZstsubsystem), but theminimalCLI had no client-side sender.cmd_activatesent onlySessionConfig+WireContributionover RPC — no file payload. The daemon readproject_pathdirectly on the host (only works when co-located).Changes
client.rs— newupload_workspace_filesmethod: opens a channel, setsMINIMAL_SESSION_ID, requests theWorkspaceFilesTarZstsubsystem, streams a zstd-compressed tarball, and drains the responsefile_upload.rs— new module:tar_directoryrecursively walks a directory and builds an in-memory tar archive preserving relative paths, symlinks, and empty directorieslib.rs—cmd_activatecallsupload_workspace_filesafter session creation, before the attach promptCargo.toml— addedasync-tar,async-compression,constcatdepsOpen questions
.git/,node_modules/, build artifacts, etc. Should we support.gitignoreparsing, a hardcoded default exclude list (.git,node_modules,target,__pycache__,.venv), a.minimalignorefile, or some combination? Tracked as an open question on feature: File sync INTO a Session #767.Closes #767 (file sync INTO a session).
Summary by CodeRabbit
.git,target, andnode_modules.