Skip to content
KarpelesLabPublic

About

A portable Linux sandbox in Rust: runs real Linux userlands and Docker images by emulating syscalls, not hardware — on KVM, Apple HVF, a software x86-64/arm64 interpreter, or in the browser via WebAssembly.

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Repository files navigation

nixvm

CI crates.io docs.rs License: MIT Browser demo

A portable, VM-style sandbox that runs a real Linux userland by emulating Linux syscalls directly — not by emulating hardware.

nixvm lets you run normally-dangerous tasks (npm install, pip install, untrusted build scripts, apk add, …) inside an isolated Linux environment, with or without hardware virtualization.

How it's different

A traditional VM (QEMU, Firecracker) boots a full guest Linux kernel and emulates hardware: a CPU, interrupt controller, virtio devices, block devices. That's heavy and slow to start.

nixvm takes the gVisor-style approach instead: there is no guest kernel. A real Linux userland (Alpine) runs directly on the host CPU or a software interpreter. When a guest process executes a syscall/svc instruction, control traps out and nixvm's own "kernel" — written in Rust — services the syscall. Files, memory, processes, threads, signals, and networking are all implemented in userspace, under our control.

Try it in your browser

▶ karpeleslab.github.io/nixvm

Because the whole thing — the CPU interpreters and the syscall kernel — is portable Rust with no OS dependencies, it compiles to WebAssembly and runs entirely in the page. There's no server: a real Alpine Linux userland boots locally, in a sandbox, inside your browser's sandbox.

  • Pick a guest architecture — arm64 or x86-64 — and press Start: the matching stock Alpine minirootfs boots to an interactive busybox sh (dynamic linking, real ld-musl, fork/exec — the same kernel the native build runs).
  • Everything is client-side: the rootfs is repacked into an in-memory squashfs under a copy-on-write tmpfs upper, and the guest runs on the software CPU interpreter for whichever arch you picked.

CI rebuilds and redeploys it to GitHub Pages on every push to master (see .github/workflows/pages.yml); the frontend is the Vue app in web/ and the wasm entry point is src/wasm.rs (which also still exposes the original static-ELF runner, run_elf).

Status

Functional on the software-interpreter path: a real Rust syscall kernel, two working CPU interpreters (aarch64 and a growing x86-64), static, static-PIE, and dynamically-linked ELF loading (real ld-musl boots stock Alpine and loads large C++ programs' full shared-library graphs, e.g. node's ~15 libs), multi-threaded/multi-process scheduling with an SMP worker-thread pool (POSIX threads share one fd table, so libuv's cross-thread wakeups work), an in-VM network stack, and several filesystem backends — all covered by 505 tests (cargo test). Node.js runs — on the software interpreter and the hardware backends: node -e … executes real JavaScript to completion — event loop, setTimeout/setInterval, promises, JSON, stdio — and exits cleanly, including a hot loop that tiers up to V8's TurboFan JIT with concurrent background compilation on. Floating point is a dependency-free soft-float (bit-identical on native and wasm) that models what f64 hardware can't: true 80-bit x87 extended precision and SSE MXCSR directed rounding + IEEE exception flags — validated bit-for-bit against the host's native arithmetic and against real x87/SSE hardware via KVM. Hardware acceleration is live on both target hosts: the HVF backend (macOS/arm64) runs a real Alpine userland, and the KVM backend (Linux/x86-64) runs a real statically-linked glibc binary end-to-end on real hardware (long mode at ring 3, syscall trapped via an LSTAR trampoline, TLS via arch_prctl). vcpu::select prefers hardware and falls back to the interpreter, which remains the portable default and the everywhere-CI path. See ROADMAP.md for the phased plan and what's next.

What works today

Area Status
Guest architectures aarch64 (interpreter, primary target); x86-64 (interpreter, growing)
Execution backends software interpreter — working (portable, CI-tested). HVF (macOS/arm64) — runs a real Alpine userland on hardware (guest at EL0 with the MMU on over its own arm64 page tables, every exception handed to the host through an EL1 vector page of hvcs, SMP, preemption and signals into syscall-free loops); needs a codesigned binary (scripts/hvf-test.sh). KVM (Linux/x86-64) — runs a real static glibc binary end-to-end on hardware (guest at CPL3 in long mode over fixed identity page tables, syscall → LSTAR trampoline → KVM_EXIT_HLT); no entitlement needed, its tests run under plain cargo test wherever /dev/kvm is accessible and skip elsewhere. vcpu::select prefers hardware and falls back to the interpreter (NIXVM_INTERP=1 forces it); dynamic linking + multi-process on the hardware backends are the follow-up. KVM/arm64 — not started.
ELF loading static (ET_EXEC), static-PIE (ET_DYN with R_*_RELATIVE/R_*_IRELATIVE fixups), and dynamic linking (PT_INTERP → ld-musl, file-backed mmap of .sos) — working; real Alpine boots on the interpreter.
Processes & threads clone/fork/execve/wait4/exit(_group); CLONE_VM+CLONE_THREAD shared-address-space threads; real futex WAIT/WAKE parking
SMP scheduler a pool of ncpus host worker threads run guest compute in parallel; syscalls are serviced serially on the main thread (a big-kernel-lock model, Kernel::set_ncpus/NIXVM_CPUS)
Signals rt_sigaction/rt_sigprocmask/kill/tgkill and default-disposition delivery (terminate/ignore) — custom handler invocation (frame push + PC redirect) is not implemented
Filesystem tmpfs (rw in-memory), overlay (COW upper/lower), passthrough (host dir, symlink/TOCTOU-safe — see below), devfs, procfs, sysfs; a real on-disk squashfs/ext reader (fstoolfs, via the optional fstool crate) exists but isn't wired into the default mount table yet
Networking in-VM AF_UNIX + AF_INET/AF_INET6 loopback (TCP stream + UDP datagram), plus host-passthrough egress (opt-in NIXVM_NET=host, native only): guest TCP/UDP to routable addresses is bridged to real host sockets, so apk update && apk add work over the real internet on x86-64. DNS via a host UDP socket + injected /etc/resolv.conf. A browser (WebSocket/pktkit) transport and a smoltcp NAT stack are the follow-ups.
I/O multiplexing poll/ppoll/select/pselect6, epoll_create1/ctl/wait/pwait2, eventfd2, timerfd_*
Docker/OCI images pull from any registry (Docker Hub, GHCR, Quay, GCR, a mirror; anonymous or with credentials) or load a docker save / OCI layout archive, flatten the layers (whiteouts, opaque dirs) and run the entrypoint as docker run would — no daemon. CLI, library and wasm (details)
GUI apps a built-in Wayland compositor (--display WxH): foot, GTK 3, weston clients and Chromium run with no compositor in the guest (details)
Browser demo (wasm) working — live demo: wasm32-unknown-unknown + interpreter, built and deployed to GitHub Pages by CI on every push to main (details)

The syscall dispatch table in src/kernel/mod.rs is broad: process/thread lifecycle (incl. waitid/clone3/execveat), sessions & process groups (setsid/setpgid/…), fd/file I/O including positioned/vectored (pread/pwrite/preadv/pwritev) and copy (sendfile/copy_file_range), truncate/ftruncate/fallocate, statx, mmap/brk/mprotect/mremap, signals, networking (loopback + host egress, incl. sendmsg/recvmsg/ sendmmsg/recvmmsg), poll/epoll/eventfd/timerfd/memfd_create, and a set of always-succeed/no-op syscalls (the uid/gid credential family, sync family, sched_* setters, Mlock*, …) real programs probe at startup. A busybox command battery, apk, and a dynamically-linked binary's startup all dispatch with zero unsupported syscalls on x86-64. Anything not in the table returns ENOSYS and is recorded in an unsupported() ledger you can inspect after a run.

Quickstart

cargo build --release
cargo test

On macOS/arm64, aarch64 guests run on Hypervisor.framework once the binary is codesigned with the com.apple.security.hypervisor entitlement (the released macOS binaries are); an unsigned one quietly uses the interpreter:

cargo build --release
scripts/sign-macos.sh        # ad-hoc signs target/release/{nixvm,run-elf} (again after any rebuild)
target/release/nixvm run --root ./alpine-rootfs -- /bin/sh

The HVF backend's own tests are #[ignore]d for the same reason (cargo test can't sign its test binary):

scripts/hvf-test.sh          # builds, ad-hoc codesigns, runs the HVF tests

The nixvm CLI and the public Sandbox::run() API drive the full image-based pipeline (resolve a cached Alpine squashfs → mount → load → run), but image fetch/caching is still a stub (ROADMAP Phase 11) and the default mount table doesn't yet wire in the squashfs reader — so that path isn't runnable end-to-end yet.

What does work today is running a static ELF directly, via the dev harnesses or the embeddable Sandbox::exec_elf():

# Run a statically-linked aarch64 ELF on the interpreter:
cargo run --bin run-elf -- ./some-static-aarch64-binary

# Same, for a statically-linked x86-64 ELF:
cargo run --bin run-elf-x86 -- ./some-static-x86_64-binary

Useful environment variables (both harnesses):

  • NIXVM_ROOT=/path/to/alpine-rootfs — mount / as overlay(passthrough(NIXVM_ROOT) read-only, tmpfs) instead of a bare tmpfs, so the guest sees a real Alpine tree with a copy-on-write upper.
  • NIXVM_CPUS=N — run guest compute on N host worker threads (the SMP scheduler); default is 1 (single-threaded cooperative scheduling).
  • NIXVM_TRACE=1 — log every dispatched syscall (pid, pc, name, raw number, args) to stderr.
  • NIXVM_INTERP=1 — skip the hardware-backend probe in vcpu::select and run on the software interpreter (a debugging/parity escape hatch).
  • NIXVM_NET=host — enable host-passthrough network egress: guest sockets to routable addresses reach the real internet via host sockets (native only; off by default = loopback-only). Lets apk/wget/curl fetch from the network.

Root images from a registry, and a persistent root

nixvm ships no images of its own: a guest root is an image from any OCI registry, pinned by digest. With the images feature, image::pull takes one for an architecture — over HTTPS in pure Rust (rsurl, TLS on purecrypto; no C, no libc of its own, so it builds for the libc-free fullrust target too), every manifest and blob checked against its digest — caches its blobs and the flattened root (a squashfs file keyed by the manifest digest) on disk, and mounts the root read-only straight from that file. A later boot of a reference by digest makes no request at all. image::ALPINE names the official Alpine image (docker.io/library/alpine:3.22@sha256:49340b34…; the same digest works on mirror.gcr.io/library/alpine when Docker Hub rate-limits). A tag-only reference (alpine:3.22) works too, but only a digest pins what you get.

fs::Overlay::persistent puts a host directory under the image as its writable layer, so a development image is the official one plus a layer of your own: install the tools once, in a guest with the network, and every later guest over the same directory has them.

use nixvm::{Arch, fs::Overlay, image, vm::Vm};
let lower = image::pull(image::ALPINE, Arch::X86_64, cache_dir, image::PullOptions::default())?;
let root = Overlay::persistent(Box::new(lower), upper_dir)?;
// Once, with NIXVM_NET=host (or the embedder's network):
//     apk add git make gcc musl-dev
// From then on, guests booted over `upper_dir` have them.
let vm = Vm::boot_root(Box::new(root), vec!["/bin/sh".into()], 1 << 30)?;

PullOptions takes credentials, a mirror (registry_base), a proxy, extra CA certificates and a custom connector (rsurl's Connector, re-exported as image::Connector) for an embedder that routes connections itself.

The upper directory holds changed files at their paths, and deletions as .wh.<name> marker files (the OCI-layer convention; see src/fs/overlay.rs). Run as a host user who isn't root, the upper can't take a guest's chown to root: apk add installs everything but reports those as errors (and exits 1).

Run Docker images

Docker without dockerd: nixvm pulls an image (or reads a docker save / OCI layout archive), flattens its layers into a read-only root under a writable layer, and runs its entrypoint with its environment, working directory and user, as docker run would. Build with the oci feature (and fetch to pull natively); blobs are cached by digest, so a second pull fetches only the manifests.

cargo build --release --features oci,fetch
target/release/nixvm run --image hello-world
NIXVM_NET=host target/release/nixvm run --image python:3.12-slim -- python3 -c 'print(1)'
target/release/nixvm run --image nginx:alpine --entrypoint sh -- -c 'nginx -t'
target/release/nixvm run --image ./saved.tar --user 1000:1000 -- id   # docker save output
target/release/nixvm pull ghcr.io/owner/image:tag

Arguments after -- replace the image's Cmd; --entrypoint, --user and --env work as in docker run. An image without a build for the host's architecture runs as the other one on the interpreter (--arch picks). NIXVM_REGISTRY_USER/NIXVM_REGISTRY_PASSWORD log in.

As a library, nixvm::oci::pull (or oci::Image::from_archive) gives an image and vm::Vm::boot_image boots it as a guest you drive — write its stdin, pump its output, interrupt it, read its exit code — or Sandbox::builder().image("alpine:3.20") runs it like the CLI does.

In the browser the pull goes over the guest's own network link: the page brings up a Network on its packet tunnel (the live demo's grouterd WebSocket), ImagePull.pull_over(net) fetches the image over it with nixvm doing the DNS, TLS (purecrypto, pure Rust) and HTTP itself — so CORS never applies and no proxy is needed — and Terminal.from_image_on(pull, net, args, env) boots it on that same link. The live demo does this with ?image=<name> (and &cmd= to replace the image's command). A page without a tunnel can instead fetch each request ImagePull names itself (through a CORS proxy) and feed the responses back, or hand Terminal.from_image_archive a docker save file the user picked.

GUI apps: the built-in Wayland display

nixvm runs a Wayland compositor of its own for the guest — no compositor (and no X server) inside it. Guest apps connect to /tmp/wayland-0 as to any compositor and draw into shared memory; nixvm composites their windows into one frame and feeds keyboard and pointer input back.

target/release/nixvm run --root ./alpine-with-foot --display 1024x768 \
    --display-png shot.png -- foot

--display WxH gives the guest the display, with WAYLAND_DISPLAY and an XDG_RUNTIME_DIR set; --display-png saves the frame whenever it changes. foot, GTK 3 apps, weston's clients and Chromium (--ozone-platform=wayland) run on it. The window policy is a kiosk's: toplevels fill the display, the newest on top with the keyboard; dialogs are centered; menus and other popups go where their positioner puts them. It speaks wl_compositor, wl_subcompositor, wl_shm, wl_seat (pointer and a US keyboard), wl_output + xdg-output, xdg-shell, xdg-decoration, wp_viewporter and a data device (a clipboard shared by the guest's apps). Apps render in software unless nixvm is built with --features gpu and run with NIXVM_GPU=egl: then it also offers zwp_linux_dmabuf_v1, and GL apps (Mesa's Wayland EGL: es2gears, weston-simple-egl, GTK's GL area; Chromium with GPU rasterization and compositing) render on the virtual GPU and show up. With a GPU the compositor itself draws on it too: windows are textures in the GPU's context, a GPU buffer is copied in GPU-to-GPU and the frame is read back only when it is taken (and only what changed) — es2gears at 1024×768 runs at ~2500 frames/s instead of ~580 when every commit was read back (NIXVM_DISPLAY_GL=0 keeps the CPU compositor). See the GPU section of ROADMAP.md.

As a library, Sandbox::builder().display(w, h) and Sandbox::display() give a display::Display handle: frame() hands over the RGBA frame when it changed (each call is a presentation: the apps' frame callbacks fire), input() sends pointer, keyboard, scroll and resize events, png() encodes it. The compositor is pure Rust with no threads, so it builds for the browser (wasm) too, where the embedder draws the frames on a <canvas> — or, with the GPU on, present() blits them onto the WebGL2 canvas the GPU renders with, and no pixels leave the GPU.

Embedding: run one command

Sandbox::execute() runs a command for a program embedding nixvm: the guest gets the stdio, working directory, binds, network and limits it is given and nothing else, nothing is printed, and the result says how the run went.

use nixvm::sandbox::{Capture, Network, Sandbox};

let out = Capture::new();
let outcome = Sandbox::builder()
    .root_dir("/srv/alpine")                  // an extracted rootfs, read-only under a tmpfs
    .no_work_dir()                            // no implicit /work
    .bind("/home/me/project", "/home/me/project")
    .cwd("/home/me/project")
    .network(Network::None)                   // or Network::Host, or an Egress of your own
    .wall_time(std::time::Duration::from_secs(60))
    .stdout(out.writer())                     // any Write; stdin/stderr likewise
    .shell("echo hi; pwd")
    .execute()?;
// outcome.exit_code, .signal, .limit (wall/CPU time, memory), .unsupported
// (syscalls nixvm lacks), .notices (faults, limits, policy denials), .backend

SandboxBuilder::mount puts a filesystem of your own at a guest path, on_notice sees the kernel's notices as they happen, and GuestMemory::with_default_layout, GuestMemory::calibrate_vdso and vcpu::select_or_interpreter are there for assembling a guest by hand.

The library reads no environment variables: the NIXVM_* switches below are honoured by the nixvm binaries, which install them with nixvm::env::Env::from_process().install(). An embedder gets the defaults unless it installs settings of its own (Env::new().with("NIXVM_TRACE", "1")).

Default sandbox layout

/           tmpfs by default; overlay(passthrough(NIXVM_ROOT), tmpfs) when set
/work       the host's current working directory, passed through read-write
/tmp,/dev,  synthesized in-sandbox (tmpfs / devfs / procfs / sysfs)
/proc,/sys

Design notes

  • Single crate, feature-gated deps. Everything lives under one nixvm crate (abi, vcpu, loader, fs, kernel, image, sandbox) instead of a workspace; heavy or platform-specific dependencies are opt-in cargo features (hvf, kvm, interp, fstool, wasm, cli) rather than separate crates. The core builds fully offline with zero third-party dependencies.
  • unsafe policy. unsafe is confined to five documented sites; the interpreter and kernel paths have none. (1) vcpu::hvf — the Hypervisor.framework FFI (hv_vm_*/hv_vcpu_*) — and vcpu::kvm, its Linux equivalent (hand-rolled /dev/kvm ioctls, structs verified against the kernel headers, plus the mmap'd per-vcpu kvm_run page). (2) vcpu::region — one 16 KiB-aligned alloc_zeroed allocation of guest RAM (a hardware backend maps its raw pointer into the guest; safe std can't express over-aligned allocation), with all raw pointer access wrapped in checked copy methods. (3) fs::passthrough — hand-declares a handful of dirfd-relative *at(2) FFI calls (openat/ unlinkat/mkdirat/symlinkat/renameat/readlinkat/fdopendir/ readdir) because safe, TOCTOU-free path confinement genuinely requires them and std exposes none of it. Resolution walks the host path one component at a time from a dirfd opened on the mount root, with O_NOFOLLOW on every component; a symlink is never handed to the kernel to auto-follow — its target is read and re-spliced into the walk, re-anchored so an absolute target or a .. chain can never resolve above the mount root. The final, actual I/O syscall is always issued directly against (parent_dirfd, name) with O_NOFOLLOW, so even a symlink swapped in mid-race fails safely instead of redirecting the I/O. (4) kernel::egress — setsockopt(2) on a host socket, for the keepalive and linger options a guest sets that std has no call for, and ioctl(2) (SIOCOUTQ; getsockopt(SO_NWRITE) on macOS) to see what the peer hasn't acknowledged yet, which a guest's lingering close waits for.

Limits and isolation

A guest can't exhaust the host: file data in guest memory, tasks, open files, guest RAM and disk growth in host directories all have ceilings, and hitting one is an ordinary error in the guest. Defaults come from Limits::for_this_host(), and can be set with SandboxBuilder::limits, Vm::set_limits or the NIXVM_LIMIT_* environment variables listed under Phase 9 in ROADMAP.md. CPU and run time have no default limit, since nothing can guess what a job needs: set them for untrusted work.

Inside the VM, namespaces work as on Linux, including unprivileged user namespaces, so tools like bubblewrap run in a guest.

On top of that, the embedder can refuse syscalls to the whole guest, whatever it does inside: SandboxBuilder::syscall_policy, or NIXVM_DENY_SYSCALLS=@mount,ptrace:kill (see src/policy.rs). Refusals are logged with NIXVM_LOG_DENIALS=1.

License

MIT — see LICENSE.

About

A portable Linux sandbox in Rust: runs real Linux userlands and Docker images by emulating syscalls, not hardware — on KVM, Apple HVF, a software x86-64/arm64 interpreter, or in the browser via WebAssembly.

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages