mvm — microVM sandboxes on libkrun

mvm runs OCI container images as hardware-isolated microVMs with a docker-style CLI, an HTTP API, and a ratatui dashboard. Each sandbox is a real KVM virtual machine (via libkrun) booting its own kernel, with the image rootfs shared over virtiofs.

$ mvm serve &
$ mvm pull alpine
$ mvm run alpine echo hello-from-microvm
    hello-from-microvm
$ mvm run --name dev alpine sleep infinity &
$ mvm exec dev uname -a
    Linux localhost 6.12.91 #1 SMP PREEMPT_DYNAMIC ... x86_64 Linux
$ printf 'stdin works\n' | mvm exec -i dev cat
    stdin works
$ mvm exec -it dev sh          # interactive shell on a pty
$ mvm run --rm -it alpine sh        # one-shot interactive sandbox
$ mvm create -it --name box alpine sh && mvm start -a box
                               # long-lived one; ctrl-p ctrl-q detaches,
                               # mvm attach box comes back to it
$ mvm stop dev && mvm rm dev

      mvm CLI / mvm-tui (thin HTTP clients)
                    │
        HTTP API (mvm serve — axum)
                    │
             Sandbox Manager
                    │
    ┌───────────────┼───────────────┐
 OCI Image      Network         Storage
  Manager       Manager         Manager
    └───────────────┼───────────────┘
                    │
      libkrun shim (one process per VM)
                    │
                   KVM

• Daemon + client. mvm serve owns all state behind a local HTTP API (default 127.0.0.1:24642, override with MVM_HOST). CLI and TUI are stateless clients.
• One shim process per VM. libkrun's krun_start_enter() takes over the calling process, so the daemon spawns a detached mvm __vm-shim per sandbox; its stdout/stderr is the guest console, pumped to console.log and live log followers. libkrun's own diagnostics go to a separate krun.log, so hypervisor noise never shows up as guest output.
• Guest agent as the guest's init. A static musl binary is injected at /.mvm/agent (libkrun's own /init.krun is PID 1 and execs it); it spawns the workload — on its own pty with -t — reaps zombies, and serves exec (with stdin/stdout/stderr streaming and exit codes) over vsock — no networking required in the guest.
• Pure-Rust image pulls. Registry auth, manifest resolution, blob verification, layer unpack and OCI whiteouts are implemented in-tree; later layers can replace existing paths, including hard-link destinations; no skopeo/podman needed.

• Linux x86_64 with KVM (/dev/kvm read-write for your user), or macOS on Apple Silicon (macOS 14+, Hypervisor.framework)
• libkrun and libkrunfw (libkrun.so.1 / libkrun.dylib, headers for building). On macOS: scripts/install-darwin.sh installs everything (rustup, zig, libkrun + gvproxy from the libkrun/krun Homebrew tap).
• Rust toolchain + the musl target matching your host arch (guests are same-arch: x86_64-unknown-linux-musl or aarch64-unknown-linux-musl) for the guest agent
• Optional: gvproxy for outbound networking / port forwarding

Guests run your host's architecture — on Apple Silicon pull arm64 images (mvm pull resolves the matching platform manifest automatically).

Rootless operation is fully supported (state lives in ~/.local/share/mvm; as root: /var/lib/mvm; override: MVM_DATA_DIR).

$ scripts/build.sh        # release binaries + static agent → dist/
$ scripts/integration.sh  # end-to-end test: boots real VMs (needs KVM/libkrun + network)

On macOS run scripts/install-darwin.sh once first; build.sh then cross-compiles the agent with cargo zigbuild and codesigns dist/mvm with the Hypervisor.framework entitlement (without it macOS refuses to start VMs).

mvm looks for mvm-agent next to its own binary, or at MVM_AGENT_PATH. The agent must be the static musl build — it runs inside guests whose libc you don't control.

run/create options: --name, -e KEY=VAL, -v host:guest[:ro], -p host:guest, --net none|tsi|gvproxy[:<socket>]|tap:<dev>, --cpus N, -m MiB, -w workdir, --rm (run only: remove the sandbox when the workload exits, docker-style).

Any command that takes a SANDBOX accepts its id, a unique id prefix, or its --name (names are unique; creating a second sandbox with a taken name is refused).

-i and -t are properties of the sandbox, fixed at create time (as in docker) — start has no -i/-t of its own, it reuses what the sandbox was created with. So a long-lived interactive VM is:

$ mvm create -it --name dev alpine sh    # or: mvm run -it --name dev …
$ mvm start dev                          # detached
$ mvm attach dev                         # ctrl-p ctrl-q to leave it running

mvm run keeps the sandbox after the workload exits (docker's default), so a short-lived run alpine true still leaves a true sandbox to start afterwards. Pass --rm to remove it on exit (the inverse of the old ephemeral default).

GET    /health
GET    /api/v1/sandboxes                 POST   /api/v1/sandboxes
GET    /api/v1/sandboxes/{id}            DELETE /api/v1/sandboxes/{id}
POST   /api/v1/sandboxes/{id}/start      POST   /api/v1/sandboxes/{id}/stop
POST   /api/v1/sandboxes/{id}/resize                  {"vcpus":N,"ram_mib":N}
POST   /api/v1/sandboxes/{id}/clone    {"spec":{...},"fork":bool}
GET    /api/v1/sandboxes/{id}/logs?follow=bool&tail=N&raw=bool  (console)
POST   /api/v1/sandboxes/{id}/exec                    (framed event stream)
POST   /api/v1/sandboxes/{id}/exec/{session}/stdin[?eof=true]
POST   /api/v1/sandboxes/{id}/exec/{session}/resize   {"cols":N,"rows":N}
POST   /api/v1/sandboxes/{id}/console/resize           {"cols":N,"rows":N}
GET    /api/v1/images                    DELETE /api/v1/images/{name}
POST   /api/v1/images/pull                            (JSON-lines progress)

Exec/log streams use length-prefixed JSON frames (u32 BE length + JSON), defined in crates/common/src/protocol.rs.

The console stream drops terminal query sequences (DSR ESC[6n, Device Attributes ESC[c) — replaying a question makes the reader's terminal answer into its own input buffer. raw=true keeps them, and is only for a client that owns a terminal and answers them, i.e. mvm attach / mvm run -it.

• Rootless userns mode (automatic): mvm serve re-execs itself inside a user namespace mapping uid 0 to your user and uids 1..65535 to your /etc/subuid range (podman-style). libkrun's in-process virtiofs server then runs as namespace-root, so guest chown/ownership works with full fidelity — image files appear root-owned, apk/apt/useradd work. Requires /etc/subuid + /etc/subgid entries and newuidmap/newgidmap; degrades gracefully (with a warning) when missing. Opt out: MVM_USERNS=0. Linux-only — on macOS the daemon runs with host credentials.
• Storage drivers (auto-selected, override MVM_STORAGE_DRIVER) — the guest root is always served over virtiofs:
  • overlay (default for root and userns mode): kernel OverlayFS, image rootfs as lower layer, per-sandbox upper. Changes persist across stop/start. Auto-probed; falls back to copy if unsupported.
  • copy: per-sandbox rootfs copy; universal fallback (and the macOS default — APFS copies are clonefile(2), cheap). Rootfs is rebuilt (wiped) on every start.
• Network profiles (--net, placed before the image):

  • none (default): fully isolated — loopback only. (mvm attaches a dead NIC to switch off libkrun's default TSI backend, which would otherwise give every guest transparent host networking.)

  • tsi: libkrun's Transparent Socket Impersonation — guest sockets are serviced by the host directly. Outbound internet + DNS with zero setup (no proxy, no NIC, no root), plus -p port maps. The guest shares the host's network identity; use gvproxy when you want NAT separation.

  • gvproxy: rootless userspace NAT (the same stack podman machine uses). Outbound internet plus -p host:guest port forwards, with no setup beyond having the gvproxy binary on PATH (MVM_GVPROXY_BIN overrides): the daemon starts a private gvproxy per sandbox and stops it with the sandbox.

    $ mvm run --net gvproxy -p 8080:80 alpine sh -c 'apk add curl && ...'

    One per sandbox is not a luxury — a gvproxy vfkit datagram endpoint learns its peer from the first packet and never re-learns, so a shared socket serves the first VM and silently leaves every later one with no route at all (and all guests boot on the same static address anyway).

    gvproxy:<socket> attaches to a gvproxy you run yourself instead — one sandbox per instance, listening in vfkit mode (libkrun speaks the vfkit datagram protocol — not -listen-qemu), with MVM_GVPROXY_CONTROL pointing at its -listen socket if you want port forwards:

    $ gvproxy -listen unix:///run/gvproxy/control.sock \
        -listen-vfkit unixgram:///run/gvproxy/gvproxy.sock &
    $ export MVM_GVPROXY_CONTROL=/run/gvproxy/control.sock
    $ mvm run --net gvproxy:/run/gvproxy/gvproxy.sock alpine ...

    Port mappings are registered through gvproxy's HTTP control API. On Linux, gvproxy <v0.8.9 does not implement vfkit unixgram sockets and exits with unsupported 'unixgram' scheme; use a newer build that includes Unix vfkit transport support (or build the upstream project).

    The guest is configured automatically (192.168.127.2/24, gw/DNS .1). Throughput is modest (userspace TCP/IP) — fine for package installs and API calls.

  • tap:<dev>: attach to an existing TAP device for near-native performance (Linux-only); you own the plumbing (and the guest needs its own IP config — mvm does no addressing):

    $ sudo ip tuntap add dev mvmtap0 mode tap && sudo ip link set mvmtap0 up
    $ mvm run --net tap:mvmtap0 alpine ...

• Without userns mode (no subuid ranges / newuidmap), guest chown over virtiofs is limited to host credentials.
• Guest chowns land on subuids on the host; clean up sandbox state through mvm rm (the daemon), not by deleting the data dir by hand.
• Guests are same-arch as the host (x86_64 on Linux/KVM, arm64 on Apple Silicon). On macOS additionally: no userns mode, storage is copy (rootfs wiped on each start), and no tap: profile.