Skip to content

Latest commit

 

History

History
230 lines (190 loc) · 11.5 KB

README.md

File metadata and controls

230 lines (190 loc) · 11.5 KB

mesh-vpn-rs

An open-source, decentralized, user-friendly mesh VPN in Rust: a single coordinator binary for the control plane, a single daemon binary for the data plane, and a CLI/TUI for humans. Think "a self-hosted Tailscale you can actually read all the code of."

This repository is a working reference implementation of the full project specification it was built from -- real coordinator/daemon binaries you can run, register nodes against, and watch establish real WireGuard (boringtun) handshakes, with a couple of clearly-documented simplifications (see Scope & Simplifications).

Toolchain notes

This was developed and fully compiled/tested against rustc 1.75 (the version available in the sandbox this was built in). A handful of transitive dependencies in Cargo.lock are pinned to slightly older versions than cargo update would otherwise pick, purely because their newer releases require edition2024 (stable since Rust 1.85). If you're on a current stable Rust (which most people are), you can safely rm Cargo.lock && cargo update to pick up newer transitive dependency versions -- nothing in this codebase depends on old dependency behavior, the pins exist solely for MSRV compatibility with the older toolchain used here.

Quickstart

cargo build --workspace --release
# binaries land in target/release/{coordinatord,mesh-daemon,meshctl}

1. Stand up the coordinator (on a VPS, homelab box, or Raspberry Pi):

coordinatord init --network-cidr 100.64.0.0/10 --bind-addr 0.0.0.0:8443
# -> writes coordinator.toml and prints an admin_token, keep it secret
coordinatord run --config coordinator.toml

2. Mint an invite for a new node (from any machine that can reach the coordinator's admin API):

meshctl invite --coordinator http://coordinator-host:8443 \
    --admin-token <token-from-step-1> --tag tag:servers
# -> prints a single-use invite token

3. Join a node to the mesh (NFR5: three commands, start to finish):

meshctl init --hostname nas --coordinator http://coordinator-host:8443 \
    --invite <token-from-step-2> --tag tag:servers
meshctl up --config mesh.toml
meshctl status --config mesh.toml   # or: meshctl tui --config mesh.toml

Repeat step 3 on more machines (each with its own invite token from step 2) and they'll show up in each other's meshctl status / meshctl tui output, resolve each other as <hostname>.mesh via MagicDNS, and -- on the same LAN -- find each other directly via mDNS.

See examples/coordinator.toml, examples/mesh.server.toml, and examples/mesh.client.toml for annotated configs, including a sample tag-based ACL policy.

Repository layout

mesh-vpn-rs/
├── Cargo.toml                 # workspace definition
├── crates/
│   ├── mesh-core/             # shared types, keys, config, ACL engine (no I/O)
│   ├── mesh-coordinator/      # control plane: IPAM, node/invite store, HTTP API
│   │   └── bin: coordinatord
│   ├── mesh-daemon/           # data plane: WireGuard engine, STUN, MagicDNS, mDNS
│   │   └── bin: mesh-daemon
│   └── meshctl/                # CLI + TUI dashboard
│       └── bin: meshctl
├── docs/architecture.md       # deeper design notes + sequence diagrams (text)
└── examples/                  # annotated coordinator.toml / mesh.toml

This deviates slightly from a single do-everything binary (section 1 of the spec's "batteries included" framing) in one respect: coordinatord, mesh-daemon, and meshctl are three small binaries instead of one binary with subcommands. In practice this maps cleanly onto the spec's own description of the roles ("coordinator, edge node, or both") and keeps the control plane, data plane, and CLI/TUI as independently testable, minimal crates -- but it's a deliberate deviation worth flagging up front.

What's real vs simplified

Every item below is implemented with working code and (where it makes sense) unit tests -- this is not a mockup. A handful of pieces are intentionally scoped down from a full production implementation; each is called out explicitly rather than silently glossed over.

Spec requirement Status
FR1 Node registration via invite token Real. POST /v1/register, single-use tokens, persisted.
FR2 IP allocation Real. CIDR-pool IPAM, restored from persisted state on restart. IPv4 only (see below).
FR3 Key distribution Real. Every peer's WireGuard public key ships in register/heartbeat responses.
FR4 NAT traversal Partial. Real hand-rolled RFC 5389 STUN client discovers public endpoints, shared via the coordinator. Full ICE candidate pairing/TURN relay is not implemented -- see below.
FR5 MagicDNS Real. Hand-rolled UDP DNS responder resolves <hostname>.mesh from the live peer table.
FR6 Declarative ACLs Real. Tag-based allow-list engine, default-deny once any rule exists, unit tested.
FR7 Traffic encryption Real. boringtun's Noise_IK + ChaCha20-Poly1305 engine, driven over real UDP sockets between real processes (see smoke test below).
FR8 TUI monitoring Real. ratatui dashboard polling a local status API; also a one-shot meshctl status table.
FR9 Local discovery Real. mdns-sd-based advertise/browse; local peers are marked Direct and preferred over WAN paths.
NFR1 Performance Not benchmarked (no throughput target was verified against real kernel WireGuard). Coordinator state is an in-memory map with JSON snapshotting, fine well past 1000 nodes on RAM; wall-clock request latency wasn't profiled.
NFR2 Security (mTLS) Simplified. Control-plane API is plain HTTP + bearer/invite tokens; see Security Notes. Data-plane traffic is fully encrypted regardless (FR7).
NFR3 Reliability Real. Coordinator state persisted to disk (atomic write+rename); daemon heartbeats with peer-list refresh; offline detection via heartbeat staleness.
NFR4 Portability Builds on Linux (tested in CI-like sandbox: Ubuntu 24.04, rustc 1.75). macOS/Windows should build (no unsafe/platform-specific code beyond what boringtun/mdns-sd/ratatui/crossterm already handle) but weren't tested here.
NFR5 Usability Real. meshctl init → meshctl up → meshctl status is the full join flow.
NFR6 Resource usage Not profiled.

The one deliberate architectural simplification: no OS TUN device

mesh-daemon's WireGuard engine (crates/mesh-daemon/src/wg.rs) implements the real WireGuard protocol: per-peer Noise_IK handshakes and ChaCha20-Poly1305 transport encryption via boringtun, driven over real UDP sockets, with handshake/keepalive timers ticked on a real interval. What it does not do is bind an OS-level TUN network interface to actually route decrypted IP packets into the kernel's network stack -- that requires CAP_NET_ADMIN (root) on every target platform, which most CI runners, containers, and sandboxes don't grant (including the one this code was developed and compiled in -- see Verifying it actually works for what was verified end-to-end).

Rather than ship untested platform-specific ioctl/utun code behind a false sense of completeness, decrypted packets are handed to a SimulatedTun that counts bytes and logs. The seam is a single trait:

#[async_trait]
pub trait TunInterface: Send + Sync {
    async fn recv(&self) -> std::io::Result<Vec<u8>>;
    async fn send(&self, packet: &[u8]) -> std::io::Result<()>;
}

Wiring in a real device (e.g. via the tun or tokio-tun crates) on a machine/container with CAP_NET_ADMIN is the natural next step -- see wg.rs module docs for a sketch.

Other noted simplifications

  • IPv4 only. PeerInfo::mesh_ip is typed IpAddr so IPv6 can be layered into the wire protocol later, but mesh-coordinator::ipam only implements an IPv4 pool today.
  • STUN uses its own socket, not the WireGuard data socket, so the discovered public mapping may not exactly match what peers need on symmetric NATs. Sharing one socket for both is a small, well-scoped follow-up (see stun_loop in mesh-daemon/src/main.rs).
  • No TURN-style relay data path yet. ConnectionState::Relay exists in the type system and the coordinator model, but only direct paths (WAN via STUN-assisted hole-punching, or LAN via mDNS) are wired up end-to-end.
  • No MagicDNS OS integration. The resolver itself works (see below), but hooking it into /etc/resolver (macOS) or resolvectl domain routing (Linux) so ping laptop.mesh works without manually pointing a client at port 5353 is left as a packaging step.

Verifying it actually works

This isn't just "it compiles" -- the full stack was run end-to-end during development:

$ coordinatord init --network-cidr 100.64.0.0/24 --bind-addr 127.0.0.1:8443
$ coordinatord run --config coordinator.toml &

$ meshctl invite --coordinator http://127.0.0.1:8443 --admin-token <...> --tag tag:servers
Invite token: mesh-invite-...

$ meshctl init --hostname node-a --coordinator http://127.0.0.1:8443 --invite <...> --tag tag:servers
$ mesh-daemon --config mesh.toml &

Observed, for two nodes registered this way on the same machine (different ports):

  • Coordinator correctly allocated sequential IPs (100.64.0.1, .2, .3, skipping a stale/offline registration -- also confirming heartbeat staleness detection works) from the CIDR pool.
  • Both daemons found each other via mDNS on the loopback/local network and marked each other Direct.
  • Both daemons initialized a real boringtun WireGuard session for each other (WireGuard session initialized peer=... hostname=... in the logs).
  • MagicDNS correctly resolved node-b.mesh → 100.64.0.2 over raw UDP DNS queries, and correctly returned NXDOMAIN for an unknown name.
  • meshctl status rendered a live table of peers, tags, and connection state for both nodes.
  • The full cargo test --workspace suite (11 tests: ACL evaluation, IPAM allocation/release, key round-tripping, STUN packet parsing, MagicDNS query/response encoding) passes, as does cargo clippy --workspace with zero warnings.

Security notes

  • mTLS (NFR2): not implemented in this build. The coordinator's HTTP API authenticates node registration via single-use invite tokens and admin actions via a static bearer token, but does not itself terminate TLS. Run it behind a TLS-terminating reverse proxy (Caddy, nginx, Traefik) for anything beyond a trusted LAN, or extend mesh-coordinator/src/main.rs to use axum-server's rustls acceptor for mTLS directly. Mesh data-plane traffic is unaffected -- WireGuard encrypts every packet between nodes regardless of how the control plane is secured.
  • Admin token: treat like a root password; it's generated by coordinatord init and grants invite minting + node removal.
  • Private keys never leave the node they're generated on; only public keys are sent to the coordinator (FR3).

Future work

  • Wire a real TunInterface (see above) behind CAP_NET_ADMIN.
  • Full ICE-style connectivity checks and a relay/TURN data path for ConnectionState::Relay.
  • IPv6 pool support in IPAM.
  • mTLS between nodes and the coordinator.
  • OS resolver integration for MagicDNS.
  • Cross-platform packaging (.deb, .rpm, Homebrew, Windows installer) -- none of this exists yet; today it's cargo build --release.

License

MIT OR Apache-2.0, matching most of the Rust ecosystem this project builds on.