Skip to content

nstun: run the network loop in a separate, confined helper process - #390

Open
aravindor wants to merge 1 commit into
google:masterfrom
aravindor:nstun-helper-process
Open

aravindor wants to merge 1 commit into
google:masterfrom
aravindor:nstun-helper-process

Conversation

@aravindor

Copy link
Copy Markdown

CONTRIBUTING asks for larger changes to be discussed first, so please treat this PR as the proposal. I'm happy to change its shape (fork or re-exec, kafel or hand-written BPF, on by default or opt-in behind a config field) before you spend time on a detailed review.

Why

nstun parses every frame the jailed process writes to its TUN device, plus SOCKS5 and HTTP CONNECT proxy replies. Until now that loop ran as a std::thread inside the supervisor, which keeps the privileges nsjail was started with and has no seccomp filter. A memory-safety bug anywhere in that path meant code execution in the supervisor, and in listen mode control over every other jail it runs. This change keeps the parsers exactly as they are, but moves them into a separate process that can't do much if one of them is ever exploited.

What changes

  • nstun_init_parent() does all of its setup as before (TUN fd, rules, HOST_TO_GUEST listeners, epoll), compiles a small kafel policy, and then fork()s a helper instead of starting a thread. The supervisor closes its copies of the TUN fd, the epoll fd and the listeners, and keeps only the helper's pid.
  • The helper, before entering the unchanged networkLoop():
    • sets PR_SET_PDEATHSIG(SIGKILL) and checks the supervisor is still its parent;
    • reseeds util::rnd64() (used for TCP initial sequence numbers; its state is thread-local, so a forked helper could otherwise inherit the forking thread's state) and calls tzset(), so neither needs to open a file later;
    • closes every descriptor it doesn't use (other jails' fds, the supervisor's listen socket, and so on) and points stdin/stdout/stderr at /dev/null;
    • drops all capabilities (bounding set best effort, ambient set, then capset() to zero) and sets PR_SET_NO_NEW_PRIVS;
    • installs the seccomp policy with SECCOMP_RET_KILL_PROCESS as the default action.
  • The policy allows only what the loop uses: read/write/writev/close, epoll, socket() limited to AF_INET/AF_INET6, connect/bind/accept4/shutdown, sendto/recvfrom/recvmmsg, set/getsockopt, getsockname, memory management with PROT_EXEC refused in mmap, clocks, getpid/gettid, getrandom, rt_sigreturn and exit. No execve, open, ptrace, clone, ioctl, mount, setns or kill.
  • nstun_destroy_parent() kills and reaps the helper, the same way pasta's helper is handled, so the stop eventfd is no longer needed and is removed.
  • subproc.cc: if the reaper sees a helper exit, it marks it reaped. If the helper was killed by a signal (a crash or a policy violation), it logs a warning and kills the jail, since the jail has lost its network and something went wrong in the parser. If the helper simply returns (for example on TUN EOF), the jail is left alone, which matches the old behaviour of the thread returning.

There is no change on the data path: frames still go straight from the TUN fd into the same loop.

Testing

All local, on Linux 6.18, as an unprivileged user with user namespaces:

  • gcc and clang builds with the project's -Werror flags; clang-format clean for the changed hunks.
  • tests/nstun_buffer_budget_test, nstun_policy_test, nstun_ip_test, nstun_udp_peer_test: pass. make test-cmdline: pass.
  • tests/nstun-listen-lifecycle-authoritative.sh with COUNT=10: PASS. Supervisor tasks, threads and fds return to baseline after every connection.
  • tests/nstun.cfg with a jail that does TCP (curl to a local HTTP server), UDP (echo to a local socat) and ICMP (ping) through nstun: all work. While it runs, the helper shows Seccomp: 2, NoNewPrivs: 1, CapEff: 0, and holds only /dev/null on 0-2, the log fd, the TUN fd, the epoll fd and its own flow sockets.
  • HOST_TO_GUEST TCP redirect, and guest-to-host TCP through ENCAP_SOCKS5 and ENCAP_CONNECT (local test proxy): work, same as master.
  • Negative test, with a temporary build that calls getuid() in the helper after the filter is installed: the kernel kills the helper with SIGSYS (audit record syscall=102), nsjail logs the warning and kills the jail, in both standalone and listen mode.
  • kill -9 of the supervisor: the helper exits through PDEATHSIG, nothing is left behind. Time-limit kill and SIGINT to the whole process group: everything is torn down.

Notes for review

  • fork() happens in the supervisor while unotify threads may be running. The helper only uses malloc (made fork-safe by glibc) and the logging code, which takes no locks. If you'd rather avoid the question entirely, I can switch to re-exec of /proc/self/exe with an internal flag and pass the context over a socket.
  • The policy is compiled once per jail, at nstun setup. That's small next to clone and namespace setup, but it could be cached if you prefer.
  • A compromised helper could still ignore the nstun rules and connect anywhere from the host network namespace, because seccomp can't inspect connect() addresses. It can no longer exec, open files, use the supervisor's capabilities, or touch other jails.

The nstun loop parses every frame the jailed process writes to its TUN
device, plus SOCKS5 and HTTP CONNECT proxy replies. It ran as a thread
inside the supervisor, which keeps the privileges nsjail was started
with and has no seccomp filter, so a memory-safety bug in that path
meant code execution in the supervisor.

nstun_init_parent() now does its setup as before, then fork()s a helper
instead of starting a thread. Before entering the unchanged
networkLoop(), the helper sets PR_SET_PDEATHSIG, reseeds util::rnd64(),
closes every fd it doesn't use, drops all capabilities, sets
PR_SET_NO_NEW_PRIVS and installs a kafel seccomp allowlist with
KILL_PROCESS as the default action. The supervisor closes its copies of
the TUN fd, epoll fd and listeners and keeps only the helper pid.

nstun_destroy_parent() kills and reaps the helper, so the stop eventfd
is removed. If the reaper sees the helper killed by a signal, it logs a
warning and kills the jail.
@google-cla

google-cla Bot commented Nov 12, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant