Repository navigation
Conversation
The nstun loop parses every frame the jailed process writes to its TUN device, plus SOCKS5 and HTTP CONNECT proxy replies. It ran as a thread inside the supervisor, which keeps the privileges nsjail was started with and has no seccomp filter, so a memory-safety bug in that path meant code execution in the supervisor. nstun_init_parent() now does its setup as before, then fork()s a helper instead of starting a thread. Before entering the unchanged networkLoop(), the helper sets PR_SET_PDEATHSIG, reseeds util::rnd64(), closes every fd it doesn't use, drops all capabilities, sets PR_SET_NO_NEW_PRIVS and installs a kafel seccomp allowlist with KILL_PROCESS as the default action. The supervisor closes its copies of the TUN fd, epoll fd and listeners and keeps only the helper pid. nstun_destroy_parent() kills and reaps the helper, so the stop eventfd is removed. If the reaper sees the helper killed by a signal, it logs a warning and kills the jail.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CONTRIBUTING asks for larger changes to be discussed first, so please treat this PR as the proposal. I'm happy to change its shape (fork or re-exec, kafel or hand-written BPF, on by default or opt-in behind a config field) before you spend time on a detailed review.
Why
nstun parses every frame the jailed process writes to its TUN device, plus SOCKS5 and HTTP CONNECT proxy replies. Until now that loop ran as a
std::threadinside the supervisor, which keeps the privileges nsjail was started with and has no seccomp filter. A memory-safety bug anywhere in that path meant code execution in the supervisor, and in listen mode control over every other jail it runs. This change keeps the parsers exactly as they are, but moves them into a separate process that can't do much if one of them is ever exploited.What changes
nstun_init_parent()does all of its setup as before (TUN fd, rules, HOST_TO_GUEST listeners, epoll), compiles a small kafel policy, and thenfork()s a helper instead of starting a thread. The supervisor closes its copies of the TUN fd, the epoll fd and the listeners, and keeps only the helper's pid.networkLoop():PR_SET_PDEATHSIG(SIGKILL)and checks the supervisor is still its parent;util::rnd64()(used for TCP initial sequence numbers; its state is thread-local, so a forked helper could otherwise inherit the forking thread's state) and callstzset(), so neither needs to open a file later;/dev/null;capset()to zero) and setsPR_SET_NO_NEW_PRIVS;SECCOMP_RET_KILL_PROCESSas the default action.socket()limited toAF_INET/AF_INET6, connect/bind/accept4/shutdown, sendto/recvfrom/recvmmsg, set/getsockopt, getsockname, memory management withPROT_EXECrefused in mmap, clocks, getpid/gettid, getrandom, rt_sigreturn and exit. No execve, open, ptrace, clone, ioctl, mount, setns or kill.nstun_destroy_parent()kills and reaps the helper, the same way pasta's helper is handled, so the stop eventfd is no longer needed and is removed.subproc.cc: if the reaper sees a helper exit, it marks it reaped. If the helper was killed by a signal (a crash or a policy violation), it logs a warning and kills the jail, since the jail has lost its network and something went wrong in the parser. If the helper simply returns (for example on TUN EOF), the jail is left alone, which matches the old behaviour of the thread returning.There is no change on the data path: frames still go straight from the TUN fd into the same loop.
Testing
All local, on Linux 6.18, as an unprivileged user with user namespaces:
-Werrorflags; clang-format clean for the changed hunks.tests/nstun_buffer_budget_test,nstun_policy_test,nstun_ip_test,nstun_udp_peer_test: pass.make test-cmdline: pass.tests/nstun-listen-lifecycle-authoritative.shwith COUNT=10: PASS. Supervisor tasks, threads and fds return to baseline after every connection.tests/nstun.cfgwith a jail that does TCP (curl to a local HTTP server), UDP (echo to a local socat) and ICMP (ping) through nstun: all work. While it runs, the helper showsSeccomp: 2,NoNewPrivs: 1,CapEff: 0, and holds only /dev/null on 0-2, the log fd, the TUN fd, the epoll fd and its own flow sockets.ENCAP_SOCKS5andENCAP_CONNECT(local test proxy): work, same as master.getuid()in the helper after the filter is installed: the kernel kills the helper with SIGSYS (audit recordsyscall=102), nsjail logs the warning and kills the jail, in both standalone and listen mode.kill -9of the supervisor: the helper exits through PDEATHSIG, nothing is left behind. Time-limit kill and SIGINT to the whole process group: everything is torn down.Notes for review
fork()happens in the supervisor while unotify threads may be running. The helper only uses malloc (made fork-safe by glibc) and the logging code, which takes no locks. If you'd rather avoid the question entirely, I can switch to re-exec of/proc/self/exewith an internal flag and pass the context over a socket.connect()addresses. It can no longer exec, open files, use the supervisor's capabilities, or touch other jails.