Conversation
Threads an optional fwmark from the endpoint builder down through the IP transports to netwatch's bind_full_with_mark, so the underlay UDP sockets can be policy-routed with an ip rule (e.g. to keep iroh's own traffic off a full-tunnel default route). None leaves the sockets unmarked; SO_MARK is applied on Linux and Android and is a no-op elsewhere. Requires a netwatch release exposing UdpSocket::bind_full_with_mark (n0-computer/net-tools#179).
| /// Sets an fwmark on the underlay UDP sockets (Linux `SO_MARK`). | ||
| /// | ||
| /// The mark is applied to every IP transport socket the endpoint binds, and | ||
| /// reapplied on rebind. It lets you policy-route iroh's own traffic with an | ||
| /// `ip rule`, for example to keep it off a full-tunnel default route so the | ||
| /// tunnel does not swallow the packets carrying it. No-op on non-Linux | ||
| /// platforms. | ||
| pub fn socket_mark(mut self, mark: u32) -> Self { | ||
| self.socket_mark = Some(mark); | ||
| self | ||
| } |
There was a problem hiding this comment.
I wonder if the BindOpts is not a more suitable place for this. It would mean you have to do .clear_ip_transports().bind_addr_with_opts(...) to use this. But on the other hand you get to control it per socket that is bound, which seems like it would be useful.
There was a problem hiding this comment.
Did you mean to resolve this without a comment? I currently still think this would be the preferred way to expose this in iroh, so would like to hear other opinions or different tradeoffs that I may not have thought of.
There was a problem hiding this comment.
because BindOpts is used only for the QUIC socket bind, but we need the fwmark to bypass all of the sockets we open, including the tcp relay
bind_full_with_mark was reshaped into bind_with(BindOptions) during review; use the form that actually merged.
|
Don't forget to open this PR if it is ready for review. |
|
do we need to wait for a new release in net-tools? |
Feel free to point this PR to a git ref of the main branch of net-tools using a patch section in the workspace Cargo.toml. We'll then resolve that at release time. We're aiming for a release on Monday, so this can possibly still make it. |
The relay connection is dialed with a plain TcpStream::connect, so a caller that needs iroh's traffic to leave by a particular path has no way to reach that socket. A VPN carrying the relay connection inside its own tunnel is the case that matters: the connection dies as soon as the tunnel comes up. Take an optional hook and run it on the socket before it connects. The socket is handed out as a SocketRef, which implements AsFd and AsSocket, so a caller can set options with the socket crate of its choice, and the family comes along because the options that matter are per-family.
SO_MARK only answers for Linux. Apple platforms have no fwmark, and the way to keep a socket off a tunnel route there is to pin it to an interface with IP_BOUND_IF, which is a different call on a different value. Modelling each of those as its own builder option means a new option every time a platform needs one, so hand out the socket instead. The hook runs on every socket the endpoint opens, the UDP transport sockets and the relay connection alike, before bind or connect and again on every rebind, so a hook that reads the current default route re-reads it when the route changes. An error fails the bind rather than leaving a socket that silently missed its configuration. The UDP half needs the BindOptions hook from n0-computer/net-tools#182, which is not released yet, hence the patch section on the workspace.
# Conflicts: # Cargo.lock # iroh/src/socket/transports.rs
Adds an opt-in fwmark on the endpoint's underlay UDP sockets so callers can policy-route iroh's own traffic, for example to keep it off a full-tunnel
0.0.0.0/0default route (the standard WireGuard/TailscaleSO_MARK+ip ruleapproach) so the tunnel does not swallow the packets carrying it.What
Builder::socket_mark(u32), plumbed throughsocket::Options->Transports::bind->IpTransports::bind->IpTransport::bind->netwatch::UdpSocket::bind_withwithBindOptions::set_mark.None): no behaviour change unless a mark is set.SO_MARKon Linux/Android (a no-op elsewhere) and is reapplied on rebind, since netwatch stores it in the socket state.Draft: blocked on a netwatch release
The netwatch side (n0-computer/net-tools#179) is merged on their main as
UdpSocket::bind_with(BindOptions), and this branch now targets that merged API. It still cannot build against a published netwatch, since no release carries it yet. Once one is tagged I will bump iroh's netwatch requirement and mark this ready for review. CI is expected to be red until then.Happy to adjust the API shape (naming, or a transport-level option instead of endpoint-level) to taste.