jibiJibi's personal blogZola2026-07-28T00:00:00+00:00https://jibi.io/atom.xmlTCP local bypass2026-07-28T00:00:00+00:002026-07-28T00:00:00+00:00
Unknown
https://jibi.io/blog/tcp-local-bypass/<p>While running some local TCP services that were bandwidth bound, I started
wondering if there was a way to squeeze some extra Gbps out of my box.
TCP and the <code>socket()</code> API are great: they provide an abstraction for a reliable
stream of bytes that works both with 2 local endpoints on the same host as well
as with 2 endpoints talking to each other through multiple hops over undersea
cables. But that abstraction has some costs that for local connections perhaps
can be avoided.</p>
<p>So what options do we have to save on those costs?</p>
<p>Shared memory is a good first stop: no network stack overhead and potentially
the lowest possible latency if you synchronize your processes over a spinlock
(otherwise eventfd or similar will work great as well).
For fun I quickly <del>vib</del>-<em>sketched</em> an implementation for a <a rel="external" href="https://github.com/jibi/shm-rpc-rs">SHM RPC
lib</a>. But it didn't take me too long to
discard that option: I wanted something transparent, that keeps using the
familiar <code>socket</code> API and addressing format, and that doesn't require me to
patch and recompile my services.</p>
<p><code>LD_PRELOAD</code> could be another option: we hook into <code>socket()</code> / <code>bind()</code> /
<code>connect()</code> / <code>accept()</code> / <code>read()</code> / <code>write()</code> / <code>recv()</code> / <code>send()</code> / <code>poll()</code>
/ <code>epoll()</code> / <code>select()</code> / <code>close()</code> (did I forget something? probably) and
transparently pipe the TCP payload data into the SHM implementation. But
unfortunately this doesn't fly for static binaries, and requires passing the
<code>LD_PRELOAD</code> env variable and the <code>.so</code> object around, as well as granting
access to the shared memory region. A bit easier for non containerized
workloads, more work for containers. Anyway, discarded for the same reason as
before: needs to be transparent, should not require me to mess too much with my
services, and ideally the kernel should keep handling as much as possible of the
connection lifecycle. Only the data transfer part should be bypassed.</p>
<p>The next option to consider would then be kernel bypass, and as I have played
already in the past with different technologies in this space I knew in advance
this could help me deliver some extra Gbps. But since here I'm only interested
in <em>local</em> connections, it seems like I don't even need to push my data down to
the NIC? Maybe there's something simpler we can do here?</p>
<h2 id="rough-idea">Rough idea</h2>
<p>Let's start from a very hand-wavy kind of idea: let's move in kernel space, take
<code>tcp_sendmsg()</code> and "pipe" it, whatever that means in practice, straight into
<code>tcp_recvmsg()</code>.</p>
<p>(Why not hook higher up in the call stack? <code>sys_{sendto,recvfrom}()</code>,
<code>sock_{send,recv}()</code> and <code>inet_{send,recv}msg()</code> are mostly thin
wrappers/dispatchers, so hooking <code>tcp_{send,recv}msg()</code> lets us ignore for free
all the non TCP traffic).</p>
<p>What comes into one side comes out straight onto the other end. No network stack
overhead, no TCP/IP headers, no SKBs, nothing: just the TCP payload copied from
the sender to the receiver. And since in a previous post we learnt <a rel="external" href="https://jibi.io/blog/xdp-rust-lkm/">how to write
Rust LKM modules</a>, we'll resume from there,
and reuse all the stuff we already know.</p>
<p>But first, what's really all this overhead about? We can once again turn to
ftrace to get a sense of what's going on by using <code>trace-cmd</code> to set up tracing
and collect the events.</p>
<p>Here's what happens when you send one byte from a <code>nc</code> client to a <code>nc</code> server
running on the same box (note that this is just the data transmission part, the
connection is already established). We'll start from <code>tcp_sendmsg()</code> and go
through the entire network stack calltrace:</p>
<pre class="scroll-code"><code>tcp_sendmsg() {
lock_sock_nested() {
__cond_resched()
_raw_spin_lock_bh()
_raw_spin_unlock_bh() {
__local_bh_enable_ip()
}
}
tcp_sendmsg_locked() {
tcp_rate_check_app_limited()
tcp_send_mss() {
tcp_current_mss() {
ipv4_mtu() {
__rcu_read_lock()
__rcu_read_unlock()
}
tcp_established_options()
}
}
tcp_stream_memory_free()
tcp_stream_alloc_skb() {
__alloc_skb() {
kmem_cache_alloc_node_noprof() {
__cond_resched()
__alloc_tagging_slab_alloc_hook()
}
kmalloc_reserve() {
kmem_cache_alloc_node_noprof() {
__cond_resched()
__alloc_tagging_slab_alloc_hook()
}
}
__build_skb_around()
}
sk_forced_mem_schedule() {
mem_cgroup_sk_charge() {
try_charge_memcg()
mod_memcg_state() {
css_rstat_updated()
}
}
}
}
tcp_skb_entail() {
tcp_chrono_start()
tcp_cwnd_restart() {
tcp_init_cwnd()
cubictcp_cwnd_event()
}
}
sk_page_frag_refill() {
skb_page_frag_refill() {
alloc_pages_noprof() {
alloc_pages_mpol() {
policy_nodemask()
__alloc_frozen_pages_noprof() {
__next_zones_zonelist()
get_page_from_freelist() {
__zone_watermark_ok()
_raw_spin_trylock()
_raw_spin_unlock()
__pgalloc_tag_add() {
static_key_count()
page_ext_get() {
__rcu_read_lock()
}
static_key_count()
static_key_count()
page_ext_put() {
__rcu_read_unlock()
}
}
prep_compound_page()
}
}
arch_irq_work_raise() {
x2apic_send_IPI_self() {
irq_enter_rcu() {
irqtime_account_irq()
}
__sysvec_irq_work() {
__wake_up() {
_raw_spin_lock_irqsave()
__wake_up_common()
_raw_spin_unlock_irqrestore()
}
_raw_spin_lock()
_raw_spin_unlock()
__wake_up() {
_raw_spin_lock_irqsave()
__wake_up_common()
_raw_spin_unlock_irqrestore()
}
}
irq_exit_rcu() {
irqtime_account_irq()
sched_core_idle_cpu()
}
}
}
}
}
}
}
tcp_wmem_schedule()
__check_object_size() {
check_stack_object()
is_vmalloc_addr()
__virt_addr_valid()
}
tcp_push() {
__tcp_push_pending_frames() {
tcp_write_xmit() {
ktime_get()
tcp_tso_segs()
tcp_small_queue_check.isra.0()
__tcp_transmit_skb() {
skb_clone() {
__copy_skb_header()
}
tcp_established_options()
skb_push()
__tcp_select_window()
tcp_options_write.isra.0()
bpf_skops_write_hdr_opt.isra.0()
tcp_v4_send_check()
cubictcp_cwnd_event()
ip_queue_xmit() {
__ip_queue_xmit() {
__rcu_read_lock()
__sk_dst_check() {
ipv4_dst_check() {
__rcu_read_lock()
__rcu_read_unlock()
}
}
skb_push()
__rcu_read_lock()
__rcu_read_unlock()
ip_local_out() {
__ip_local_out() {
ip_send_check()
__rcu_read_lock() {
}
nf_hook_slow() {
ipv4_conntrack_defrag()
ipv4_conntrack_local() {
nf_conntrack_in() {
ipv4_get_l4proto()
nf_ct_get_tuple()
hash_conntrack_raw()
__nf_conntrack_find_get()
nf_conntrack_tcp_packet() {
_raw_spin_lock_bh()
nf_ct_seq_offset()
_raw_spin_unlock_bh() {
__local_bh_enable_ip()
}
__nf_ct_refresh_acct()
}
}
}
nf_nat_ipv4_local_fn() {
nf_nat_inet_fn()
}
nft_do_chain_ipv4() {
nft_do_chain() {
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_update_chain_stats() {
__local_bh_enable_ip()
}
}
}
}
__rcu_read_unlock()
}
ip_output() {
__rcu_read_lock()
__rcu_read_lock()
nf_hook_slow() {
nft_do_chain_ipv4() {
nft_do_chain() {
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_update_chain_stats() {
__local_bh_enable_ip()
}
}
}
nf_nat_ipv4_out() {
nf_nat_inet_fn()
}
nf_confirm()
}
__rcu_read_unlock()
ip_finish_output() {
__cgroup_bpf_run_filter_skb() {
__rcu_read_lock() {
}
__rcu_read_unlock()
}
__ip_finish_output() {
__rcu_read_lock()
__rcu_read_unlock()
ip_finish_output2() {
__rcu_read_lock()
__dev_queue_xmit() {
qdisc_pkt_len_segs_init()
netdev_core_pick_tx()
validate_xmit_skb() {
netif_skb_features() {
skb_network_protocol()
}
skb_csum_hwoffload_help()
validate_xmit_xfrm()
}
dev_hard_start_xmit() {
loopback_xmit() {
tcp_wfree() {
sk_free()
}
eth_type_trans()
__netif_rx() {
netif_rx_internal() {
enqueue_to_backlog() {
_raw_spin_lock_irqsave()
_raw_spin_unlock_irqrestore()
}
}
}
}
}
__local_bh_enable_ip() {
do_softirq.part.0() {
__do_softirq() {
handle_softirqs() {
irqtime_account_irq()
net_rx_action() {
__usecs_to_jiffies()
skb_defer_free_flush()
__napi_poll() {
process_backlog() {
_raw_spin_lock_irq()
_raw_spin_unlock_irq()
__rcu_read_lock()
__netif_receive_skb() {
__netif_receive_skb_one_core() {
__netif_receive_skb_core.constprop.0()
ip_rcv() {
ip_rcv_core()
__rcu_read_lock()
nf_hook_slow() {
ip_sabotage_in()
ipv4_conntrack_defrag()
nft_do_chain_ipv4() {
nft_do_chain() {
nft_update_chain_stats() {
__local_bh_enable_ip()
}
}
}
ipv4_conntrack_in() {
nf_conntrack_in()
}
nft_do_chain_ipv4() {
nft_do_chain() {
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_match_eval() {
rpfilter_mt()
}
nft_counter_eval() {
__local_bh_enable_ip() {
}
}
nft_immediate_eval()
nft_update_chain_stats() {
__local_bh_enable_ip()
}
}
}
nf_nat_ipv4_pre_routing() {
nf_nat_inet_fn()
}
}
__rcu_read_unlock()
ip_rcv_finish_core()
ip_local_deliver() {
__rcu_read_lock()
nf_hook_slow() {
nft_do_chain_ipv4() {
nft_do_chain() {
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
}
}
nf_nat_ipv4_local_in() {
nf_nat_inet_fn()
}
nf_confirm()
}
__rcu_read_unlock()
ip_local_deliver_finish() {
__rcu_read_lock()
ip_protocol_deliver_rcu() {
raw_local_deliver() {
__rcu_read_lock()
__rcu_read_unlock()
}
tcp_v4_rcv() {
__inet_lookup_established() {
inet_ehashfn()
}
tcp_inbound_hash() {
tcp_do_parse_auth_options()
}
sk_filter_trim_cap() {
__cgroup_bpf_run_filter_skb() {
__rcu_read_lock()
__rcu_read_unlock()
}
security_sock_rcv_skb() {
bpf_lsm_socket_sock_rcv_skb()
}
__rcu_read_lock()
__rcu_read_unlock()
}
tcp_v4_fill_cb()
_raw_spin_lock()
tcp_v4_do_rcv() {
ipv4_dst_check() {
__rcu_read_lock()
__rcu_read_unlock()
}
tcp_rcv_established() {
tcp_mstamp_refresh() {
ktime_get()
}
tcp_ack() {
ktime_get_seconds()
tcp_in_ack_event()
}
tcp_urg()
tcp_data_queue() {
dst_release()
tcp_try_rmem_schedule() {
__sk_mem_schedule() {
__sk_mem_raise_allocated() {
mem_cgroup_sk_charge() {
try_charge_memcg()
mod_memcg_state() {
css_rstat_updated()
}
}
sk_leave_memory_pressure() {
tcp_leave_memory_pressure()
}
}
}
}
tcp_queue_rcv()
tcp_event_data_recv() {
tcp_measure_rcv_mss()
tcp_data_ecn_check()
}
tcp_data_ready() {
sock_def_readable() {
__rcu_read_lock()
__wake_up_sync_key() {
_raw_spin_lock_irqsave()
__wake_up_common() {
pollwake() {
default_wake_function() {
try_to_wake_up() {
_raw_spin_lock_irqsave()
select_task_rq_fair() {
__rcu_read_lock()
wake_affine()
cpus_share_cache()
__rcu_read_unlock()
}
ttwu_queue_wakelist() {
scx_allow_ttwu_queue()
__smp_call_single_queue() {
call_function_single_prep_ipi()
}
}
_raw_spin_unlock_irqrestore()
}
}
}
pollwake()
}
_raw_spin_unlock_irqrestore()
}
__rcu_read_unlock()
}
}
}
tcp_check_space()
__tcp_ack_snd_check() {
tcp_send_ack() {
__tcp_send_ack.part.0() {
__alloc_skb() {
napi_skb_cache_get()
__local_bh_enable_ip()
kmem_cache_alloc_node_noprof() {
__alloc_tagging_slab_alloc_hook()
}
kmalloc_reserve() {
kmem_cache_alloc_node_noprof() {
__alloc_tagging_slab_alloc_hook()
}
}
__build_skb_around()
}
__tcp_transmit_skb() {
tcp_established_options()
skb_push()
__tcp_select_window()
tcp_options_write.isra.0()
bpf_skops_write_hdr_opt.isra.0()
tcp_v4_send_check()
ip_queue_xmit() {
__ip_queue_xmit() {
__rcu_read_lock()
__sk_dst_check() {
ipv4_dst_check() {
__rcu_read_lock()
__rcu_read_unlock()
}
}
skb_push()
__rcu_read_lock()
__rcu_read_unlock()
ip_local_out() {
__ip_local_out() {
ip_send_check()
__rcu_read_lock()
nf_hook_slow() {
ipv4_conntrack_defrag()
ipv4_conntrack_local() {
nf_conntrack_in() {
ipv4_get_l4proto()
nf_ct_get_tuple()
hash_conntrack_raw()
__nf_conntrack_find_get()
nf_conntrack_tcp_packet() {
_raw_spin_lock_bh()
nf_ct_seq_offset()
_raw_spin_unlock_bh() {
__local_bh_enable_ip()
}
__nf_ct_refresh_acct()
}
}
}
nf_nat_ipv4_local_fn() {
nf_nat_inet_fn() {
}
}
nft_do_chain_ipv4() {
nft_do_chain() {
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_update_chain_stats() {
__local_bh_enable_ip()
}
}
}
}
__rcu_read_unlock()
}
ip_output() {
__rcu_read_lock()
__rcu_read_lock()
nf_hook_slow() {
nft_do_chain_ipv4() {
nft_do_chain() {
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_update_chain_stats() {
__local_bh_enable_ip()
}
}
}
nf_nat_ipv4_out() {
nf_nat_inet_fn()
}
nf_confirm()
}
__rcu_read_unlock()
ip_finish_output() {
__cgroup_bpf_run_filter_skb() {
__rcu_read_lock()
__rcu_read_unlock()
}
__ip_finish_output() {
__rcu_read_lock()
__rcu_read_unlock()
ip_finish_output2() {
__rcu_read_lock()
__dev_queue_xmit() {
qdisc_pkt_len_segs_init()
netdev_core_pick_tx()
validate_xmit_skb() {
netif_skb_features() {
skb_network_protocol()
}
skb_csum_hwoffload_help()
validate_xmit_xfrm()
}
dev_hard_start_xmit() {
loopback_xmit() {
__sock_wfree() {
}
eth_type_trans()
__netif_rx() {
netif_rx_internal() {
enqueue_to_backlog() {
_raw_spin_lock_irqsave()
_raw_spin_unlock_irqrestore()
}
}
}
}
}
__local_bh_enable_ip()
}
__rcu_read_unlock()
}
}
}
__rcu_read_unlock()
}
}
__rcu_read_unlock()
}
}
}
}
}
}
}
}
_raw_spin_unlock()
}
}
__rcu_read_unlock()
}
}
}
}
}
__rcu_read_unlock()
_raw_spin_lock_irq()
_raw_spin_unlock_irq()
__rcu_read_lock()
__netif_receive_skb() {
__netif_receive_skb_one_core() {
__netif_receive_skb_core.constprop.0()
ip_rcv() {
ip_rcv_core()
__rcu_read_lock()
nf_hook_slow() {
ip_sabotage_in()
ipv4_conntrack_defrag()
nft_do_chain_ipv4() {
nft_do_chain() {
nft_update_chain_stats() {
__local_bh_enable_ip()
}
}
}
ipv4_conntrack_in() {
nf_conntrack_in()
}
nft_do_chain_ipv4() {
nft_do_chain() {
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_match_eval() {
rpfilter_mt()
}
nft_counter_eval() {
__local_bh_enable_ip() {
}
}
nft_immediate_eval()
nft_update_chain_stats() {
__local_bh_enable_ip()
}
}
}
nf_nat_ipv4_pre_routing() {
nf_nat_inet_fn()
}
}
__rcu_read_unlock()
ip_rcv_finish_core()
ip_local_deliver() {
__rcu_read_lock()
nf_hook_slow() {
nft_do_chain_ipv4() {
nft_do_chain() {
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_meta_get_eval() {
nft_meta_store_ifname()
}
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
nft_counter_eval() {
__local_bh_enable_ip()
}
nft_immediate_eval()
}
}
nf_nat_ipv4_local_in() {
nf_nat_inet_fn()
}
nf_confirm()
}
__rcu_read_unlock()
ip_local_deliver_finish() {
__rcu_read_lock()
ip_protocol_deliver_rcu() {
raw_local_deliver() {
__rcu_read_lock()
__rcu_read_unlock()
}
tcp_v4_rcv() {
__inet_lookup_established() {
inet_ehashfn()
}
tcp_inbound_hash() {
tcp_do_parse_auth_options()
}
sk_filter_trim_cap() {
__cgroup_bpf_run_filter_skb() {
__rcu_read_lock()
__rcu_read_unlock()
}
security_sock_rcv_skb() {
bpf_lsm_socket_sock_rcv_skb()
}
__rcu_read_lock()
__rcu_read_unlock()
}
tcp_v4_fill_cb()
_raw_spin_lock()
tcp_add_backlog() {
skb_condense()
dst_release()
}
_raw_spin_unlock()
}
}
__rcu_read_unlock()
}
}
}
}
}
__rcu_read_unlock()
_raw_spin_lock_irq()
_raw_spin_unlock_irq()
}
}
skb_defer_free_flush() {
napi_consume_skb() {
skb_release_head_state()
skb_release_data()
kfree_skbmem()
}
}
}
irqtime_account_irq()
}
}
}
}
}
__rcu_read_unlock()
}
}
}
__rcu_read_unlock()
}
}
__rcu_read_unlock()
}
}
tcp_update_skb_after_send()
tcp_rate_skb_sent()
}
tcp_event_new_data_sent() {
tcp_rbtree_insert()
tcp_rearm_rto() {
sk_reset_timer() {
mod_timer() {
lock_timer_base() {
_raw_spin_lock_irqsave()
}
detach_if_pending()
calc_wheel_index()
enqueue_timer()
_raw_spin_unlock_irqrestore()
}
}
}
tcp_check_space()
}
tcp_chrono_stop()
tcp_schedule_loss_probe.part.0() {
jiffies_to_usecs()
__usecs_to_jiffies()
jiffies_to_usecs()
__usecs_to_jiffies()
sk_reset_timer() {
mod_timer()
}
}
}
}
}
}
release_sock() {
_raw_spin_lock_bh()
__release_sock() {
_raw_spin_unlock_bh() {
__local_bh_enable_ip()
}
tcp_v4_do_rcv() {
tcp_rcv_established() {
tcp_mstamp_refresh() {
ktime_get()
}
inet_sk_rx_dst_set()
tcp_validate_incoming()
tcp_ack() {
ktime_get_seconds()
tcp_sync_mss()
tcp_rack_advance()
tcp_rate_skb_delivered()
tcp_ack_tstamp()
__sk_mem_reclaim() {
__sk_mem_reduce_allocated() {
mem_cgroup_sk_uncharge() {
mod_memcg_state() {
css_rstat_updated()
}
refill_stock()
}
}
}
__kfree_skb() {
skb_release_head_state()
skb_release_data() {
skb_free_head() {
kmem_cache_free() {
__memcg_slab_free_hook()
__alloc_tagging_slab_free_hook()
}
}
}
kfree_skbmem() {
kmem_cache_free() {
__memcg_slab_free_hook()
__alloc_tagging_slab_free_hook()
}
}
}
tcp_chrono_stop()
tcp_ack_update_rtt() {
jiffies_to_usecs()
__usecs_to_jiffies()
}
cubictcp_acked()
tcp_rack_update_reo_wnd()
tcp_in_ack_event()
tcp_schedule_loss_probe() {
tcp_schedule_loss_probe.part.0()
}
tcp_rearm_rto()
tcp_newly_delivered()
tcp_rate_gen()
cubictcp_cong_avoid()
tcp_update_pacing_rate()
tcp_xmit_recovery()
}
tcp_urg()
tcp_data_queue() {
__kfree_skb() {
skb_release_head_state()
skb_release_data() {
skb_free_head() {
kmem_cache_free() {
__memcg_slab_free_hook()
__alloc_tagging_slab_free_hook()
}
}
}
kfree_skbmem() {
kmem_cache_free() {
__memcg_slab_free_hook()
__alloc_tagging_slab_free_hook()
}
}
}
}
tcp_check_space()
}
}
_raw_spin_lock_bh()
}
tcp_release_cb()
_raw_spin_unlock_bh() {
__local_bh_enable_ip()
}
}
}
</code></pre>
<p>To my surprise, a single <code>tcp_sendmsg()</code> call on loopback captures the entire
roundtrip. Here's a non exhaustive summary to get a rough idea of what's going
on:</p>
<ol>
<li>
<p>Socket lock acquired: <code>lock_sock_nested()</code> at the top of <code>tcp_sendmsg()</code>. The
sender holds this lock for the entire duration of the roundtrip.</p>
</li>
<li>
<p>Send data: <code>tcp_sendmsg_locked()</code> calculates MSS (<code>tcp_send_mss()</code>),
allocates the SKB (<code>tcp_stream_alloc_skb()</code>), attaches the skb to the socket
(<code>tcp_skb_entail()</code>), refills the page fragment (<code>sk_page_frag_refill()</code>) then
calls <code>tcp_push()</code> -> <code>tcp_write_xmit()</code> -> <code>__tcp_transmit_skb()</code> which builds
the TCP/IP header and calls <code>ip_queue_xmit()</code> which figures out the packet is
local and calls <code>ip_local_out()</code>. The packet then traverses netfilter
(<code>nf_hook_slow()</code>) twice (once at <code>OUTPUT</code>, once at <code>POSTROUTING</code>): conntrack
(<code>nf_conntrack_in()</code>, <code>nf_conntrack_tcp_packet()</code>) updates the connection state,
and nftables (<code>nft_do_chain()</code>) evaluates the ruleset. The packet then reaches
<code>ip_finish_output()</code> -> <code>ip_finish_output2()</code> -> <code>dev_hard_start_xmit()</code> ->
<code>loopback_xmit()</code>, which frees the write-side skb (<code>tcp_wfree()</code>) and enqueues
the packet to the RX backlog via <code>__netif_rx()</code> -> <code>enqueue_to_backlog()</code>.</p>
</li>
<li>
<p>Receive data: <code>__local_bh_enable_ip()</code> after <code>loopback_xmit()</code> triggers
softirq processing inline. <code>net_rx_action()</code> -> <code>process_backlog()</code> ->
<code>ip_rcv()</code> -> netfilter again (<code>PREROUTING</code>, <code>INPUT</code>: conntrack, nftables
ruleset evaluation) -> <code>tcp_v4_rcv()</code> -> <code>tcp_rcv_established()</code>, which inside
processes the ACK fields (<code>tcp_ack()</code>), queues the payload into the receive
buffer (<code>tcp_data_queue()</code>), then <code>tcp_data_ready()</code> -> <code>sock_def_readable()</code> ->
<code>__wake_up_sync_key()</code> -> <code>pollwake()</code> -> <code>try_to_wake_up()</code> wakes the receiver
task via <code>ttwu_queue_wakelist()</code>.</p>
</li>
<li>
<p>Send ACK: still in the same softirq, <code>__tcp_ack_snd_check()</code> ->
<code>tcp_send_ack()</code> -> <code>__tcp_send_ack.part.0()</code> allocates a new SKB and calls
<code>__tcp_transmit_skb()</code> again. The ACK traverses the full IP/netfilter stack a
second time, hits <code>loopback_xmit()</code> again, and is enqueued to the RX backlog via
<code>__netif_rx()</code>. A nested softirq picks it up: <code>tcp_v4_rcv()</code> ->
<code>tcp_add_backlog()</code>.</p>
</li>
<li>
<p>Socket lock released: <code>release_sock()</code> at the bottom of <code>tcp_sendmsg()</code>.
<code>__release_sock()</code> drains the backlog: <code>tcp_v4_do_rcv()</code> ->
<code>tcp_rcv_established()</code> -> <code>tcp_ack()</code> processes the ACK, updating RTT
(<code>tcp_ack_update_rtt()</code>), congestion window, pacing rate, and freeing the
original SKB. Only now does the sender learn the data was received.</p>
</li>
</ol>
<p>That's quite a lot of stuff going on for sending one byte. And the 2 peers are
running on the host namespace. Surely for real workloads with lots of traffic
GSO and GRO will help with batching some of these operations, but you get an
idea of the amount of logic that is needed to move bytes locally if you use TCP.</p>
<p>I won't include the calltrace for the 2-peers-on-2-containers case, but you
would get roughly twice the depth of the host namespace case, as you cross the 2
containers' network stacks plus the host network for L2/L3 routing.</p>
<p>Let's then measure some numbers with plain <code>iperf -s</code> and <code>iperf -c <ip> -n 100G</code> on my
laptop first to get a sense of the baseline numbers.</p>
<p>Client and server on the host namespace:</p>
<pre><code>[nix-shell:~]$ iperf -s
[..]
[ ID] Interval Transfer Bitrate
[ 5] 0.00-29.00 sec 200 GBytes 59.2 Gbits/sec receiver
</code></pre>
<p>So <code>60Gbits/s</code> is our number to beat, and I genuinely have no idea how much we
can save (if anything).</p>
<p>Client and server on different containers/network namespaces:</p>
<pre><code>[ ID] Interval Transfer Bitrate
[ 5] 0.00-19.07 sec 100 GBytes 45.0 Gbits/sec receiver
</code></pre>
<p>This instead shows that inter-container costs us ~25% of the original bandwidth.</p>
<h2 id="let-s-improve-our-dev-environment">Let's improve our dev environment</h2>
<p>As calling random and unexported kernel functions with random args and
forgetting to acquire/release random locks or refcounts might panic your kernel
and corrupt the fs, let's do responsible development and set up a VM.</p>
<p>I won't admit it took me a handful of panics to overcome my laziness and set up
a VM. Which tbf is a shame.</p>
<p>If I'd followed my trusted playbook, I would have had to download a Debian ISO,
create a new libvirt instance with <code>virt-manager</code> (click click), manually
install the system (click click again, my bad I never really looked into
cloud-init images), <code>apt install</code> some dev packages, <code>ssh-copy-id</code> to set up SSH
keys, create a snapshot just in case, and only then I would have been ready to
test things (so laziness was kind of justified).</p>
<p>But now on NixOS all I need to do is add to my <code>flake.nix</code> a new <code>nixosSystem</code>
instance under <code>nixosConfigurations.vm</code>, and in a minute or so I have a bootable
qcow2 image already configured with a serial console (in case our module breaks
the network), with the packages I need (mostly iperf), with my <code>/nix</code> store
already mounted (this means I can build the module on my host and it's instantly
available in the VM), and with my SSH keys already in <code>authorized_keys</code>. Quite
handy.</p>
<p>So, let's do that and add a minimal VM config:</p>
<pre><code> nixosConfigurations.vm = nixpkgs.lib.nixosSystem {
inherit system;
modules = [
nix-dev-vm-ssh.nixosModules.default
{
boot = {
loader.grub.device = "nodev";
kernelPackages = pkgs.linuxPackages_latest;
};
networking = {
hostName = "tcp-local-bypass-vm";
useDHCP = true;
};
environment.systemPackages = with pkgs; [ iperf3 ];
users.users.root.password = "root";
fileSystems."/" = {
device = "/dev/disk/by-label/nixos";
fsType = "ext4";
};
virtualisation.vmVariant.virtualisation = {
cores = 2;
memorySize = 1024;
graphics = false;
};
system.stateVersion = "26.05";
}
];
};
</code></pre>
<p>and an app to start it with <code>nix run .#vm</code>:</p>
<pre><code> apps.${system} = {
vm = {
type = "app";
meta.description = "Boot the test VM";
program = pkgs.lib.getExe (
let
vm = nixosVmConfig.system.build.vm;
hostname = nixosVmConfig.networking.hostName;
in
nix-dev-vm-ssh.lib.mkVmExe {
inherit pkgs nixosVmConfig;
text = ''
exec ${vm}/bin/run-${hostname}-vm "$@"
'';
}
);
};
</code></pre>
<p>here <code>nixosVmConfig</code> is just a let binding for
<code>self.nixosConfigurations.vm.config</code> while that
<a rel="external" href="https://github.com/jibi/nix-dev-vm-ssh"><code>nix-dev-vm-ssh</code></a> is just a small flake
to generate ephemeral keys and configure SSH access to the VM. You can surely
point the flake to your own SSH key, but I wanted something that doesn't require
me to hardcode the path of an existing key (i.e. you can <code>git clone && nix run</code>
it and it just works) and doesn't need <code>--impure</code> to run.</p>
<p>Anyway, a minute or so and our VM is ready:</p>
<pre><code>➜ tcp-local-bypass git:(master) nix run .#vm
[..]
<<< Welcome to NixOS 26.11.20260726.624af66 (x86_64) - ttyS0 >>>
Run 'nixos-help' for the NixOS manual.
tcp-local-bypass-vm login: root
Password:
[root@tcp-local-bypass-vm:~]# uname -a
Linux tcp-local-bypass-vm 7.1.5 #1-NixOS SMP PREEMPT_DYNAMIC Fri Jul 24 14:21:27 UTC 2026 x86_64 GNU/Linux
</code></pre>
<h2 id="testing-the-waters">Testing the waters</h2>
<p>Let's start with something simple to test the basics: let's hook into
<code>tcp_sendmsg()</code> and if the connection is local skip the network stack and push the
data directly into the receiving socket (no idea yet what that means in practice
but bear with me).</p>
<p>We'll need some ftrace help also here but there's a twist: we'll need to
<em>selectively</em> redirect the execution to our hook, only if the socket is local.
Which means inside the ftrace callback, whose signature looks something like:</p>
<pre><code>unsafe extern "C" fn cb(ip: c_ulong,
parent_ip: c_ulong, op:
*mut ftrace_ops,
fregs: *mut ftrace_regs)
</code></pre>
<p>we need to access the <code>tcp_sendmsg()</code> args, specifically the <code>struct sock_common</code>,
to determine if it's local or not.</p>
<p>Let's start by defining a macro to extract the args from the registers into
their actual type (this is of course x86_64 specific, but for a POC it will be fine):</p>
<pre><code>macro_rules! ftrace_args {
($fregs:expr, $($name:ident : $ty:ty),+ $(,)?) => {
ftrace_args!(@step $fregs, [di si dx cx r8 r9], $($name: $ty),+)
};
(@step $fregs:expr, [$reg:ident $($_rest:ident)*], $name:ident : $ty:ty) => {
let $name: $ty = unsafe {
let __regs = &(*($fregs as *const __arch_ftrace_regs)).regs;
__regs.$reg as usize as $ty
};
};
(@step $fregs:expr, [$reg:ident $($rest:ident)*],
$name:ident : $ty:ty, $($rn:ident : $rt:ty),+) => {
ftrace_args!(@step $fregs, [$reg], $name: $ty);
ftrace_args!(@step $fregs, [$($rest)*], $($rn: $rt),+);
};
}
</code></pre>
<p>and a helper to redirect the execution to a different function:</p>
<pre><code>#[inline]
pub unsafe fn ftrace_redirect(fregs: *mut ftrace_regs, target: *const ()) {
let fregs = fregs as *mut __arch_ftrace_regs;
unsafe {
(*fregs).regs.ip = target as c_ulong;
}
}
</code></pre>
<p>which then will be used in:</p>
<pre><code>static mut TCP_SENDMSG_OPS: ftrace_ops = unsafe { mem::zeroed() };
unsafe extern "C" fn tcp_sendmsg_bypass(sk: *mut sock, msg: *mut msghdr, size: usize) -> c_int {
// TODO
0
}
unsafe extern "C" fn tcp_sendmsg_cb(
_ip: c_ulong,
_parent_ip: c_ulong,
_op: *mut ftrace_ops,
fregs: *mut ftrace_regs,
) {
ftrace_args!(fregs, sk: *const sock_common);
if !is_local(sk) {
return;
}
unsafe { ftrace_redirect(fregs, tcp_sendmsg_bypass as *const ()) };
}
// ..
ftrace_register(&raw mut TCP_SENDMSG_OPS, c"tcp_sendmsg", tcp_sendmsg_cb)?;
</code></pre>
<p>Next we need to figure out if we are sending data to a local socket (and get a
reference to that socket). For that we can use the <code>__inet_lookup_established()</code>
kernel function.</p>
<p>The idea was to take the tuple of the sender socket and swap it, but
<code>__inet_lookup_established()</code> does already that internally: although not super
clear from the name, the function takes a tuple and returns the local socket
that should <em>receive</em> the packet associated with the provided tuple:</p>
<pre><code>unsafe fn find_local_peer(sk: *const sock_common) -> *mut sock {
unsafe {
let net = (*sk).skc_net.net;
let saddr = (*sk).__bindgen_anon_1.__bindgen_anon_1.skc_rcv_saddr;
let sport = (*sk).__bindgen_anon_3.__bindgen_anon_1.skc_num;
let daddr = (*sk).__bindgen_anon_1.__bindgen_anon_1.skc_daddr;
let dport = (*sk).__bindgen_anon_3.__bindgen_anon_1.skc_dport;
let dif = (*sk).skc_bound_dev_if;
__inet_lookup_established(
net,
saddr,
(sport as u16).to_be(),
daddr,
u16::from_be(dport as u16),
dif,
0,
)
}
}
</code></pre>
<p>We'll wrap <code>sock_common</code> into a proper type with clean methods later on, for now
let's enjoy all the inconsistencies of this short snippet
(naming: <code>skc_rcv_saddr</code>/<code>skc_daddr</code>, <code>skc_num</code>/<code>skc_dport</code>, and endianness:
source port in host order/dest port in network order).</p>
<p>Lastly we need to find something to write the data into. After <del>randomly trying
a few functions from the previous ftrace calltrace</del> a careful analysis, I
spotted:</p>
<pre><code>int tcp_send_rcvq(struct sock *sk, struct msghdr *msg, size_t size)
</code></pre>
<p>(to be precise, <code>tcp_queue_rcv()</code> which then this nice helper calls) which
seems to copy the data directly into the receiving queue of the socket by
calling <code>tcp_queue_rcv()</code>, and, bonus point, also takes the exact arguments we
have already in <code>tcp_sendmsg()</code> and deals with skb allocation for us (not
great as ideally we want a way to skip skb allocation, but as something to get
us started, fine).</p>
<p>Unfortunately the symbol is not exported, so we'll have to resort to the usual
kprobe <del>hack</del> trick to resolve its address:</p>
<pre><code>fn lookup_sym(name: *const c_char) -> *mut c_void {
let mut addr: *mut c_void = null_mut();
let mut kp: kprobe = unsafe { mem::zeroed() };
kp.symbol_name = name as *const c_char;
if unsafe { register_kprobe(&mut kp) } == 0 {
addr = kp.addr as *mut c_void;
unsafe { unregister_kprobe(&mut kp) };
}
addr
}
static mut TCP_SEND_RCVQ: Option<TcpSendRcvqFn> = None;
// ..
let addr = lookup_sym(c"tcp_send_rcvq".as_ptr() as *const c_char);
unsafe { TCP_SEND_RCVQ = Some(mem::transmute(addr)) };
</code></pre>
<p>We need one more binding (or better, a thin wrapper in <code>bindings.h</code> as the
function is defined as <code>static inline</code>) for <code>__sock_put()</code>, as we need to release
the ref counter of the socket we get out of <code>__inet_lookup_established()</code>, and
then we can put everything together:</p>
<pre><code>unsafe extern "C" fn tcp_sendmsg_bypass(_sk: *mut sock, msg: *mut msghdr, size: usize) -> c_int {
let sk_common = _sk as *const sock_common;
let peer = unsafe { find_local_peer(sk_common) };
if peer.is_null() {
return unsafe { tcp_sendmsg(_sk, msg, size) };
}
let tcp_send_rcvq_fn = unsafe { TCP_SEND_RCVQ.unwrap() };
let ret = unsafe { tcp_send_rcvq_fn(peer, msg, size) };
unsafe { ____sock_put(peer) };
ret
}
unsafe extern "C" fn tcp_sendmsg_cb(
_ip: c_ulong,
_parent_ip: c_ulong,
_op: *mut ftrace_ops,
fregs: *mut ftrace_regs,
) {
ftrace_args!(fregs, sk: *const sock_common);
let peer = unsafe { find_local_peer(sk) };
if peer.is_null() {
return;
}
unsafe { ____sock_put(peer) };
unsafe { ftrace_redirect(fregs, tcp_sendmsg_bypass as *const ()) };
}
</code></pre>
<p>Let's load it in the VM, start an <code>iperf -s</code> server and test it:</p>
<pre><code>[root@tcp-local-bypass-vm:~]# iperf -c 10.0.2.15 -n 100G
</code></pre>
<p>aand.. nothing happens. Turns out writing on the queue is only half of the story, we
need to notify the receiver that there's some data. A quick check with <code>strace</code>
confirms that our server is stuck on <code>select()</code>:</p>
<pre><code>[root@tcp-local-bypass-vm:~]# strace iperf -c 10.0.2.15 -n 100G
[..]
pselect6(5, [4], [], NULL, NULL, NULL
</code></pre>
<p>for that we can use <code>tcp_data_ready()</code> (we need once again the usual kprobe trick
to resolve its address):</p>
<pre><code> let tcp_send_rcvq_fn = unsafe { TCP_SEND_RCVQ.unwrap() };
let tcp_data_ready_fn = unsafe { TCP_DATA_READY.unwrap() };
let ret = unsafe { tcp_send_rcvq_fn(peer, msg, size) };
unsafe { tcp_data_ready_fn(peer) };
unsafe { ____sock_put(peer) };
ret
}
</code></pre>
<p>and..</p>
<pre><code>[root@tcp-local-bypass-vm:~]# iperf -c 10.0.2.15 -n 100G
Connecting to host 10.0.2.15, port 5201
[ 5] local 10.0.2.15 port 57268 connected to 10.0.2.15 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.01 sec 12.1 MBytes 101 Mbits/sec 0 320 KBytes
[ 5] 1.01-2.00 sec 0.00 Bytes 0.00 bits/sec 0 320 KBytes
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 320 KBytes
</code></pre>
<p>good, some progress, but also something goes extremely wrong after the first few
bytes, and dmesg starts spitting this in a loop:</p>
<pre><code>[44904.448599] TCP recvmsg seq # bug: copied CE8330AB, seq CE8440AB, rcvnxt CEEF30AB, fl 0
[44904.448600] WARNING: net/ipv4/tcp.c:2761 at tcp_recvmsg_locked+0x16f/0xa20, CPU#1: iperf/2757
..
[44904.448663] Call Trace:
[44904.448664] <TASK>
[44904.448665] tcp_recvmsg+0x85/0x1e0
[44904.448668] inet6_recvmsg+0x52/0x130
[44904.448669] ? security_socket_recvmsg+0x46/0x110
[44904.448671] sock_recvmsg+0x59/0xa0
[44904.448673] sock_read_iter+0x97/0x100
[44904.448675] vfs_read+0x35b/0x390
[44904.448678] ksys_read+0xbf/0xf0
[44904.448679] do_syscall_64+0xef/0x1540
[44904.448681] ? clear_bhb_loop+0x30/0x80
[44904.448682] ? clear_bhb_loop+0x30/0x80
[44904.448684] entry_SYSCALL_64_after_hwframe+0x77/0x7f
</code></pre>
<p>what's going on? The message is telling us that the invariants are not really
invarianting: <code>tcp_recvmsg()</code> got some unexpected sequence number in its receive
queue. And the reason is that the sender may end up calling <code>tcp_send_rcvq()</code>
concurrently as it keeps sending more data, while that function is supposed to
be called while holding the socket's lock.</p>
<p>Let's add some locking:</p>
<pre><code> unsafe { ____lock_sock(peer) };
let ret = unsafe { tcp_send_rcvq_fn(peer, msg, size) };
unsafe { release_sock(peer) };
</code></pre>
<p>and test it again:</p>
<pre><code>[root@tcp-local-bypass-vm:~]# iperf -s
..
[ ID] Interval Transfer Bitrate
[ 5] 0.00-62.00 sec 100 GBytes 13.9 Gbits/sec receiver
</code></pre>
<p>hurray, it works! But that's pretty much the end of the good news :/ how is it
possible that we can only achieve roughly a quarter of the bandwidth we would
get <em>without</em> the bypass? Are we even bypassing? <code>trace-cmd</code> to the rescue:</p>
<pre class="scroll-code"><code>tcp_sendmsg() {
__inet_lookup_established() {
inet_ehashfn();
}
__sock_put_shim();
__inet_lookup_established() {
inet_ehashfn();
}
__lock_sock_shim() {
lock_sock_nested() {
_raw_spin_lock_bh();
_raw_spin_unlock_bh() {
__local_bh_enable_ip();
}
}
}
tcp_send_rcvq() {
alloc_skb_with_frags() {
__alloc_skb() {
__local_bh_enable_ip();
kmem_cache_alloc_node_noprof() {
__alloc_tagging_slab_alloc_hook();
}
kmalloc_reserve() {
kmem_cache_alloc_node_noprof() {
__alloc_tagging_slab_alloc_hook();
}
}
}
}
skb_put();
tcp_try_rmem_schedule() {
__sk_mem_schedule() {
__sk_mem_raise_allocated() {
mem_cgroup_sk_charge() {
try_charge_memcg();
mod_memcg_state() {
css_rstat_updated();
}
}
sk_leave_memory_pressure() {
tcp_leave_memory_pressure();
}
}
}
}
skb_copy_datagram_from_iter() {
__check_object_size() {
check_stack_object();
is_vmalloc_addr();
__virt_addr_valid();
__check_heap_object();
}
}
tcp_queue_rcv();
}
__sk_data_ready_shim() {
sock_def_readable() {
__rcu_read_lock();
__wake_up_sync_key() {
_raw_spin_lock_irqsave();
__wake_up_common() {
pollwake() {
default_wake_function() {
try_to_wake_up() {
_raw_spin_lock_irqsave();
select_task_rq_fair() {
__rcu_read_lock();
wake_affine();
cpus_share_cache();
__rcu_read_unlock();
}
ttwu_queue_wakelist() {
scx_allow_ttwu_queue();
__smp_call_single_queue() {
call_function_single_prep_ipi();
}
}
_raw_spin_unlock_irqrestore() {
housekeeping_any_cpu();
arch_irq_work_raise() {
x2apic_send_IPI_self();
}
}
}
}
}
pollwake();
}
_raw_spin_unlock_irqrestore() {
irq_enter_rcu() {
irqtime_account_irq();
}
__sysvec_irq_work() {
__wake_up() {
_raw_spin_lock_irqsave();
__wake_up_common();
_raw_spin_unlock_irqrestore();
}
_raw_spin_lock();
_raw_spin_unlock();
__wake_up() {
_raw_spin_lock_irqsave();
__wake_up_common();
_raw_spin_unlock_irqrestore();
}
}
irq_exit_rcu() {
irqtime_account_irq();
sched_core_idle_cpu();
}
raw_irqentry_exit_cond_resched();
}
}
__rcu_read_unlock();
}
}
release_sock() {
_raw_spin_lock_bh();
tcp_release_cb();
_raw_spin_unlock_bh() {
__local_bh_enable_ip();
}
}
__sock_put_shim();
}
</code></pre>
<p>Yes, the calltrace is much shorter, and we can spot the 2
<code>__inet_lookup_established</code>, <code>tcp_send_rcvq()</code> and <code>sk_data_ready</code>. How do we
explain that?</p>
<p>Well, the issue seems to be that we are locking the sender and the receiver
together: after sending <code>SNDBUF</code> bytes of data, the sender needs to wait for the
receiver to receive them, copy them etc before it can send any more data (and
then the receiver needs to wait for the sender to refill the receive queue).</p>
<p>Waters tested, I think we managed to get our feet wet, but now it's time to find
a proper approach.</p>
<h3 id="quick-tcp-repair-detour">Quick TCP_REPAIR detour</h3>
<p>Fun fact: did you know about the <code>TCP_REPAIR</code> sockopt? That's an old option
(added back in 2012) which I haven't really heard about, that allows checkpointing
and restoring TCP connections.</p>
<p>Does this have anything to do with our kernel bypass attempt? Well, yes:
<code>tcp_send_rcvq()</code> is used exactly to send data directly to the receive queue of
a socket that is put in <code>TCP_REPAIR</code> mode (i.e. detached from the stack).</p>
<p>But the API is meant to allow restoring a TCP connection rather than being used
as a fast pipe, and that's another reason why, I guess, you see those numbers.</p>
<h2 id="proper-bypass">Proper bypass</h2>
<p>Ok, in the previous attempt we verified that the ftrace machinery works, that we
can detect local connections and selectively bypass them, and we informally
showed we can't rely too much on existing kernel functions, as even
<code>tcp_send_rcvq()</code> forces the sender to lock to the receiver, and ends up
allocating skbs (which would be nice to avoid).</p>
<p>So how do we decouple the sender from the receiver? Here's an unrelated hint from
one of my favourite stompboxes:</p>
<p><img src="https://jibi.io/blog/tcp-local-bypass/ts_buffer.png" alt="ts" /></p>
<p>Exactly! A buffer, or more precisely, a ring buffer in our case: each stream
will have its own ring, which can be a simple SPSC (read: lockless).</p>
<p>Because of this, the module will have to hook into both <code>tcp_sendmsg()</code> and
<code>tcp_recvmsg()</code> (as the latter won't read anymore from <code>sk->sk_receive_queue</code>).
This means we'll need to also hook into <code>tcp_poll()</code>, and wire that up with the
state of the ring buffer.</p>
<p>All the relevant connection state will be stored in a hashtable (one entry per
stream): this means we need to also hook into something that can notify us when
a new connection is established and closed, to create and delete the related
entry.</p>
<p>Great, now that we have a plan sketched, let's start digging into the details.</p>
<h3 id="spsc-ring-buffer">SPSC ring buffer</h3>
<p>Nothing too exciting here, just a (mostly) textbook lockless SPSC ring buffer
implementation with a bit of knowledge on how to write and read data into/from
the <code>msghdr</code> buffer.</p>
<p>Let's start from the <code>Ring</code> struct:</p>
<pre><code>const SIZE: usize = 512 * 1024;
const MASK: usize = SIZE - 1;
pub struct Ring {
buf: KVBox<MaybeUninit<[u8; SIZE]>>,
head: AtomicUsize,
tail: AtomicUsize,
}
</code></pre>
<p>there's a block of contiguous memory (I started with 8MiB, ran some benchmarks
and ended up with a smaller 512KiB one) with a power of two size, mostly for
cheap wraparound indexing, and the two <code>head</code> and <code>tail</code> atomic monotonic
counters.</p>
<p>Given it's SPSC we don't really need locks: just <code>AtomicUsize</code> for the counters,
and <code>Acquire</code>/<code>Release</code> memory barriers (which on x86 are just compiler side
reordering directives and don't produce any actual instruction) for <code>load</code> and
<code>store</code> accesses are enough to synchronize the reader and writer.</p>
<p>Then there's the constructor:</p>
<pre><code>impl Ring {
pub fn alloc() -> Result<Self, AllocError> {
let buf = KVBox::new_uninit(GFP_ATOMIC)?;
Ok(Self {
buf,
head: AtomicUsize::new(0),
tail: AtomicUsize::new(0),
})
}
</code></pre>
<p>it allocates the buffer with <code>kvmalloc()</code> (that <code>KVBox</code> is just the handy kernel
cousin of a <code>Box</code>) using <code>GFP_ATOMIC</code>, as we can't really sleep
(allocation happens in softirq context, but even if that wasn't the case ftrace
disables preemption), so under heavy load this might fail to allocate enough
memory. To reduce that chance we could use a <code>mempool</code>, but for this POC let's
keep things simple.</p>
<p>Lastly there's <code>write()</code> and <code>read()</code> (and a bunch of other) helpers. Here's
<code>write()</code> for example:</p>
<pre><code> pub unsafe fn write(&self, msg: *mut msghdr, len: usize) -> Result<usize> {
let head = self.head.load(Relaxed);
let tail = self.tail.load(Acquire);
let space = SIZE - head.wrapping_sub(tail);
if space == 0 {
return Ok(0);
}
let to_write = len.min(space);
let offset = head & MASK;
let contig = SIZE - offset;
let first_write_len = to_write.min(contig);
let mut copied = unsafe { self.copy_from(offset, first_write_len, msg) };
if copied == 0 && first_write_len > 0 {
return Err(EFAULT);
}
if copied == first_write_len && copied < to_write {
copied += unsafe { self.copy_from(0, to_write - copied, msg) };
}
self.head.store(head.wrapping_add(copied), Release);
Ok(copied)
}
</code></pre>
<p>as the <code>head</code> counter is written only by the same <code>write()</code> method (again, SPSC
so we assume writers are serialized) <code>head.load()</code> can be <code>Relaxed</code>.</p>
<p>Different story for <code>tail.load()</code> and <code>head.store()</code>: as the <code>read()</code> method,
which can be a different thread, is the one storing <code>tail</code> and loading <code>head</code>,
those accesses need to be respectively <code>Acquire</code> and <code>Release</code> to avoid memory
reordering.</p>
<p>Then we calculate how many contiguous bytes we can write, from the current
offset to the end of the buffer, and copy at most that many out of the <code>msghdr</code>
buffer (<code>copy_from()</code> is just a tiny wrapper for <code>__copy_from_iter()</code> to hide
all the Rust -> C casting noise). As that copy can fail completely, we check if
<code>copied == 0</code> and return an error in that case: no space in the ring buffer is
fine and the caller can retry, a partial copy is also fine (and we still need to
update the counters for proper accounting) but no bytes read from the <code>msghdr</code>
buffer means something went wrong with that buffer (as it points at userspace
memory, it could get partially unmapped and fault mid copy).</p>
<p>Then we write the remaining bytes (if any) starting back from the beginning of
the buffer, but only if the first copy finished completely: wrapping after a
short copy would of course leave a hole in the ring, which would be incorrect.</p>
<h3 id="socket-wait-queues">Socket wait queues</h3>
<p>Great, we have a construct to share data between a writer and a reader, next we
need something that allows the 2 sides of the connection to block (if the socket
is configured as blocking of course, different story for <code>O_NONBLOCK</code>) until a
condition is satisfied, rather than busy-looping.</p>
<p>In Linux each <code>struct sock</code> socket contains a wait queue: a <code>struct socket_wq</code>
which wraps a <code>wait_queue_head_t</code>. This is what puts a task to sleep when it
calls <code>read()</code> without enough data available, <code>write()</code> without enough space left,
or <code>poll()</code>/<code>select()</code>/<code>epoll()</code> with no pending events.</p>
<p>In our case the waitqueues will be wired up to the same entry points:
<code>tcp_sendmsg()</code>, <code>tcp_recvmsg()</code> and <code>tcp_poll()</code>.</p>
<p>But enough context, how do we use these waitqueues? First we need a couple of
them, one for each side of the connection. The first option would be to store
both <code>struct sock</code>s into our connection state, as each socket already has a
waitqueue, but that complicates a bit the lifecycle of the connection entry, as
we need to partially initialize it between the <code>connect()</code> and <code>accept()</code> events
(as each event will give us access to one <code>struct sock</code>). Since we don't really
need the socket for anything else, let's just define our own waitqueues, and
wrap the type around a more ergonomic Rust struct:</p>
<pre><code>pub struct WaitQueue {
head: KBox<UnsafeCell<wait_queue_head_t>>,
}
impl WaitQueue {
pub fn new() -> Result<Self, AllocError> {
let head = KBox::new(UnsafeCell::new(unsafe { mem::zeroed() }), GFP_ATOMIC)?;
unsafe { ____init_waitqueue_head(head.get()) };
Ok(Self { head })
}
pub fn wait(&self, cond: impl Fn() -> bool) -> Result<()> {
let wq_head = self.as_ptr();
unsafe {
let mut wq: wait_queue_entry = mem::zeroed();
wq.private = __current();
wq.func = Some(woken_wake_function);
__init_list_head(&raw mut wq.entry);
add_wait_queue(wq_head, &raw mut wq);
let result = loop {
if cond() {
break Ok(());
}
if __signal_pending() != 0 {
break Err(ERESTARTSYS);
}
wait_woken(&raw mut wq, TASK_INTERRUPTIBLE, i64::MAX);
};
remove_wait_queue(wq_head, &raw mut wq);
result
}
}
pub fn wake_up(&self, mode: u32) {
unsafe { __wake_up_interruptible_sync_poll(self.as_ptr(), mode) };
}
}
</code></pre>
<p>Usual Linux-flavoured-Rust that kinda looks like the C it's wrapping: there's a
constructor (<code>new()</code>) that allocates a new waitqueue with <code>KBox</code> and initializes
it with <code>__init_waitqueue_head()</code>, and then the actual wait logic in <code>wait()</code>.</p>
<p><code>wait()</code> starts by building a new <code>wait_queue_entry</code> for the current task that gets
registered on the queue with <code>add_wait_queue()</code>. This happens before checking
the wait condition so we can't miss a wakeup that races with the check. Then it
loops: each iteration checks again <code>cond()</code>: if met, we're done, otherwise it
checks for a pending signal (like <code>SIGINT</code> delivered to the task while it
sleeps). If there's one, we bail with <code>ERESTARTSYS</code> so the syscall is
interrupted/restarted rather than hanging unkillably. Otherwise we put the task
asleep with <code>wait_woken()</code> until something calls <code>wake_up()</code> (or a signal
arrives). Then we loop again and run the check again, as a wakeup is not a
guarantee that the condition was met. Finally we clean up everything by
unregistering the task from the queue with <code>remove_wait_queue()</code>.</p>
<h3 id="handling-the-connection-state">Handling the connection state</h3>
<p>Next stop is connections bookkeeping: we need something to store and retrieve
the ring buffers and waitqueues associated with a connection. For that we can
take inspiration from the kernel and its inet hashtable.</p>
<p>I went for a simple, global (it's a POC, no need for per-CPU), bucket list
hashtable, keyed by <code>(addr1, addr2, port1, port2)</code>:</p>
<pre><code>#[derive(Copy, Clone, PartialEq, Eq)]
pub struct ConnectionKey {
addr1: u32,
addr2: u32,
port1: u16,
port2: u16,
}
</code></pre>
<p>why <code>1</code>/<code>2</code> rather than source/destination? The key is normalized: we always
pick the lower <code>IP:port</code> first, so both ends of the connection will produce
the same key and map to the same entry.</p>
<p>Then we have the actual entry:</p>
<pre><code>pub struct ConnectionPayload {
client_addr: u32,
client_port: u16,
client_wq: WaitQueue,
client_ring: Ring,
server_wq: WaitQueue,
server_ring: Ring,
}
</code></pre>
<p>As a single entry covers both sockets/directions, it will have a ring buffer and
waitqueue per socket/direction. In addition to that we store also the client IP
and port, which allows us to tell which side of the connection we are dealing with:</p>
<pre><code> fn is_client(&self, peer: &Sock) -> bool {
peer.saddr() == self.client_addr && peer.sport() == self.client_port
}
</code></pre>
<p>The hashtable logic is actually split into 2 modules: the infrastructure which
works on generic <code>K</code> and <code>V</code> (the only requirement is for <code>K</code> to implement
<code>HashKey</code>, i.e. <code>fn raw_hash(&self) -> usize</code>), and the actual TCP hashtable
with the <code>ConnectionKey</code> and <code>ConnectionPayload</code> types we just discussed.</p>
<p>No need to go too much into the infra details: again, it's just a bucket list
hashtable with per-bucket spinlocks and RCU for concurrent lookup/deletion
(the <code>kernel</code> crate already offers some support for this, so I only had to roll
a few missing bindings).</p>
<h2 id="hooking-into-the-tcp-kernel-functions">Hooking into the TCP kernel functions</h2>
<p>Now that we have all the types needed, let's move to the ftrace hooks.</p>
<h3 id="tcp-states-and-connections-lifecycle">TCP states and connections lifecycle</h3>
<p>Let's start by creating and deleting the connection entries when a connection is
established and closed. There are probably different hooking points, but a good
initial target might be <code>tcp_set_state()</code>: by reading the socket current state
and the new state that is passed to that function, we can hook into all the TCP
state transitions.</p>
<p>Here are a few hooking points:</p>
<pre><code>unsafe extern "C" fn tcp_set_state_cb(
_ip: c_ulong,
_parent_ip: c_ulong,
_op: *mut ftrace_ops,
fregs: *mut ftrace_regs,
) {
ftrace_args!(fregs, sk: *mut sock, new_state: i32);
let Some(sock) = (unsafe { Sock::new(sk) }) else {
return;
};
let old_state = sock.state();
let new_state = new_state as u32;
match new_state {
TCP_ESTABLISHED => match old_state {
TCP_SYN_SENT => {
// handle connect()
}
TCP_SYN_RECV => {
// handle accept()
}
_ => {}
},
TCP_CLOSE => {
// handle close()
}
_ => {}
};
}
</code></pre>
<p>Taking also the opportunity to introduce a small <code>Sock</code> wrapper that hides all
the complexity of the <code>struct sock</code> kernel type:</p>
<pre><code>pub struct Sock(NonNull<sock>);
</code></pre>
<p>Next we need to figure out whether we are dealing with a local connection or
not.</p>
<p>If you think I'm just wasting time on a problem we have already solved in the
previous <code>Testing the waters</code> section, if you think we can just reuse the
<code>__inet_lookup_established</code> trick, well, I've got some bad news for you (it's
ok, it's more subtle than it looks).</p>
<p>In the initial test, we needed to figure out if <code>tcp_sendmsg()</code> was sending data
to a local connection, so a) this check would run for every chunk of data sent
and b) the connection was already established (that's why
<code>__inet_lookup_established()</code> worked fine).</p>
<p>Here we need something different: we want to run this check exactly once, as we
need to allocate the connection entry with the ringbufs and waitqueues only
once, and we need to do that <em>before</em> the connection is established (otherwise
we risk a race where the connection is established, the peers send data, but the
bypass connection data is not yet initialized).</p>
<p>So how do we deal with that? We need to check:</p>
<ul>
<li>if the client is local: easy, need to see a <code>TCP_SYN_SENT</code> -> <code>TCP_ESTABLISHED</code> state
transition in <code>tcp_set_state()</code></li>
<li>if the server is local: also easy, if the client socket destination is
<code>IFF_LOOPBACK</code> it means the kernel determined the server is also local</li>
</ul>
<p>we can do the latter check with:</p>
<pre><code>impl Sock {
pub fn dst_is_loopback(&self) -> bool {
let _rcu = rcu::read_lock();
let dst = unsafe { ____sk_dst_get(self.as_ptr()) };
!dst.is_null() && unsafe { (*(*dst).__bindgen_anon_1.dev).flags } & __IFF_LOOPBACK != 0
}
}
</code></pre>
<p>Then we run it when the client transitions to <code>TCP_ESTABLISHED</code> to create the new
connection entry in <code>tcp_set_state()</code> and configure the bypass:</p>
<pre><code> let connection_key = ConnectionKey::new(&sock);
if old_state == TCP_SYN_SENT && new_state == TCP_ESTABLISHED {
if !sock.dst_is_loopback() {
return;
}
if let Ok(payload) = ConnectionPayload::new(&sock) {
let _ = CONNECTIONS_TABLE.insert(connection_key, payload);
}
} else if new_state == TCP_CLOSE {
if let Some(connection) = CONNECTIONS_TABLE.lookup(connection_key) {
connection.wake_up_pollfree();
}
CONNECTIONS_TABLE.delete(connection_key);
}
</code></pre>
<p>then the other hooks (<code>send()</code>/<code>recv()</code>/<code>poll()</code>) can use <code>CONNECTIONS_TABLE</code> as
single source of truth for all local connections that should be bypassed (that
<code>wake_up_pollfree()</code> on teardown will make sense once we get to polling).</p>
<p>..well, kind of. Turns out this approach works only for connections within the
same network namespace. In fact, if a connection crosses namespaces, it
means the client needs to route it through another interface (e.g. a veth).
Which means <code>__sk_dst_get()</code> will not give us back <code>IFF_LOOPBACK</code> anymore
when we check it in <code>tcp_set_state()</code>.</p>
<p>There surely must be other ways to track a local packet crossing namespaces,
but that could complicate a bit the whole flow, so for the sake of simplicity
let's focus on the same-namespace scenario for now and try to get the bypass to
work within this limitation.</p>
<h3 id="sending-and-receiving-data">Sending and receiving data</h3>
<p>Let's move to the <code>tcp_sendmsg()</code> hook:</p>
<pre><code>unsafe fn redirect_if_bypassed(fregs: *mut ftrace_regs, sock: *mut sock, target: *const ()) {
let Some(sock) = (unsafe { Sock::new(sock) }) else {
return;
};
if CONNECTIONS_TABLE
.lookup(ConnectionKey::new(&sock))
.is_some()
{
unsafe { ftrace_redirect(fregs, target) };
}
}
fn __tcp_sendmsg_bypass(sock: *mut sock, msg: *mut msghdr, size: usize) -> Result<usize> {
let sock = unsafe { Sock::new_unchecked(sock) };
let flags = unsafe { (*msg).msg_flags } as c_int;
let connection = CONNECTIONS_TABLE
.lookup(ConnectionKey::new(&sock))
.ok_or(EPIPE)?;
let (_, write_ring) = connection.peer_rings(&sock);
let (peer_wq, other_peer_wq) = connection.peers_wq(&sock);
let mut written: usize = 0;
let result = (|| -> Result<()> {
while written < size {
let sock_lock = sock.lock();
let n = unsafe { write_ring.write(msg, size - written) }?;
if n > 0 {
other_peer_wq.wake_up(__EPOLLIN | __EPOLLRDNORM);
}
written += n;
if written == size {
return Ok(());
}
if sock.state() != TCP_ESTABLISHED {
return Err(EPIPE);
}
if flags & MSG_DONTWAIT as c_int != 0 || sock.is_nonblocking() {
return Err(EAGAIN);
}
drop(sock_lock);
peer_wq.wait(|| write_ring.space() > 0)?;
}
Ok(())
})();
if written == 0 {
result?;
}
Ok(written)
}
unsafe extern "C" fn tcp_sendmsg_bypass(
sock: *mut sock,
msg: *mut msghdr,
size: usize,
) -> c_int {
match __tcp_sendmsg_bypass(sock, msg, size) {
Ok(n) => n as c_int,
Err(e) => e.to_errno(),
}
}
unsafe extern "C" fn tcp_sendmsg_cb(
_ip: c_ulong,
_parent_ip: c_ulong,
_op: *mut ftrace_ops,
fregs: *mut ftrace_regs,
) {
ftrace_args!(fregs, sock: *mut sock);
unsafe { redirect_if_bypassed(fregs, sock, tcp_sendmsg_bypass as *const ()) };
}
</code></pre>
<p>Nothing too contentious: we lookup the entry in the connections table, and we
retrieve the write ring and the 2 peers' waitqueues, and then try to write on
the ring: if we managed to write at least one byte, we wake up the reader
(through its waitqueue, if it's waiting). If we couldn't write everything, then
unless the socket is nonblocking or the connection is being/is closed, we put
the socket asleep and try again once there's space.</p>
<p>Maybe worth noticing is the <code>sock.lock()</code> around each write: the ring itself is
lock free, but that's only true if there's exactly one producer and one consumer
(after all, it's called SPSC). Nothing stops two threads from calling <code>send()</code> on
the same socket concurrently, so we still need the socket's own lock to
serialize potentially concurrent producers (and its counterpart on the peer's
socket to serialize concurrent consumers): the ring's atomics only handle the
producer/consumer handoff, not multiple producers/consumers on the same side.</p>
<p>I'll skip the <code>tcp_recvmsg()</code> part, as its mostly specular to the send part.</p>
<h3 id="polling-events">Polling events</h3>
<p>Let's move to adding support for the different polling mechanisms and take a
look at <code>tcp_poll_bypass()</code>:</p>
<pre><code>unsafe extern "C" fn tcp_poll_bypass(
file: *mut file,
socket: *mut socket,
wait: *mut poll_table_struct,
) -> u32 {
let sock = unsafe { Sock::new_unchecked((*socket).sk) };
let mut mask: u32 = 0;
let Some(connection) = CONNECTIONS_TABLE.lookup(ConnectionKey::new(&sock)) else {
return mask;
};
let (peer_wq, _) = connection.peers_wq(&sock);
peer_wq.poll_wait(file, wait);
match sock.state() {
TCP_LISTEN => return sock.listen_poll(),
TCP_CLOSE_WAIT => mask |= __EPOLLIN | __EPOLLRDNORM | __EPOLLRDHUP,
TCP_CLOSE => return __EPOLLIN | __EPOLLRDNORM | __EPOLLRDHUP | __EPOLLHUP,
_ => {}
};
let events = unsafe { __poll_requested_events(wait) };
let (read_ring, write_ring) = connection.peer_rings(&sock);
if events & (__EPOLLIN | __EPOLLRDNORM) != 0 {
if read_ring.avail() > 0 {
mask |= __EPOLLIN | __EPOLLRDNORM;
}
}
if events & (__EPOLLOUT | __EPOLLWRNORM) != 0 {
if write_ring.space() > 0 {
mask |= __EPOLLOUT | __EPOLLWRNORM;
}
}
mask
}
unsafe extern "C" fn tcp_poll_cb(
_ip: c_ulong,
_parent_ip: c_ulong,
_op: *mut ftrace_ops,
fregs: *mut ftrace_regs,
) {
ftrace_args!(fregs, _file: *const (), socket: *mut socket);
unsafe { redirect_if_bypassed(fregs, (*socket).sk, tcp_poll_bypass as *const ()) };
}
</code></pre>
<p>What's going on here? Let's start from a bit of context on polling first.</p>
<p>Linux offers a few different mechanisms to poll the state of a file, namely
<code>select()</code>, <code>poll()</code> and <code>epoll()</code> (I'm ignoring <code>io_uring</code> here), and as a
socket is also a file, it can also be polled with any of these.</p>
<p>All of them share a common entry point, <code>vfs_poll()</code>, the generic poll dispatch
for files. This kernel function doesn't inspect any socket state itself, it just
dispatches the poll request to the <code>(struct *file_operations)->poll()</code> callback
(part of the <code>struct file</code> we are polling), which for socket files points to
<code>sock_poll()</code>. This in turn calls into the socket's own poll mechanism, <code>(struct *proto_ops)->poll()</code>, which for TCP sockets is set to <code>tcp_poll()</code>. That's what
we are hooking into here.</p>
<p>Great, but what does <code>tcp_poll()</code> actually do? You can think of it as a function
that maps a set of requested events and the socket state to a set of ready
events. For example, user asks for socket readiness (as in: bytes to read),
connection is established, there are bytes in the receive queue, poll returns
<code>EPOLLIN</code>.</p>
<p>In our specific bypass function, we first retrieve as usual the bypassed
connection (we know already it's local as we called <code>redirect_if_bypassed()</code> to
jump here). Then we retrieve the peer's waitqueue and pass that, together with
the socket's file and the <code>wait</code> <code>poll_table_struct</code> to the <code>poll_wait()</code>
kernel function, as also the original <code>tcp_poll()</code> does (here we made
<code>poll_wait</code> a method of our <code>WaitQueue</code> wrapper, so that <code>WaitQueue</code> doesn't
need to expose publicly an <code>as_ptr()</code> method).</p>
<p><code>poll_wait()</code>, as the name suggests, is what actually performs the polling and
the wait. Nah, I'm kidding. The name is a bit confusing, but all the function
does is calling the callback (if not NULL) set by the <code>vfs_poll()</code> caller
(<code>select()</code>, <code>poll()</code> or <code>epoll()</code>) in the <code>poll_table_struct</code>.</p>
<p>Confused by this kind of circular dance? Bear with me. Whatever mechanism calls
into <code>vfs_poll()</code> has no idea about the actual polling logic that such file
implements (recall we had to peel file -> socket -> TCP socket to get to
<code>tcp_poll()</code>). Likewise, <code>tcp_poll()</code> has no knowledge of how the <code>vfs_poll()</code>
caller handles polling. So <code>poll_wait()</code> is there to allow these 2 layers to
meet: the polling mechanisms pass a callback whose job is to decide what gets
registered onto the file waitqueue.</p>
<p><code>select()</code> and <code>poll()</code> register the task directly onto the socket waitqueue and
block it right there until it's woken up. Just to reiterate as the name keeps
suggesting otherwise: no wait happens in <code>poll_wait()</code>. <code>select()</code> and <code>poll()</code>
call <code>vfs_poll()</code> a first time with the callback set, which is when the task
gets registered onto the waitqueue, then control goes back to
<code>select()</code>/<code>poll()</code>, which are the ones doing the actual sleep. When the task
wakes up they call <code>vfs_poll()</code> again, this time with a NULL callback, so
<code>tcp_poll()</code> skips the registration and only translates the requested events
into the ready ones.</p>
<p><code>epoll()</code> is a bit more involved, as it works in 2 steps. First,
<code>epoll_ctl(EPOLL_CTL_ADD, ...)</code> sets its own callback in the <code>poll_table_struct</code>
and calls <code>vfs_poll()</code>, which ends up registering an epoll indirection layer
onto the socket waitqueue. This means that whenever there's a wakeup event, the
socket waitqueue doesn't wake up the task, it calls into epoll instead.
Then <code>epoll_wait()</code> blocks the task on a second waitqueue, the one owned by the
epoll instance itself, and it's that indirection layer's job to wake it up
whenever the socket waitqueue fires.</p>
<p>Worth noting the socket waitqueue is not just a polling thing: it's the same one
we saw in the sendmsg/recvmsg bypass, where a task blocked on <code>send()</code> or
<code>recv()</code> registers itself and gets woken up directly. A single wakeup walks all
the registered entries, so it can wake a blocked reader and notify epoll at
once.</p>
<p>Back to our bypass function: once <code>poll_wait()</code> is done, all that's left is
setting the events mask based on the state of the socket and of the rings.</p>
<p>One last thing about those <code>epoll()</code> registrations: they outlive the syscall, so
we can't just drop a connection entry when the connection closes, or epoll would
be left pointing at a freed waitqueue. That's what the <code>wake_up_pollfree()</code> call
we saw in <code>tcp_set_state()</code> is for: it tells the polling mechanisms to detach
from the waitqueue before we delete the entry.</p>
<h2 id="does-this-work">Does this work</h2>
<p>Let's wire up everything together properly now.</p>
<p>First let's define a NixOS module for our bypass service that will take care of
building and loading the lkm when the <code>services.tcpLocalBypass.enable</code> config is
enabled. The various kernel modules defined in
<a rel="external" href="https://github.com/NixOS/nixpkgs/tree/master/pkgs/os-specific/linux"><code>nixpkgs</code></a>
are a good starting point to learn about how to build one:</p>
<pre><code> let
system = "x86_64-linux";
pkgs = nixpkgs.legacyPackages.${system};
mkModulePackage =
{ pkgs, kernel }:
kernel.stdenv.mkDerivation {
pname = "tcp-local-bypass";
version = "0.1.0";
src = self;
nativeBuildInputs = kernel.moduleBuildDependencies ++ [ pkgs.clang ];
makeFlags = kernel.commonMakeFlags ++ [
"KERNEL_DIR=${kernel.dev}/lib/modules/${kernel.modDirVersion}/build"
];
installFlags = [ "INSTALL_MOD_PATH=$(out)" ];
enableParallelBuilding = true;
};
in
{
nixosModules.default =
{
config,
lib,
pkgs,
...
}:
let
kernel = config.boot.kernelPackages.kernel;
in
{
options = {
services.tcpLocalBypass.enable = lib.mkEnableOption "tcp_local_bypass kernel module";
};
config = lib.mkIf config.services.tcpLocalBypass.enable {
boot = {
extraModulePackages = [
(mkModulePackage { inherit pkgs kernel; })
];
kernelModules = [ "tcp_local_bypass" ];
};
};
};
</code></pre>
<p>Then we can wire it and enable it in our test VM:</p>
<pre><code> nixosConfigurations.vm = nixpkgs.lib.nixosSystem {
modules = [
self.nixosModules.default
{
services.tcpLocalBypass.enable = true;
}
</code></pre>
<p>and if we rerun the VM app we should see the module loaded:</p>
<pre><code>➜ tcp-local-bypass git:(master) nix run .#vm
..
[root@tcp-local-bypass-vm:~]# dmesg | grep tcp_local_bypass
[ 4.980560] tcp_local_bypass: loading out-of-tree module taints kernel.
[ 5.189205] tcp_local_bypass: tcp_local_bypass: loaded
</code></pre>
<p>With everything in place, let's run one more <code>iperf</code> test:</p>
<pre><code>[root@tcp-local-bypass-vm:~]# iperf -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.0.2.15, port 33708
[ 5] local 10.0.2.15 port 5201 connected to 10.0.2.15 port 33720
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 19.9 GBytes 171 Gbits/sec
[ 5] 1.00-2.00 sec 20.5 GBytes 176 Gbits/sec
[ 5] 2.00-3.00 sec 19.2 GBytes 165 Gbits/sec
[ 5] 3.00-4.00 sec 20.6 GBytes 177 Gbits/sec
[ 5] 4.00-5.00 sec 20.7 GBytes 177 Gbits/sec
[ 5] 5.00-6.00 sec 19.9 GBytes 171 Gbits/sec
[ 5] 6.00-7.00 sec 20.9 GBytes 179 Gbits/sec
[ 5] 7.00-8.00 sec 21.0 GBytes 180 Gbits/sec
[ 5] 8.00-9.00 sec 20.3 GBytes 174 Gbits/sec
[ 5] 9.00-10.00 sec 17.2 GBytes 147 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 200 GBytes 172 Gbits/sec receiver
</code></pre>
<p><img src="https://jibi.io/blog/tcp-local-bypass/success.gif" alt="success" /></p>
<p>I'm a bit impressed that we managed to achieve roughly 3x the original bandwidth
with just a couple of ringbufs and waitqueues, a handful of ftrace hooks and no
fancy optimizations, but I'll take that, it was a fun experiment :D</p>
<p>Full code is on <a rel="external" href="https://github.com/jibi/tcp-local-bypass">github</a> if you want
to try it (in a VM).</p>
<h2 id="what-about-cross-container-traffic">What about cross container traffic</h2>
<p>Now that we know this thing works, we can resume our cross namespace quest.</p>
<p>Brief recap first. The current approach to determine if a connection is local is
based on hooking into the <code>TCP_SYN_SENT</code> -> <code>TCP_ESTABLISHED</code> state transition
in <code>tcp_set_state()</code>, which implicitly tells us the client is local.
There we look for the destination of the client socket with <code>__sk_dst_get()</code>.
If it's <code>IFF_LOOPBACK</code>, it means the connection is going to loopback, which in
turn means the server is also local.</p>
<p>This is great and everything, problem is it doesn't work with local connections
that cross network namespaces: when a client needs to hop through veth/bridge
devices to reach the server, even if local, the destination returned by
<code>__sk_dst_get()</code> would not show up as <code>IFF_LOOPBACK</code>.</p>
<p>So how do we deal with that? What other tricks do we have to tell if the client
and the server of a connection are both local?</p>
<h3 id="lookup-the-listener">Lookup the listener</h3>
<p>As a first attempt I tried <code>__inet_lookup_listener()</code>: we pass the server IP and port as
destination IP and port and see if we get anything back, but unfortunately that
doesn't work as I would expect: that function simply looks for a socket that can
accept a connection for the destination IP and port, it doesn't care if the
destination can actually be routed to that socket.</p>
<p>This means if we have a local server listening on <code>0.0.0.0:80</code> and ask
<code>__inet_lookup_listener()</code> if there's a listener for <code>example.com:80</code>, it
will happily return us our local server. As we clearly don't want to
bypass/blackhole all external traffic that happens to have a destination port
that matches a local server, this solution is a no-go.</p>
<h3 id="flip-the-check">Flip the check</h3>
<p>What about flipping the check? Rather than "client socket connected, is the
server local", can we check "server socket accepted a connection, is the client
local"? The idea seems worth exploring, but the devil is in the details.</p>
<p>First, the timing. The bypass has to be set up while the client is still in
<code>connect()</code>, not when the server calls <code>accept()</code>. Once the handshake completes
the client can start sending right away, without waiting for the server to
accept, and with no connection entry yet those bytes would be lost.</p>
<p>Secondly, the lookup itself. To tell if the client is local we'd call
<code>__inet_lookup_established()</code>, but the buckets in <code>tcp_hashinfo</code> are hashed by
tuple and namespace, so there's no way to find all sockets for a given tuple
without walking every namespace. <code>O(#namespaces)</code> would already be bad enough,
but it's not even correct: nothing guarantees that the socket we find in some
random namespace is the one that actually connected to our server.</p>
<h3 id="just-add-more-hooks">Just add more hooks</h3>
<p>All hope is gone then? Not really. Let's zoom out a bit: the kernel already
knows about every local client and server socket, well before the
<code>TCP_SYN_SENT</code> -> <code>TCP_ESTABLISHED</code> transition we've been looking at. We just
need to observe the right code path.</p>
<p>The idea is to detect both ends and pair them up: record when a local client
starts a connection, record when a local server accepts one, and if the two
match on the same 4-tuple, we know both ends are local and we can set up the
bypass.</p>
<p>For the client side we add a hook to <code>tcp_connect()</code>, which runs right after
<code>inet_hash_connect()</code> has picked the source port (so the 4-tuple is settled).
There we record the tuple as a pending/half confirmed connection, along with
the client socket so we can recognize it later.</p>
<p>For the server side we hook into <code>inet_csk_reqsk_queue_hash_add()</code>, which fires
when a server receives a SYN and queues the request socket, so well before
<code>accept()</code>. If a pending marker for that tuple is already there, we've found
both ends: the pairing is confirmed and the tuple is an actual local connection.</p>
<p>Then in <code>tcp_set_state()</code> we swap the old <code>IFF_LOOPBACK</code> check for a
lookup in the pairing table: if the tuple is confirmed and the client socket
matches, the connection is local, we create the entry in our connection table,
and the connection is bypassed.</p>
<h3 id="what-about-nat">What about NAT?</h3>
<p>Great, now we can really catch all local connections, right?</p>
<p>..right?</p>
<p>Well, no. Somehow we forgot about NAT (think for example about kubeproxy
rewriting a VIP to a local backend).</p>
<p>The good news is that there's still something we can do about that: we need more
hooks, this time in netfilter's <code>__nf_conntrack_confirm()</code>: when netfilter creates
a new entry, we record it, so that we can insert both sides of the connection
(pre/post NAT) in our connection table.</p>
<p>The bad news is that this is already getting longer than I was expecting, so
perhaps we can keep giving this whole thing a try for another post :)</p>
<h2 id="the-picture">The picture</h2>
<p><img src="https://jibi.io/blog/tcp-local-bypass/tappeiner.png" alt="Tappeiner" /></p>
<p>Tappeiner Trail in Meran, looking at the city and at the Adige valley</p>
fun bisecting nixpkgs2026-05-12T00:00:00+00:002026-05-12T00:00:00+00:00
Unknown
https://jibi.io/blog/nixpkgs-bisect/<p>After updating some stuff on my Raspberry Pi and trying to deploy a new NixOS
generation, I encountered this build error:</p>
<pre><code>$ nixos-rebuild switch --flake .#rpi --target-host home --sudo
[..]
error: Cannot build '/nix/store/2j8nsb924fjw7vgkrsmhi9k1fs3jka58-bisca-backend-aarch64-unknown-linux-gnu-0.1.0.drv'.
Reason: builder failed with exit code 101.
Output paths:
/nix/store/80qy9xhbqhr9vrw50jylzakhcbm83a4b-bisca-backend-aarch64-unknown-linux-gnu-0.1.0
[..]
> cargo:warning=aarch64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-m64'
>
> --- stderr
>
>
> error occurred in cc-rs: command did not execute successfully (status code exit status: 1): LC_ALL="C" "aarch64-unknown-linux-gnu-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-w" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_ISNAN" "-DHAVE_LOCALTIME_R" "-DSQLITE_ENABLE_UNLOCK_NOTIFY" "-o" "/build/backend/target/release/build/libsqlite3-sys-a4661d61eaf4765c/out/c877a2978823c39d-sqlite3.o" "-c" "sqlite3/sqlite3.c"
>
>
> warning: build failed, waiting for other jobs to finish...
For full logs, run:
nix log /nix/store/2j8nsb924fjw7vgkrsmhi9k1fs3jka58-bisca-backend-aarch64-unknown-linux-gnu-0.1.0.drv
</code></pre>
<p>Now, the inner details of bisca are not really interesting here.
What's relevant is that it is a Rust service which needs to be deployed on an
aarch64 rpi zero, and since it's some private stuff I wrote, there's of course
no cached package on cache.nixos.org. And since I can't really build it on the
rpi due to CPU/memory constraints, I need to cross compile it on my laptop.</p>
<p>For that there are a couple of options: emulate a full aarch64 system with QEMU
binfmt (slow) or do proper cross compilation with x86_64 native compilers that
produce aarch64. Here I'm going for the latter:</p>
<pre><code>{
config,
bisca,
lib,
pkgs,
...
}:
let
buildSystem = pkgs.stdenv.buildPlatform.system;
system = pkgs.stdenv.hostPlatform.system;
crossPkgs = import bisca.inputs.nixpkgs {
localSystem = buildSystem;
crossSystem = system;
overlays = [ bisca.overlays.default ];
};
in
{
# ..
services = {
bisca = {
enable = true;
backend.package = crossPkgs.bisca-backend;
};
}
</code></pre>
<p>Enough context, let's go back to the actual issue. Given this is my first time
dealing with a random failure after bumping <code>nixpkgs</code>, I have to first figure
out a way to debug it, which is what this write-up is all about.</p>
<h2 id="the-plan-bisect-all-the-things">The plan: bisect all the things</h2>
<p>The initial plan was almost naive: run git bisect on <code>nixpkgs</code>, with the
<code>$LAST_WORKING_REV</code> from my <code>nix-config</code> as good commit and <code>nixpkgs-unstable</code>
as bad commit. Then, for each commit try to build bisca pointing its <code>nixpkgs</code>
input to my local repo where I was bisecting:</p>
<pre><code>$ nix build ~/nix-config#nixosConfigurations.rpi.config.services.bisca.backend.package \
--override-input bisca/nixpkgs "git+file://~/nixpkgs?rev=${rev}"
</code></pre>
<p>With <code>--override-input</code> we achieve exactly that: point bisca's <code>nixpkgs</code> to our
local repo, and since <code>crossPkgs</code> is based on bisca's own <code>nixpkgs</code> (the
<code>crossPkgs = import bisca.inputs.nixpkgs</code> line), the package will be built with
that revision.</p>
<p>Let's put everything together:</p>
<pre><code>$ git bisect start nixpkgs-unstable b40629efe5d6ec48dd1efba650c797ddbd39ace0
Bisecting: 15846 revisions left to test after this (roughly 14 steps)
[3113894db88dad207079a49632efb947f11cf80d] Merge master into staging-nixos
$ git bisect run sh -c '
nix build ~/nix-config#nixosConfigurations.rpi.config.services.bisca.backend.package \
--override-input bisca/nixpkgs "git+file://~/nixpkgs?rev=$(git rev-parse HEAD)" \
--no-link
'
</code></pre>
<p>now we can go grab a coffee, and once we are back <code>git bisect</code> should have found
the first bad revision: after all, we are just building bisca ~14 times, right?</p>
<p>Well, not that fast. With this approach we'll most likely come back to a
nix-daemon busy compiling random things, rather than to a finished bisect.</p>
<p>Why is that? Because we are allowing every single commit in <code>nixpkgs</code> to be used
as <code>nixpkgs</code> input, and this means we can end up in some revision that produces
derivations that are not cached by Hydra. To see why some commits are cached and
others aren't, a quick (simplified) recap on how branching works in <code>nixpkgs</code> is
in order.</p>
<h3 id="nixpkgs-branching-model">nixpkgs branching model</h3>
<p>Problem statement: all changes landing on <code>master</code> cause Hydra to rebuild
everything that depends on said changes. This might end up causing a <a rel="external" href="https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#changes-causing-mass-rebuilds">mass
rebuild</a>
(which is bad use of the infra) so commits usually get merged and piled up in
the <code>staging</code> branch and then regularly merged into <code>staging-next</code>, which is
manually built/tested on Hydra. If there are no major regressions, this gets
merged into <code>master</code>.</p>
<p>Here's the same thing as a diagram from the
<a rel="external" href="https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#staging">CONTRIBUTING</a>
doc:</p>
<img src="nixpkgs-branching.png" width="600" >
<p>The relevant bit for us: Hydra builds and caches (besides manual runs) what
lands on the tip of <code>master</code> and <code>staging-next</code> (<code>master</code> then gets
fast-forwarded into <code>nixpkgs-unstable</code> when the build is successful). The
<code>staging</code> branch itself is not built on Hydra, and neither are mid-PR commits
inside any of the merges.</p>
<p>So if we land on a mid-PR commit, we'll likely hit cache misses, which could
require rebuilding lots of stuff (including rustc, llvm etc). And that takes
quite a while.</p>
<h3 id="optimizing-the-bisect-process">Optimizing the bisect process</h3>
<p>Can we optimize this? Partially yes: we walk only the branch's mainline (merge
commits and commits pushed directly onto <code>master</code>, cached by Hydra), avoiding
the inner commits of the branches/PRs that got merged in.</p>
<p>Why <em>partial</em>: this keeps us in cached territory while we're following
<code>master</code>'s mainline, and even when we peel into a <code>staging-next -> master</code> merge
we stay cached (<code>staging-next</code> tips are built by Hydra too). But the regression
likely lives in a <code>staging -> staging-next</code> merge, and once we peel into that
range we're walking the staging mainline, which Hydra doesn't build. So we'll
have to deal with cache misses <em>eventually</em>, just on a much smaller set of
commits (and I'd expect commits within the same <code>staging</code> branch to share more
derivations, so fewer cache misses as the bisect narrows down).</p>
<p>Great, how do we do that in practice? git bisect has exactly the right tool for
following the mainline of a branch. From its man:</p>
<pre><code> --first-parent
Follow only the first parent commit upon seeing a merge commit.
In detecting regressions introduced through the merging of a branch, the merge commit will be identified as introduction of the bug and its ancestors will be ignored.
This option is particularly useful in avoiding false positives when a merged branch contained broken or non-buildable commits, but the merge itself was OK.
</code></pre>
<p>Let's give it another try with:</p>
<pre><code>$ git bisect start --first-parent nixpkgs-unstable b40629efe5d6ec48dd1efba650c797ddbd39ace0
</code></pre>
<p>This time, after a few minutes git bisect is done:</p>
<pre><code>$ git bisect log
[..]
# first bad commit: [9cadaf6932b7c926e468f777549d57f04a7212da] staging-next 2026-04-07 (#507470)
</code></pre>
<p>and without much surprise.. it's a merge commit.</p>
<h3 id="peeling-off-the-first-layer">Peeling off the first layer</h3>
<p>As anticipated, the next step is to peel/bisect into the <code>staging-next -> master</code> specific PR and find which <code>staging -> staging-next</code> PR introduced the
breakage. We start from the merge commit that git bisect gave us, get its first
parent (in <code>master</code>'s mainline, good one) and its second parent
(<code>staging-next</code>'s HEAD, bad one), and bisect the mainline commits in that range:</p>
<pre><code>$ git bisect start --first-parent 9cadaf6932b7c926e468f777549d57f04a7212da^2 9cadaf6932b7c926e468f777549d57f04a7212da^1
</code></pre>
<p>without much surprise, this also didn't take long:</p>
<pre><code>$ git bisect log
[..]
# first bad commit: [d79f9e22fa7067d6ae209c14c207ec0af3fea5d4] Merge branch 'staging' into staging-next
</code></pre>
<p>Now unfortunately we move into the painful part: we need to bisect the <code>staging</code>
branch, where almost all PRs get merged into, and which is not built on Hydra:</p>
<pre><code>$ git bisect start --first-parent d79f9e22fa7067d6ae209c14c207ec0af3fea5d4^2 d79f9e22fa7067d6ae209c14c207ec0af3fea5d4^1
[..]
Bisecting: 181 revisions left to test after this (roughly 8 steps)
</code></pre>
<p>Roughly 8 commits to test, with all their cache misses. Still an improvement
from the initial 14 commits to test though.</p>
<p>For this last round I had to let <code>git bisect</code> run for a few hours, but this time
it gave me an actual commit!</p>
<pre><code>$ git bisect log
[..]
# first bad commit: [21ad3e929a5074dec3565eb53d29ed7ad715996e] treewide: Remove continuation escape at end of commands (#497857)
</code></pre>
<p>The next thing to do was to look into the bad commit, try to revert it on top of
<code>nixpkgs-unstable</code> to see if bisca would build with that (it did), and fill a
PR.</p>
<p>Quiz time: what was wrong with the buggy commit? (<a rel="external" href="https://github.com/NixOS/nixpkgs/pull/519017">solution</a>)</p>
<h3 id="closing-words">Closing words</h3>
<p>This was fun, and next time something breaks it'll be definitely easier to hunt
down what's causing the breakage, but in the meantime, what happens while my fix
gets reviewed and merged? Do I keep my rpi broken? Of course not, I just point
bisca's nixpkgs to my branch:</p>
<pre><code>{
inputs = {
# ..
bisca = {
url = "git+ssh://git@github.com/jibi/bisca";
inputs.nixpkgs.url = "github:jibi/nixpkgs/partial-revert-remove-trailing-line-continuations";
};
</code></pre>
<p>and everything will build fine.</p>
<h2 id="the-picture">The picture</h2>
<p><img src="https://jibi.io/blog/nixpkgs-bisect/braies.jpg" alt="Braies" /></p>
<p>View of the Croda del Becco (Seekofel) peak of the Dolomites from Lago di Braies (Pragser Wildsee).</p>
XDP without eBPF (aka learning some Rust LKM)2026-03-04T00:00:00+00:002026-03-04T00:00:00+00:00
Unknown
https://jibi.io/blog/xdp-rust-lkm/<p>It's been about 10 years since <a rel="external" href="https://medium.com/@tom_84912/happy-birthday-xdp-a971b8ac75e6">XDP was born at Netdev Conf 1.1 in Seville</a>!</p>
<p>While I wasn't involved in the design of the technical bits (I barely knew any Linux kernel networking internals, skb details, etc. at the time), I was there giving my first talk, or rather my first <a rel="external" href="https://netdevconf.info/1.1/proceedings/slides/dangaard-network-performance.pdf">BoF session</a>, on that subject, discussing what we believed at Cloudflare was needed from the kernel for something that would later become XDP, based on our experience with similar frameworks:</p>
<ul>
<li>Processing packets at the very lowest layer, skipping all the network stack overhead</li>
<li>Filtering directly on the NIC RX ring (a circular buffer), avoiding skb allocation and a lot of costly <code>kmalloc()</code> / <code>kfree()</code></li>
<li>The ability to run safe programs (BPF) to decide whether traffic should be allowed or not</li>
</ul>
<p>eBPF was of course the obvious choice, and the amount of infrastructure and projects built around XDP/eBPF since then has been impressive. But I started wondering: is there a way to reuse that same cool XDP infrastructure without all the eBPF <del>limitations :P</del> constraints? Of course those are there for a reason, but let's pretend we know what we are doing: can we run some code in the XDP path without the verifier vetoing its soundness?</p>
<p>This thought is not new: some time ago I wrote a <a rel="external" href="https://github.com/jibi/xdp-rust-kernel-patch">POC</a> to run Rust code linked to a regular C Linux kernel module in the XDP path instead of the usual eBPF bytecode. It was a fun experiment, but it required patching and recompiling the kernel to add a new function pointer that the XDP path could call. Wouldn't it be nicer if we didn't need to recompile the kernel?</p>
<p>I've been reading about (and then mostly neglecting) Rust kernel development for far too long, so I figured it was time to give it a shot. Reworking that POC into a Rust LKM that runs in the XDP path without patching/recompiling the whole kernel felt like a good place to start.</p>
<p>Now, I can think of at least a couple of ways to run arbitrary code in the XDP path without recompiling the kernel:</p>
<ul>
<li>(Live)patch the XDP dispatcher to execute our code instead of whatever is in the <code>bpf_func</code> pointer</li>
<li>Populate a <code>struct bpf_prog</code>, point its <code>bpf_func</code> at our code, and pass that struct to <code>dev_xdp_install()</code></li>
</ul>
<p>We can give both approaches a try. Livepatching has the nice property that something else sets up XDP for us (we just redirect the execution of the eBPF program), but the downside is that we still have to do the whole eBPF compile/load/verify ritual for a program that will never run. The second option is self contained in a Rust module, but the NIC might end up in a partially configured XDP state (e.g. I don't even know if <code>ip link</code> will report <code>xdp</code> being enabled).</p>
<p>Let's start with the first one. Regarding livepatching, we have at least a couple of options here as well: livepatch would be the cleaner production tool, but unfortunately it's disabled by default on NixOS, and ftrace is the livepatch underlying mechanism anyway, so we'll go with that.</p>
<p>So here's the plan: enable XDP on an interface, patch the dispatcher with ftrace, then rewrite the module in Rust.</p>
<h2 id="enabling-xdp">Enabling XDP</h2>
<p>As I mentioned, in order to enable XDP on a given interface we need to load an eBPF program, which in our case will be a dummy one that constantly returns <code>XDP_PASS</code> (easy):</p>
<pre><code>#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
SEC("xdp")
int xdp_prog(struct xdp_md *ctx)
{
return XDP_PASS;
}
char _license[] SEC("license") = "GPL";
</code></pre>
<p>Next we can enter the <code>xdp-dev</code> dev shell defined in a <code>flake.nix</code> that looks something like:</p>
<pre><code>{
description = "Rust XDP Nix dev shell";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
};
outputs = { nixpkgs, ... }:
let
system = "x86_64-linux";
pkgs = import nixpkgs { inherit system; };
in {
devShells.${system} = {
xdp-dev = pkgs.mkShell {
name = "xdp-dev";
buildInputs = with pkgs; [
iproute2
libbpf
linuxHeaders
llvmPackages.clang-unwrapped
llvmPackages.llvm
pkg-config
];
shellHook = ''
export CFLAGS="$CFLAGS -I$(pkg-config --variable=includedir libbpf)"
export CFLAGS="$CFLAGS -I${pkgs.linuxHeaders}/include"
'';
};
};
};
}
</code></pre>
<p>build/load the eBPF program:</p>
<pre><code>➜ xdp_prog git:(master) nix develop ..#xdp-dev -c $SHELL
➜ xdp_prog git:(master) clang -O2 -g -target bpf $CFLAGS -c xdp_prog.c -o xdp_prog.o
➜ xdp_prog git:(master) sudo ip link set dev wlp0s20f3 xdp obj xdp_prog.o sec xdp
</code></pre>
<p>and traffic, as expected, will be unaffected:</p>
<pre><code>64 bytes from 192.168.1.1: icmp_seq=31 ttl=64 time=3.19 ms
64 bytes from 192.168.1.1: icmp_seq=32 ttl=64 time=3.26 ms
</code></pre>
<p>But that's boring, and it requires us to pull in a full clang/LLVM toolchain just to put together an ELF with literally two instructions (<code>r0 = 2</code>, <code>exit</code>), so let's build it from scratch without external dependencies.</p>
<p>As mentioned, we need just a couple of instructions:</p>
<pre><code>let prog = [BpfInsn::mov32_imm(0, XDP_PASS), BpfInsn::exit()];
</code></pre>
<p>and a couple of BPF syscalls (and related struct definitions for the <code>BpfAttrProgLoad</code> and <code>BpfAttrLinkCreate</code> attributes) to load and attach the XDP program:</p>
<pre><code>Bpf::syscall(
Self::BPF_PROG_LOAD,
&attr as *const _ as *const c_void,
mem::size_of::<BpfAttrProgLoad>(),
)
..
Bpf::syscall(
Self::BPF_LINK_CREATE,
&link_attr as *const _ as *const c_void,
mem::size_of::<BpfAttrLinkCreate>(),
)
</code></pre>
<p>This is also a good chance to test the newer <code>BPF_LINK_CREATE</code> syscall rather than the usual netlink approach (ok, maybe not that new, as it's been around for ~6 years).</p>
<p>Next, we build it with <code>rustc</code> and we're ready to run it:</p>
<pre><code>➜ enable-xdp git:(master) nix develop nixpkgs#rustc -c $SHELL
➜ enable-xdp git:(master) rustc enable-xdp.rs
➜ enable-xdp git:(master) sudo ./enable-xdp wlp0s20f3
</code></pre>
<p>(I renamed the tool to <code>enable-xdp</code> so loading a dummy eBPF program just to toggle XDP can remain an implementation detail, we can pretend it never happened and we can move on with our day :D).</p>
<p>To quickly test if this thing is working, we can make it return <code>XDP_DROP</code>. And as we load it, we can see ingress traffic starts getting dropped:</p>
<pre><code>64 bytes from 192.168.1.1: icmp_seq=21 ttl=64 time=3.45 ms
64 bytes from 192.168.1.1: icmp_seq=22 ttl=64 time=3.10 ms
From 192.168.1.120 icmp_seq=46 Destination Host Unreachable
From 192.168.1.120 icmp_seq=47 Destination Host Unreachable
</code></pre>
<h2 id="lkm-on-nixos">LKM on NixOS</h2>
<p>Since I've never done LKM development in Rust, on NixOS, or tried ftrace, let's go step by step.</p>
<p>First let's make sure we can build an LKM in a non FHS system. Turns out the only tricky part was figuring out the right package and pointing our Makefile to the correct kernel dir. Here's the <code>kernel-dev</code> dev shell from <code>flake.nix</code>:</p>
<pre><code>kernel-dev = pkgs.mkShell {
name = "kernel-dev";
buildInputs = with pkgs; [
linux_latest.dev
];
shellHook = ''
export KERNEL_DIR=${pkgs.linux_latest.dev}/lib/modules/$(uname -r)/build
'';
};
</code></pre>
<p>then the usual Makefile:</p>
<pre><code>obj-m += xdp_ftrace_c.o
all:
$(MAKE) -C $(KERNEL_DIR) M=$(PWD) modules
</code></pre>
<p>a dummy module:</p>
<pre><code>#include <linux/module.h>
static int __init ftrace_hook_init(void) { return 0; }
static void __exit ftrace_hook_exit(void) {}
module_init(ftrace_hook_init);
module_exit(ftrace_hook_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("ftrace example");
</code></pre>
<p>and <code>make</code>:</p>
<pre><code>➜ xdp_ftrace_c git:(master) nix develop ..#kernel-dev -c $SHELL
➜ xdp_ftrace_c git:(master) make
..
LD [M] xdp_ftrace_c.ko
</code></pre>
<p>Great, we have a <code>.ko</code> object. Next stop: figuring out ftrace.</p>
<h2 id="meet-ftrace">Meet ftrace</h2>
<p>ftrace is a kernel framework that lets us hook into almost any kernel function (with a few exceptions like <code>__always_inline</code> and <code>notrace</code>).</p>
<p>A kernel with ftrace support is compiled with the <code>-mfentry</code> compiler option which adds a trampoline to the prologue of every traceable function: in practice just a <code>call __fentry__</code> so that all functions will jump to this instrumentation function which can then do various things (in userspace it can be used for example for profiling) and then we return back to the original function.</p>
<p>For performance reasons, as an extra call/ret pair for each function invocation adds quite some overhead, at runtime the kernel replaces all those <code>call __fentry__</code> with five NOP bytes. Then, when tracing is enabled, those NOPs get overwritten with the actual ftrace trampoline call only for the functions that need instrumentation.</p>
<p>For more details there are various talks from the author, like <a rel="external" href="https://www.youtube.com/watch?v=93uE_kWWQjs">this one</a>. It's interesting how it works on x86, as atomically overwriting 5 bytes in the text section requires a bit of a magical dance with multiple passes when dealing with SMP (as instructions can cross cache/page boundaries):</p>
<ul>
<li>The first of the five NOP bytes is overwritten with an <code>int3</code> breakpoint instruction (then a memory barrier syncs all CPUs on that). This breakpoint jumps to an ftrace handler that increments the IP by 5 bytes so the CPU is effectively back to executing the traced function as if nothing happened</li>
<li>Then the other 4 bytes are overwritten with the address of the ftrace trampoline (then another memory barrier)</li>
<li>Then the <code>int3</code> instruction, which is just a single byte, can be atomically replaced with the <code>call</code> instruction's opcode</li>
</ul>
<p>The talk mentions it took Intel engineers a few months to reluctantly accept this would work :D Anyway, we'll look a bit more into the details later on.</p>
<h3 id="ftrace-api">ftrace API</h3>
<p>First we need to define a filter that tells ftrace when to run our callback, basically specify what we want to instrument and which callback to run:</p>
<pre><code>static struct ftrace_ops
my_fops = {
.func = my_cb,
};
ftrace_set_filter(&my_fops, "some_func", 0, 0);
register_ftrace_function(&my_fops);
</code></pre>
<p>then we define the actual callback in which we can inspect registers, parent caller, do printk, go crazy etc.:</p>
<pre><code>static void notrace
my_cb(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) {
..
}
</code></pre>
<p>In addition to inspecting the function state through its registers, we can also edit them. Changing the instruction pointer is exactly what we need to do in order to hijack the execution of the XDP dispatcher to our own function. We can do that with:</p>
<pre><code>ftrace_regs_set_instruction_pointer(fregs, (unsigned long)some_func_hook);
</code></pre>
<p>where <code>some_func_hook</code> will have the same signature as the instrumented function.</p>
<p>But what are we going to instrument?</p>
<h3 id="finding-our-xdp-target">Finding our XDP target</h3>
<p>We now need to find a good target to patch, ideally the logic that only runs the bpf program. <code>bpf_prog_run_xdp</code> appears to be the entry point for both generic <code>SKB_MODE</code> and native <code>DRV_MODE</code>:</p>
<pre><code>static __always_inline u32 bpf_prog_run_xdp(const struct bpf_prog *prog,
struct xdp_buff *xdp)
{
...
u32 act = __bpf_prog_run(prog, xdp, BPF_DISPATCHER_FUNC(xdp));
</code></pre>
<p><code>BPF_DISPATCHER_FUNC</code> is just a macro that returns the name of the XDP dispatcher function:</p>
<pre><code>#define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_##name##_func
</code></pre>
<p>The actual dispatcher is defined using the <code>DEFINE_BPF_DISPATCHER</code> macro:</p>
<pre><code>#define DEFINE_BPF_DISPATCHER(name) \
__BPF_DISPATCHER_SC(name); \
noinline __bpfcall unsigned int bpf_dispatcher_##name##_func( \
const void *ctx, \
const struct bpf_insn *insnsi, \
bpf_func_t bpf_func) \
{ \
return __BPF_DISPATCHER_CALL(name); \
} \
</code></pre>
<p>and then called with <code>__BPF_DISPATCHER_CALL</code>:</p>
<pre><code>#define __BPF_DISPATCHER_CALL(name) \
static_call(bpf_dispatcher_##name##_call)(ctx, insnsi, bpf_func)
</code></pre>
<p>so our target function is <code>bpf_dispatcher_xdp_func</code>.</p>
<h2 id="putting-everything-together-in-c">Putting everything together (in C)</h2>
<p>We should now have everything to put together a C LKM that patches <code>bpf_dispatcher_xdp_func</code>.</p>
<p>Here's a slightly stripped down version (and here the <a rel="external" href="https://github.com/jibi/xdp-rust-lkm/blob/master/xdp_ftrace_c/xdp_ftrace_c.c">full version</a>):</p>
<pre><code>static noinline notrace __bpfcall unsigned int
xdp_force_pass(const void *ctx, const struct bpf_insn *insnsi, bpf_func_t bpf_func)
{
return XDP_PASS;
}
static void notrace bpf_dispatcher_xdp_func_cb(unsigned long ip, unsigned long parent_ip,
struct ftrace_ops *op, struct ftrace_regs *fregs)
{
ftrace_regs_set_instruction_pointer(fregs, (unsigned long)xdp_force_pass);
}
static struct ftrace_ops bpf_dispatcher_xdp_func_fops = {
.func = bpf_dispatcher_xdp_func_cb,
.flags = FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_RECURSION | FTRACE_OPS_FL_IPMODIFY,
};
static int __init ftrace_hook_init(void)
{
int ret;
ret = ftrace_set_filter(&bpf_dispatcher_xdp_func_fops, "bpf_dispatcher_xdp_func", 0, 0);
if (ret) {
return ret;
}
ret = register_ftrace_function(&bpf_dispatcher_xdp_func_fops);
if (ret) {
ftrace_set_filter(&bpf_dispatcher_xdp_func_fops, NULL, 0, 0);
return ret;
}
return 0;
}
</code></pre>
<p>let's build it and load it:</p>
<pre><code>➜ xdp_ftrace_c git:(master) sudo insmod ./xdp_ftrace_c.ko
</code></pre>
<p>and we can confirm that the ftrace livepatch is working by looking at the traffic that starts flowing again, as the dispatcher is now returning <code>XDP_PASS</code> rather than the eBPF program's <code>XDP_DROP</code>:</p>
<pre><code>From 192.168.1.120 icmp_seq=13 Destination Host Unreachable
From 192.168.1.120 icmp_seq=14 Destination Host Unreachable
64 bytes from 192.168.1.1: icmp_seq=21 ttl=64 time=2.13 ms
64 bytes from 192.168.1.1: icmp_seq=22 ttl=64 time=1.10 ms
</code></pre>
<h3 id="digging-into-ftrace-internals">Digging into ftrace internals</h3>
<p>Let's do a quick detour and peek into the ftrace internals: I wanted to see how the trampoline actually works and get a sense of the cost of doing livepatching.</p>
<p>Let's start by looking at the kernel object:</p>
<pre><code>➜ ~ gdb -q $(nix-store -q --outputs $(nix-store -q -d /run/booted-system/kernel) | rg dev -m1)/vmlinux
Reading symbols from /nix/store/b4kf7sphmydjzswhm3yyn90qj2a7v4iv-linux-6.19-dev/vmlinux...
(gdb) disassemble bpf_dispatcher_xdp_func
Dump of assembler code for function bpf_dispatcher_xdp_func:
0xffffffff81e2a2f0 <+0>: endbr64
0xffffffff81e2a2f4 <+4>: call 0xffffffff812e7bf0 <__fentry__>
</code></pre>
<p>after the <code>endbr64</code> instruction (which has nothing to do with ftrace, it just allows the function to be targeted by an indirect jump without faulting) we can see in the kernel image, as expected, that our target function has in its prologue the <code>call __fentry__</code> emitted by <code>gcc -mfentry</code>.</p>
<p>If we inspect the running image though, we should see that it got replaced with NOPs. Let's first get comfy with gdb by making it use the symbols for the live kernel:</p>
<pre><code>TEXT=0x$(sudo awk '/ _stext$/{print $1}' /proc/kallsyms)
VMLINUX=$(nix-store -q --outputs $(nix-store -q -d /run/booted-system/kernel) | rg dev -m1)/vmlinux
sudo gdb -q -c /proc/kcore -ex "set confirm off" -ex "add-symbol-file $VMLINUX $TEXT"
</code></pre>
<p>Here's the same function on a live kernel:</p>
<pre><code>(gdb) disassemble bpf_dispatcher_xdp_func
Dump of assembler code for function bpf_dispatcher_xdp_func:
0xffffffffa4a2a2f0 <+0>: endbr64
0xffffffffa4a2a2f4 <+4>: nopl 0x0(%rax,%rax,1)
</code></pre>
<p>we can see that the call has been patched by the kernel with NOPs, so there's virtually no cost.</p>
<p>After loading our kernel module, which enables the tracing callback for <code>bpf_dispatcher_xdp_func</code>, those NOPs get overwritten:</p>
<pre><code>TEXT=0x$(sudo awk '/ _stext$/{print $1}' /proc/kallsyms)
VMLINUX=$(nix-store -q --outputs $(nix-store -q -d /run/booted-system/kernel) | rg dev -m1)/vmlinux
MODTEXT=$(sudo cat /sys/module/xdp_ftrace_c/sections/.text)
MOD=~/xdp-rust-lkm/xdp_ftrace_c/xdp_ftrace_c.ko
sudo gdb -q -c /proc/kcore -ex "set confirm off" -ex "add-symbol-file $VMLINUX $TEXT" -ex "add-symbol-file $MOD $MODTEXT"
..
(gdb) disassemble bpf_dispatcher_xdp_func
Dump of assembler code for function bpf_dispatcher_xdp_func:
0xffffffffa4a2a2f0 <+0>: endbr64
0xffffffffa4a2a2f4 <+4>: call 0xffffffffc2b19000
</code></pre>
<p>if we check <code>0xffffffffc2b19000</code>, we'll see we jump to an ftrace trampoline: in our specific case, <a rel="external" href="https://elixir.bootlin.com/linux/v6.19-rc5/source/arch/x86/kernel/ftrace_64.S#L201"><code>ftrace_regs_caller</code></a> as we requested access to traced function registers (with <code>FTRACE_OPS_FL_SAVE_REGS</code>).</p>
<p>It's probably easier to follow the kernel sources, but in short the trampoline is saving some registers in the <code>ftrace_regs</code> struct:</p>
<pre><code>(gdb) x/40i 0xffffffffc2b19000
0xffffffffc2b19000: pushf
0xffffffffc2b19001: sub $0xa8,%rsp
0xffffffffc2b19008: mov %rax,0x50(%rsp)
0xffffffffc2b1900d: mov %rcx,0x58(%rsp)
0xffffffffc2b19012: mov %rdx,0x60(%rsp)
0xffffffffc2b19017: mov %rsi,0x68(%rsp)
0xffffffffc2b1901c: mov %rdi,0x70(%rsp)
0xffffffffc2b19021: mov %r8,0x48(%rsp)
0xffffffffc2b19026: mov %r9,0x40(%rsp)
0xffffffffc2b1902b: movq $0x0,0x78(%rsp)
0xffffffffc2b19034: mov %rbp,%rdx
0xffffffffc2b19037: mov %rdx,0x20(%rsp)
</code></pre>
<p>setting the <code>ip</code> (<code>%rdi</code>) and <code>parent_ip</code> (<code>%rsi</code>) callback args:</p>
<pre><code> 0xffffffffc2b1903c: mov 0xb8(%rsp),%rsi
0xffffffffc2b19044: mov 0xb0(%rsp),%rdi
0xffffffffc2b1904c: mov %rdi,0x80(%rsp)
0xffffffffc2b19054: sub $0x5,%rdi
0xffffffffc2b19058: cs nopl 0x0(%rax,%rax,1)
</code></pre>
<p>setting the <code>struct ftrace_ops</code> callback's arg (it's referenced relative to the <code>%rip</code> register, so this must be a trampoline generated dynamically?):</p>
<pre><code> 0xffffffffc2b19061: mov 0xf6(%rip),%rdx # 0xffffffffc2b1915e
</code></pre>
<p>saving some other registers and setting some other fields in the <code>ftrace_ops</code> struct:</p>
<pre><code> 0xffffffffc2b19068: mov %r15,(%rsp)
0xffffffffc2b1906c: mov %r14,0x8(%rsp)
0xffffffffc2b19071: mov %r13,0x10(%rsp)
0xffffffffc2b19076: mov %r12,0x18(%rsp)
0xffffffffc2b1907b: mov %r11,0x30(%rsp)
0xffffffffc2b19080: mov %r10,0x38(%rsp)
0xffffffffc2b19085: mov %rbx,0x28(%rsp)
0xffffffffc2b1908a: mov 0xa8(%rsp),%rcx
0xffffffffc2b19092: mov %rcx,0x90(%rsp)
0xffffffffc2b1909a: mov $0x18,%rcx
0xffffffffc2b190a1: mov %rcx,0xa0(%rsp)
0xffffffffc2b190a9: mov $0x10,%rcx
0xffffffffc2b190b0: mov %rcx,0x88(%rsp)
0xffffffffc2b190b8: lea 0xb8(%rsp),%rcx
0xffffffffc2b190c0: mov %rcx,0x98(%rsp)
0xffffffffc2b190c8: lea (%rsp),%rcx
0xffffffffc2b190cc: cs nopl 0x0(%rax,%rax,1)
</code></pre>
<p>and calling into the next function:</p>
<pre><code> 0xffffffffc2b190d5: call 0xffffffffa4092080 <ftrace_ops_assist_func>
</code></pre>
<p>which is another ftrace helper used mostly to deal with safe recursion of the tracing logic (as we requested with the <code>FTRACE_OPS_FL_RECURSION</code> flag). Eventually, this helper calls into a single instruction stub:</p>
<pre><code>(gdb) disassemble ftrace_ops_assist_func
Dump of assembler code for function ftrace_ops_assist_func:
..
0xffffffffa4092100 <+128>: mov (%rdx),%rax
0xffffffffa4092103 <+131>: call 0xffffffffc0401d70
</code></pre>
<p>that does an indirect <code>%rax</code> jump:</p>
<pre><code>(gdb) x/i 0xffffffffc0401d70
0xffffffffc0401d70: jmp *%rax
</code></pre>
<p>which, as we saw, contains our <code>ftrace_ops</code> struct (it was assigned to <code>%rdx</code> in <code>ftrace_regs_caller</code>) and the first field of this struct (<code>func</code>) is pointing exactly to our <code>bpf_dispatcher_xdp_func_cb</code> hook:</p>
<pre><code>(gdb) p **(struct ftrace_ops **) 0xffffffffc2b1915e
$1 = {func = 0xffffffffc2afe030 <bpf_dispatcher_xdp_func_cb>, next = 0xffff8e24c56ac400, flags = 6231, private = 0x0,
</code></pre>
<p>As we know, this function just calls <code>ftrace_regs_set_instruction_pointer</code> to set the saved <code>%rip</code> in the <code>ftrace_regs</code> struct (the third arg/<code>%rcx</code> register of the callback) to the address of the <code>xdp_force_pass</code> function:</p>
<pre><code>(gdb) disassemble bpf_dispatcher_xdp_func_cb
Dump of assembler code for function bpf_dispatcher_xdp_func_cb:
0xffffffffc2afe030 <+64>: endbr64
0xffffffffc2afe034 <+68>: movq $0xffffffffc2afe010,0x80(%rcx)
0xffffffffc2afe03f <+79>: xor %ecx,%ecx
0xffffffffc2afe041 <+81>: jmp 0xffffffffa4cacce0 <its_return_thunk>
..
(gdb) info symbol 0xffffffffc2afe010
xdp_force_pass in section .text of /home/jibi/xdp-rust-lkm/xdp_ftrace_c/xdp_ftrace_c.ko
</code></pre>
<p>then we jump to a <code>ret</code> instruction:</p>
<pre><code>(gdb) disassemble its_return_thunk
Dump of assembler code for function its_return_thunk:
0xffffffffa4cacce0 <+0>: ret
</code></pre>
<p>which returns to <code>ftrace_ops_assist_func</code> (since it got there via a <code>call</code> into the <code>jmp *%rax</code> trampoline), which then returns to the initial trampoline.</p>
<p>From there we restore all the registers from the <code>ftrace_regs</code> struct, overwriting <code>%rip</code> with the <code>xdp_force_pass</code> function's address, and effectively redirecting the control flow to our target function once we return from the trampoline:</p>
<pre><code>(gdb) x/30i 0xffffffffc2b19000+0xd5
0xffffffffc2b190d5: call 0xffffffffa4092080 <ftrace_ops_assist_func>
..
0xffffffffc2b190ea: mov 0x80(%rsp),%rax
0xffffffffc2b190f2: mov %rax,0xb0(%rsp)
..
0xffffffffc2b19151: add $0xa8,%rsp
0xffffffffc2b19158: popf
0xffffffffc2b19159: jmp 0xffffffffa4cacce0 <its_return_thunk>
</code></pre>
<p>To sum up: <code>bpf_dispatcher_xdp_func</code> -> <code>ftrace_regs_caller</code> trampoline -> <code>ftrace_ops_assist_func</code> helper -> <code>bpf_dispatcher_xdp_func_cb</code> (our hook) which overwrites the saved IP in <code>ftrace_regs</code> -> <code>ret</code> back to <code>ftrace_regs_caller</code> which restores registers and resumes execution at the address we patched (<code>xdp_force_pass</code>).</p>
<p>I have to admit I naively assumed the whole patching dance would be more straightforward, but after going through it, it makes sense that we want to preserve registers, be careful about recursion, and all the other things.</p>
<p>Now that we understand the trampoline and the cost model, let's port it to Rust.</p>
<h2 id="let-it-rust">Let it Rust</h2>
<p>Let's start by getting a dummy module to compile. We can take a look at the <a rel="external" href="https://github.com/Rust-for-Linux/rust-out-of-tree-module">templates</a> in the Rust for Linux repo:</p>
<pre><code>module! {
type: XdpFtrace,
name: "xdp_ftrace_rust",
authors: ["jibi"],
description: "Rust ftrace XDP POC",
license: "GPL",
}
struct XdpFtrace {}
impl kernel::Module for XdpFtrace {
fn init(_module: &'static ThisModule) -> Result<Self> {
pr_info!("Loading Rust ftrace XDP POC");
Ok(XdpFtrace {})
}
}
impl Drop for XdpFtrace {
fn drop(&mut self) {
pr_info!("Unloading Rust ftrace XDP POC");
}
}
</code></pre>
<p>Makefile is surprisingly identical to a C makefile:</p>
<pre><code>obj-m := xdp_ftrace_rust.o
</code></pre>
<p>so all that's left is to add a <code>rustc</code> to our previous <code>kernel-dev</code> Nix dev shell:</p>
<pre><code>buildInputs = with pkgs; [
linux_latest.dev
rustc-unwrapped
];
</code></pre>
<p>and our module will build just fine:</p>
<pre><code>➜ xdp_ftrace_rust git:(master) nix develop ..#kernel-dev -c $SHELL
➜ xdp_ftrace_rust git:(master) make
make -C /nix/store/b4kf7sphmydjzswhm3yyn90qj2a7v4iv-linux-6.19-dev/lib/modules/6.19.0/build M=/home/jibi/xdp-rust-lkm/xdp_ftrace_rust modules
..
LD [M] xdp_ftrace_rust.ko
</code></pre>
<h2 id="bindings-bindings-bindings">Bindings bindings bindings</h2>
<p>Now we need to call into ftrace from Rust:</p>
<pre><code>➜ linux-6.19 rg ftrace -i rust
➜ linux-6.19
</code></pre>
<p>Oh no.</p>
<p>Unfortunately there are no ftrace Rust bindings available, so we need to get a bit creative.</p>
<p>First option is to implement them by hand: add a bunch of <code>extern "C"</code> declarations for <code>ftrace_set_filter()</code>, <code>register_ftrace_function()</code>, etc. (easy but manual work I'd rather avoid) and rewrite all the relevant types needed by ftrace. That can get a bit hairy (euphemism) quite fast, due to inner structs, Linux kernel lists, #ifdefs, and the like.</p>
<p>So let's try to generate Rust bindings with <code>bindgen</code>. Here we also have a couple of options: we can create a dummy bindings.h file which includes ftrace.h, or we can rely on BTF symbols:</p>
<pre><code>➜ ~ bpftool btf dump file /sys/kernel/btf/vmlinux format c | rg 'struct ftrace_ops \{|ftrace_set_filter|register_ftrace_function'
struct ftrace_ops {
</code></pre>
<p>Unfortunately, while BTF does include function prototypes, <code>bpftool</code> only exposes them in the raw format, which means we would either have to parse and resolve type IDs by hand to recover the actual C signatures, or we'd need to write the function declarations by ourselves:</p>
<pre><code>extern "C" {
fn ftrace_set_filter(
ops: *mut ftrace_ops,
buf: *const c_char,
len: c_int,
reset: c_int,
) -> c_int;
fn register_ftrace_function(ops: *mut ftrace_ops) -> c_int;
fn unregister_ftrace_function(ops: *mut ftrace_ops) -> c_int;
}
</code></pre>
<p>although both could work, I'd prefer a generic solution that doesn't require me to manually parse <code>bpftool btf</code> or figure out/translate each time the signature of whatever function I need next, so we'll go with the <code>#include <linux/ftrace.h></code> approach.</p>
<p>Surely we can simply create a header file with:</p>
<pre><code>#include <linux/ftrace.h>
</code></pre>
<p>add bindgen to our Nix dev shell:</p>
<pre><code>buildInputs = with pkgs; [
linux_latest.dev
rust-bindgen-unwrapped
rustc-unwrapped
];
</code></pre>
<p>and bindgen will do its magic, right?</p>
<pre><code>➜ xdp_ftrace_rust git:(master) bindgen ./bindings.h
./bindings.h:4:10: fatal error: 'linux/ftrace.h' file not found
Unable to generate bindings: clang diagnosed error: ./bindings.h:4:10: fatal error: 'linux/ftrace.h' file not found
</code></pre>
<p>Almost. Let's move to the Makefile and start adding some includes, as the kernel headers are not part of the standard headers:</p>
<pre><code>BINDGEN_CLANG_FLAGS := -- \
-I$(KERNEL_SRC)/include \
-I$(KERNEL_SRC)/arch/$(ARCH)/include \
-I$(KERNEL_SRC)/include/uapi \
-I$(KERNEL_BUILD)/include \
-I$(KERNEL_BUILD)/arch/$(ARCH)/include/generated
</code></pre>
<p>should be much better now:</p>
<pre><code>➜ xdp_ftrace_rust git:(master) make $(pwd)/bindings.rs
bindgen /home/jibi/xdp-rust-lkm/xdp_ftrace_rust/bindings.h
/nix/store/b4kf7sphmydjzswhm3yyn90qj2a7v4iv-linux-6.19-dev/lib/modules/6.19.0/build/source/include/linux/compiler_attributes.h:55:9: warning: '__always_inline' macro redefined [-Wmacro-redefined]
/nix/store/b4kf7sphmydjzswhm3yyn90qj2a7v4iv-linux-6.19-dev/lib/modules/6.19.0/build/source/include/uapi/linux/stddef.h:10:9: note: previous definition is here
/nix/store/b4kf7sphmydjzswhm3yyn90qj2a7v4iv-linux-6.19-dev/lib/modules/6.19.0/build/source/include/asm-generic/rwonce.h:64:8: error: unknown type name '__no_sanitize_or_inline'
/nix/store/b4kf7sphmydjzswhm3yyn90qj2a7v4iv-linux-6.19-dev/lib/modules/6.19.0/build/source/include/asm-generic/rwonce.h:82:8: error: unknown type name '__no_sanitize_or_inline'
/nix/store/b4kf7sphmydjzswhm3yyn90qj2a7v4iv-linux-6.19-dev/lib/modules/6.19.0/build/source/include/linux/overflow.h:53:9: error: call to undeclared function 'unlikely'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
..
fatal error: too many errors emitted, stopping now [-ferror-limit=]
make: *** [Makefile:32: /home/jibi/xdp-rust-lkm/xdp_ftrace_rust/bindings.rs] Error 1
</code></pre>
<p>Never mind, let's cut this game. Here's the full Makefile an hour later (not sure anymore parsing by hand <code>bpftool btf .. format raw</code> was that bad of an idea):</p>
<p>making bindgen happy with <code>no_std</code>:</p>
<pre><code>BINDGEN_FLAGS := \
--use-core \
--ctypes-prefix core::ffi \
--no-layout-tests \
</code></pre>
<p>selecting only the symbols we actually need:</p>
<pre><code> --allowlist-type 'ftrace_ops' \
--allowlist-type '__arch_ftrace_regs' \
--allowlist-var 'FTRACE_OPS_FL_.*' \
--allowlist-function 'ftrace_set_filter' \
--allowlist-function 'register_ftrace_function' \
--allowlist-function 'unregister_ftrace_function' \
</code></pre>
<p>reassuring the compiler we know what we are doing:</p>
<pre><code> --raw-line '\#![allow(non_camel_case_types)]' \
--raw-line '\#![allow(non_snake_case)]' \
--raw-line '\#![allow(non_upper_case_globals)]' \
--raw-line '\#![allow(dead_code)]' \
--raw-line '\#![allow(unreachable_pub)]' \
--raw-line '\#![allow(unnecessary_transmutes)]' \
--raw-line '\#![allow(unsafe_op_in_unsafe_fn)]' \
--raw-line '\#![allow(improper_ctypes_definitions)]'
</code></pre>
<p>setting the C standard and enabling ms extensions for anonymous structs/unions:</p>
<pre><code>BINDGEN_CLANG_FLAGS := -- \
-std=gnu11 \
-fms-extensions \
</code></pre>
<p>including the kernel config to have all the features (ifdefs) needed enabled:</p>
<pre><code> -include $(KERNEL_SRC)/include/linux/kconfig.h \
-include $(KERNEL_BUILD)/include/generated/autoconf.h \
</code></pre>
<p>adding the usual headers:</p>
<pre><code> -I$(KERNEL_SRC)/include \
-I$(KERNEL_SRC)/arch/$(ARCH)/include \
-I$(KERNEL_SRC)/include/uapi \
-I$(KERNEL_BUILD)/include \
-I$(KERNEL_BUILD)/arch/$(ARCH)/include/generated \
</code></pre>
<p>kernel headers are expecting to know the module name for some reason:</p>
<pre><code> -DKBUILD_MODNAME=\"xdp_ftrace_rust\" \
-DKBUILD_BASENAME=\"xdp_ftrace_rust\" \
</code></pre>
<p>a bunch of defines we need that would usually be defined by the kernel makefiles:</p>
<pre><code> -D__KERNEL__ \
-D__TARGET_ARCH_$(ARCH) \
-DCC_USING_FENTRY \
</code></pre>
<p>and finally let's suppress some warnings:</p>
<pre><code> -Wno-unknown-attributes \
-Wno-ignored-attributes \
-Wno-duplicate-decl-specifier \
-Wno-address-of-packed-member \
-Wno-gnu-variable-sized-type-not-at-end \
-Wno-microsoft-anon-tag
</code></pre>
<p>Very (not) straightforward. Bindgen will now create a bindings.rs file that we can include in our module:</p>
<pre><code>mod bindings;
use bindings::*;
</code></pre>
<p>and we are ready to build our (still dummy for now) Rust kernel module with all the required ftrace bindings:</p>
<pre><code>➜ xdp_ftrace_rust git:(master) make
..
BTF [M] xdp_ftrace_rust.ko
</code></pre>
<p>so all that's left to do is port the existing C module to Rust.</p>
<h2 id="putting-everything-together-in-rust">Putting everything together (in Rust)</h2>
<p>First let's define a type for our <code>XdpHook</code>, the signature of the function that the XDP dispatcher calls:</p>
<pre><code>type XdpHook = unsafe extern "C" fn(
ctx: *const c_void,
insnsi: *const bpf_insn,
bpf_func: bpf_func_t,
) -> u32;
</code></pre>
<p>Then we need a couple of static variables:</p>
<pre><code>static mut FTRACE_OPS: ftrace_ops = unsafe { mem::zeroed() };
static XDP_HOOK: AtomicPtr<()> = AtomicPtr::new(null_mut());
</code></pre>
<p>although it's tempting to store it inside the <code>XdpFtrace</code> object, <code>FTRACE_OPS</code> needs to be static, as ftrace requires that. Then we have <code>XDP_HOOK</code>, which is just a pointer to our hook. It's not strictly needed, but it saves us from hardcoding the hook in the ftrace callback (and so must be protected with <code>AtomicPtr</code>).</p>
<p>Then we implement the remaining methods for the <code>XdpFtrace</code> object.</p>
<p>The constructor just sets the static <code>XDP_HOOK</code> with address of the <code>XdpHook</code> function we want to invoke, and calls <code>register()</code> to initialize ftrace:</p>
<pre><code>impl XdpFtrace {
const TARGET_FUNC: &[u8] = b"bpf_dispatcher_xdp_func\0";
fn install_hook(xdp_hook: XdpHook) -> Result<()> {
XDP_HOOK.store(xdp_hook as *mut (), SeqCst);
Self::register()
}
</code></pre>
<p><code>register</code> is just a 1:1 Rust translation of its C counterpart:</p>
<pre><code> fn register() -> Result<()> {
let ops = &raw mut FTRACE_OPS;
unsafe {
(*ops).func = Some(Self::bpf_dispatcher_xdp_func_cb);
(*ops).flags = (FTRACE_OPS_FL_SAVE_REGS
| FTRACE_OPS_FL_RECURSION
| FTRACE_OPS_FL_IPMODIFY) as c_ulong;
}
let ret = unsafe { ftrace_set_filter(ops, Self::TARGET_FUNC.as_ptr() as *mut u8, 0, 0) };
if ret != 0 {
pr_err!("ftrace_set_filter failed: {}\n", ret);
return Err(Error::from_errno(ret));
}
let ret = unsafe { register_ftrace_function(ops) };
if ret != 0 {
pr_err!("register_ftrace_function failed: {}\n", ret);
unsafe { ftrace_set_filter(ops, null_mut(), 0, 0) };
return Err(Error::from_errno(ret));
}
Ok(())
}
</code></pre>
<p>same for <code>unregister</code>:</p>
<pre><code> fn unregister() {
let ops = &raw mut FTRACE_OPS;
unsafe {
unregister_ftrace_function(ops);
ftrace_set_filter(ops, null_mut(), 0, 0);
}
</code></pre>
<p>the <code>bpf_dispatcher_xdp_func_cb</code> ftrace callback is also mostly a 1:1 port of the C callback, with the added logic for loading the <code>XDP_HOOK</code> pointer (and since <code>ftrace_regs_set_instruction_pointer</code> is a tiny macro I just expanded it by hand rather than defining a wrapper in a separate <code>.c</code> file, but yes, this isn't portable):</p>
<pre><code> unsafe extern "C" fn bpf_dispatcher_xdp_func_cb(
_ip: c_ulong,
_parent_ip: c_ulong,
_op: *mut ftrace_ops,
fregs: *mut ftrace_regs,
) {
let hook_ptr = XDP_HOOK.load(SeqCst);
if hook_ptr.is_null() {
return;
}
unsafe {
let fregs = fregs as *mut __arch_ftrace_regs;
(*fregs).regs.ip = hook_ptr as usize as c_ulong;
}
}
</code></pre>
<p>then a simple Rust module implementation to wire everything together:</p>
<pre><code>impl kernel::Module for XdpFtrace {
fn init(_module: &'static ThisModule) -> Result<Self> {
Self::install_hook(Self::xdp_force_pass)?;
Ok(XdpFtrace)
}
}
impl Drop for XdpFtrace {
fn drop(&mut self) {
Self::unregister();
}
}
</code></pre>
<p>and finally the actual hook, the reason for all of this: the arbitrary logic that we want to run in XDP without asking the verifier for permission.</p>
<p>In this case, just to show everything is working, we overwrite the loaded eBPF program (which drops all traffic) with something that constantly returns <code>XDP_PASS</code>:</p>
<pre><code>impl XdpFtrace {
unsafe extern "C" fn xdp_force_pass(_: *const c_void, _: *const bpf_insn, _: bpf_func_t) -> u32 {
xdp_action_XDP_PASS as u32
}
}
</code></pre>
<p>Let's add the last few missing bindings to the Makefile:</p>
<pre><code> --allowlist-type 'xdp_buff' \
--allowlist-type 'bpf_func_t' \
--allowlist-type 'xdp_action' \
--allowlist-var 'XDP_.*' \
--allowlist-var 'xdp_action_.*' \
</code></pre>
<p>and after building and loading the Rust module, we should see (if we have <code>enable-xdp</code> loading an <code>XDP_DROP</code> program) that traffic goes back to flowing:</p>
<pre><code>From 192.168.1.120 icmp_seq=4 Destination Host Unreachable
From 192.168.1.120 icmp_seq=5 Destination Host Unreachable
64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=1.13 ms
64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=1.45 ms
</code></pre>
<h2 id="what-about-dev-xdp-install">What about dev_xdp_install</h2>
<p>ftrace was fun, but as I mentioned it requires <em>enabling</em> XDP first. That means loading a dummy BPF program and going through the verifier just to put the NIC in XDP mode, even though the eBPF program never actually runs (as we replace it with our own function).</p>
<p>Now that we know how to build a Rust LKM and deal with missing bindings, it should be easy (tm) to do this without a dummy program or dispatcher live patch. The idea is to keep everything self contained in a single module and call <code>dev_xdp_install()</code> directly (the same function used by netlink / <code>BPF_LINK_CREATE</code>) but with our own <code>bpf_prog</code> and a <code>bpf_func</code> pointer that points to our code.</p>
<p>Since the reader is now an expert on running Rust LKM modules/bindings etc., I'll keep this part shorter.</p>
<p>At a high level we need to:</p>
<ul>
<li>Grab a reference to the NIC with <code>dev_get_by_index()</code></li>
<li>Resolve <code>dev_xdp_install</code> (and in generic mode also <code>generic_xdp_install</code>) with a <del>small hack</del> kprobe symbol lookup, since they're <code>static</code>/not exported</li>
<li>Figure out the correct <code>bpf_ndo</code> function for our NIC (depending also on the <code>SKB</code> or <code>DRV</code> mode)</li>
<li>Allocate a minimal <code>struct bpf_prog</code> and point <code>bpf_func</code> to our hook</li>
<li>Wire everything together by calling <code>dev_xdp_install()</code></li>
<li>Keep an eye on three different kinds of locks/refcounts that need to be released when something fails or on detach</li>
</ul>
<p>So here's a stripped-down version without <code>DRV</code> mode, detach logic, and a couple of other details (full version <a rel="external" href="https://github.com/jibi/xdp-rust-lkm/blob/master/dev_xdp_install/dev_xdp_install_main.rs">here</a>):</p>
<pre><code>fn lookup_sym(name: *const c_char) -> *mut c_void {
let mut addr: *mut c_void = null_mut();
let mut kp: kprobe = unsafe { mem::zeroed() };
kp.symbol_name = name as *const c_char;
if unsafe { register_kprobe(&mut kp) == 0 } {
addr = kp.addr as *mut c_void;
unsafe { unregister_kprobe(&mut kp) };
}
addr
}
impl DevXdpInstall {
fn get_dev(ifindex: i32) -> Result<*mut net_device> {
let dev = unsafe { dev_get_by_index(&raw mut init_net, ifindex) };
if dev.is_null() {
return Err(ENODEV);
}
Ok(dev)
}
fn get_bpf_op(dev: *mut net_device) -> Result<BpfOpFn> {
let sym = lookup_sym(b"generic_xdp_install\0".as_ptr() as *const c_char);
if sym.is_null() {
return Err(Error::from_errno(-(ENOSYS as i32)));
}
Ok(unsafe { mem::transmute(sym) })
}
fn get_dev_xdp_install() -> Result<DevXdpInstallFn> {
let sym = lookup_sym(b"dev_xdp_install\0".as_ptr() as *const c_char);
if sym.is_null() {
return Err(Error::from_errno(-(ENOSYS as i32)));
}
Ok(unsafe { mem::transmute(sym) })
}
fn alloc_prog(&mut self) -> Result<()> {
let bpf_prog = unsafe { bpf_prog_alloc(mem::size_of::<bpf_prog>() as u32, 0) };
if bpf_prog.is_null() {
return Err(ENOMEM);
}
unsafe {
bpf_prog_add(bpf_prog, 1);
..
(*bpf_prog).bpf_func = self.xdp_fn;
}
self.bpf_prog = bpf_prog;
Ok(())
}
fn do_dev_xdp_install(&mut self, attach: bool) -> Result<()> {
let mode = ..
let flags = ..
let err = unsafe {
(self.dev_xdp_install)(self.dev, mode, self.bpf_op, null_mut(), flags, self.bpf_prog)
};
if err != 0 {
return Err(Error::from_errno(err));
}
Ok(())
}
fn attach_xdp(&mut self) -> Result<()> {
self.alloc_prog().inspect_err(|_| {
unsafe { netdevice_dev_put(self.dev) };
})?;
self.do_dev_xdp_install(true).inspect_err(|_| {
unsafe { netdevice_dev_put(self.dev) };
unsafe { bpf_prog_put(self.bpf_prog) };
})?;
Ok(())
}
}
impl kernel::Module for DevXdpInstall {
fn init(_module: &'static ThisModule) -> Result<Self> {
let _rtnl = RtnlGuard::lock();
let ifindex = *module_parameters::ifindex.value();
let dev = Self::get_dev(ifindex)?;
let dev_xdp_install = Self::get_dev_xdp_install().inspect_err(|_| {
unsafe { netdevice_dev_put(dev) };
})?;
let bpf_op = Self::get_bpf_op(dev).inspect_err(|_| {
unsafe { netdevice_dev_put(dev) };
})?;
let mut state = DevXdpInstall {
dev,
dev_xdp_install,
bpf_op,
xdp_fn: Some(xdp_fn),
bpf_prog: null_mut(),
};
state.attach_xdp()?;
Ok(state)
}
}
</code></pre>
<p>This time we can test it in the opposite way: make the module install a function that always drops traffic, and after loading it we should see no more traffic flowing in.</p>
<h2 id="fun-with-the-verifier">Fun with the verifier</h2>
<p>This write up was mostly about the <em>ability</em> to run arbitrary code in the XDP path. Goal achieved, fun over. But we can't party all the time, so before wrapping it up, let's show some <em>value</em>.</p>
<p>Here's one example of something we cannot run in XDP with eBPF (or at least, not without expressing it differently, constraining it, or forcing it to use helpers).</p>
<p>The snippet below scans a packet, looks for the <code>0xcafecafe</code> bytes, and if it finds them it returns <code>XDP_DROP</code> to drop the packet. It's based on <a rel="external" href="https://github.com/aya-rs/aya">aya-rs</a>, so we can build our eBPF program in Rust (hence the slightly different function signature):</p>
<pre><code>#[xdp]
pub fn test(ctx: XdpContext) -> u32 {
let data = ctx.data() as *const u8;
let data_end = ctx.data_end() as *const u8;
let packet_len = (data_end as usize) - (data as usize);
for offset in 0..packet_len.saturating_sub(4-1) {
if unsafe { core::ptr::read_unaligned(data.add(offset) as *const u32) } == 0xcafecafe {
return xdp_action::XDP_DROP;
}
}
xdp_action::XDP_PASS
}
</code></pre>
<p>There's probably nothing too contentious about this example (besides its actual utility) but if we load it, the verifier will complain with what's likely the most common error message for anyone getting started with eBPF:</p>
<pre><code>invalid access to packet, off=0 size=4, R5(id=0,off=0,r=0)
R5 offset is outside of the packet
</code></pre>
<p>Here the verifier is trying to ensure, as part of its safety checks, that our program won't read/write outside the bounds of the packet. Allowing arbitrary kernel reads/writes would be bad, as that could panic the kernel, trigger page faults in softirq context, leak kernel memory to unprivileged userspace programs through maps, allow privilege escalation etc.</p>
<p>The error may sound confusing as our logic appears fine, so why can't the verifier prove that the access to the packet is safe? The catch is <code>packet_len</code>: once we turn <code>data_end - data</code> into a scalar, the verifier loses the direct relationship between <code>data</code> and <code>data_end</code>, so it can't conclude that <code>data + offset + 4 <= data_end</code>.</p>
<p>The only thing it can recognize is an explicit pointer to pointer bounds check:</p>
<pre><code> if (data + offset + len > data_end)
return ...
// access up to data + offset + len
</code></pre>
<p>To make sense of that, we need to look at how the <a rel="external" href="https://elixir.bootlin.com/linux/v6.19/source/kernel/bpf/verifier.c">verifier</a> works:</p>
<ul>
<li>As part of its various checks, the verifier goes through all instructions</li>
<li>When inspecting a conditional jump, it calls <code>check_cond_jmp_op()</code>, which among other things calls <code>try_match_pkt_pointers()</code></li>
<li>That tries to recognize a packet bounds comparison by checking the <code>type</code> metadata of the two registers used by the conditional jump: one must be <code>PTR_TO_PACKET</code> (a packet pointer with tracked offset/range metadata, i.e. not necessarily the original <code>ctx->data</code>) and the other <code>PTR_TO_PACKET_END</code>. Which one is src/dst depends on the jump opcode</li>
<li>Then <code>find_good_pkt_pointers()</code> is called on the successful branch to update <code>reg->range</code> to track how many bytes from <code>PTR_TO_PACKET</code> are safe to access while <code>mark_pkt_end()</code> gets called on the other branch to close the range</li>
<li>Next, when there's an actual packet access, <code>check_mem_access()</code> -> <code>check_packet_access()</code> will enforce that the access stays within <code>reg->range</code>, otherwise the program is rejected</li>
</ul>
<p>So if we add an explicit pointer bounds check:</p>
<pre><code> for offset in 0.. {
if unsafe { data.add(offset + 4) } > data_end {
break;
}
if unsafe { core::ptr::read_unaligned(data.add(offset) as *const u32) } == 0xcafecafe {
return xdp_action::XDP_DROP;
}
}
</code></pre>
<p>the verifier will be happy about packet boundaries and we can move on to...</p>
<p>the next error :D (probably more interesting):</p>
<pre><code>The sequence of 8193 jumps is too complex.
</code></pre>
<p>This time this isn't about memory safety but rather termination. As a matter of fact, the verifier has to prove that the program converges: every possible execution path must eventually reach an <code>exit</code> instruction (hint: loops are where that gets tricky).</p>
<p>To prove termination, the verifier does a DFS over all possible execution paths and tracks a state (registers, stack slots, packet bounds, etc.) per instruction. When it hits a conditional jump it can't resolve, it explores one branch and calls <code>push_stack()</code> on the other to save it on the DFS stack. Then, when it reaches an <code>exit</code> instruction, it pops the next state from the stack and resumes its visit from there.</p>
<p>In (CS) theory, deciding whether arbitrary code halts is undecidable (:wave: halting problem), so the verifier has to be conservative and rely on heuristics and hard limits (like the 1M instructions or the 8k jump sequence limits) that show up as constraints we need to follow when writing eBPF.</p>
<p>If we look at the bytecode that's being rejected:</p>
<pre><code>➜ aya-test llvm-objdump -S --no-show-raw-insn $(find | rg 'target/.*/out/test$') | rg -v ';'
..
0: w2 = *(u32 *)(r1 + 0x4)
1: w1 = *(u32 *)(r1 + 0x0)
2: r1 += 0x4
3: r3 = 0xcafecafe ll
5: r0 = 0x2
6: if r1 > r2 goto +0x4 <test+0x58>
7: r0 = 0x1
8: w4 = *(u32 *)(r1 - 0x4)
9: r1 += 0x1
10: if r4 != r3 goto -0x6 <test+0x28>
11: exit
</code></pre>
<p>and the related execution flow:</p>
<pre><code>┌───────────────────────────┐
│ 0: w2 = *(u32 *)(r1 + 0x4)│
│ 1: w1 = *(u32 *)(r1 + 0x0)│
│ 2: r1 += 0x4 │
│ 3: r3 = 0xcafecafe ll │
│ 5: r0 = 0x2 ◄──────┐
│ 6: if r1 > r2 goto +0x4 │ │
└┬──┬───────────────────────┘ │
T F │
│ │ │
│ ┌▼──────────────────────────┐ │
│ │ 7: r0 = 0x1 │ │
│ │ 8: w4 = *(u32 *)(r1 - 0x4)│ │
│ │ 9: r1 += 0x1 │ │
│ │10: if r4 != r3 goto -0x6 ├─T─┘
│ └┬──────────────────────────┘
│ F
│ │
┌▼──▼───────────────────────┐
│11: exit │
└───────────────────────────┘
</code></pre>
<p>We can see two conditional jumps: the bounds guard we just added (6), and the back edge that brings us back to the beginning of the loop when the <code>0xcafecafe</code> match fails (10). The latter is what triggers the verifier failure, as the verifier can't assume anything about packet contents, and it also can't prove that reaching instruction 5 again is equivalent to a previously visited state as <code>r1</code> keeps advancing and <code>r4</code> depends on unknown packet data.</p>
<p>So pruning (i.e. skipping visiting a state that the verifier considers equivalent to one already visited) doesn't kick in, and the DFS keeps exploring more and more iterations of the loop, each time pushing the fallthrough branch to the stack, until it hits the jump sequence limit, which is why we get the 8193 error.</p>
<p>High level, the visit looks something like:</p>
<ul>
<li>visit 0..6</li>
<li>push the fallthrough case (7) and take the exit branch (11)</li>
<li>reach <code>exit</code>, pop 7</li>
<li>visit 7..10</li>
<li>push the fallthrough (11), take the back edge (5)</li>
<li>repeat until the jump sequence limit is hit</li>
</ul>
<p>Now that we know exactly what's triggering the failure, we can look for workarounds. For example we can put a hard cap on the number of iterations:</p>
<pre><code> for offset in 0..1500 {
if unsafe { data.add(offset + 4) } > data_end {
break;
}
if unsafe { core::ptr::read_unaligned(data.add(offset) as *const u32) } == 0xcafecafe {
return xdp_action::XDP_DROP;
}
}
</code></pre>
<p>This works, but it's not as nice as iterating on the actual packet length as we're guessing a maximum. <code>1500</code> is a reasonable MTU-ish cap, but it also means we'll stop scanning early on larger frames.</p>
<p>And we can't just set it to something huge either. Even with a hard upper bound, each extra iteration forces the verifier to explore at least one more unresolved branch (match vs no-match), and eventually we hit <code>BPF_COMPLEXITY_LIMIT_JMP_SEQ</code> (8192). For example <code>0..8192</code> still trips the verifier:</p>
<pre><code>The sequence of 8193 jumps is too complex.
</code></pre>
<p>We can look into splitting (strip-mining) the loop into smaller chunks:</p>
<pre><code>const CHUNK_SIZE: usize = 5000;
const CHUNKS: usize = 2;
#[xdp]
pub fn test(ctx: XdpContext) -> u32 {
let data = ctx.data() as *const u8;
let data_end = ctx.data_end() as *const u8;
for chunk in 0..CHUNKS {
for offset in (chunk * CHUNK_SIZE)..((chunk + 1) * CHUNK_SIZE) {
if unsafe { data.add(offset + 4) } > data_end {
return xdp_action::XDP_PASS;
}
if unsafe { core::ptr::read_unaligned(data.add(offset) as *const u32) } == 0xcafecafe {
return xdp_action::XDP_DROP;
}
}
}
xdp_action::XDP_PASS
}
</code></pre>
<p>this also works, but we lose a bit in readability.</p>
<p>Alternatively we can use the <code>bpf_loop()</code> helper:</p>
<pre><code>#[repr(C)]
struct LoopCtx {
data: *const u8,
data_end: *const u8,
found: u32,
}
unsafe extern "C" fn scan_cb(index: u32, ctx: *mut c_void) -> c_long {
let ctx = unsafe { &mut *(ctx as *mut LoopCtx) };
let data = ctx.data as *const u8;
let data_end = ctx.data_end as *const u8;
let offset = (index & 0x7FFF) as usize;
if unsafe { data.add(offset + 4) } > data_end {
return 1;
}
if unsafe { core::ptr::read_unaligned(data.add(offset) as *const u32) } == 0xcafecafe {
ctx.found = 1;
return 1;
}
0
}
..
let mut loop_ctx = LoopCtx {
data,
data_end,
found: 0,
};
unsafe {
bpf_loop(
packet_len as u32,
scan_cb as *mut c_void,
&mut loop_ctx as *mut _ as *mut c_void,
0,
);
}
</code></pre>
<p>that also works, but we end up with a lot of callback and related context boilerplate just to run a loop.</p>
<p>So pick your poison: raw/chunked loops with hard limits, <code>bpf_loop()</code>s (<del>batteries</del> boilerplate included), or the Rust LKM hook 'n' pray hack.</p>
<h2 id="other-terrible-ideas-and-closing-words">Other terrible ideas and closing words</h2>
<p>The previous example was mostly about a verifier failure and the tradeoffs/workarounds it forces. But excluding eBPF from the equation isn't just about the verifier. It means we can enjoy other terrible practices: call kernel functions we are not supposed to, even <code>static</code> ones we are <em>really, really</em> not supposed to (yes, there are eBPF <code>kfuncs</code>, but those are much more limited), or allocate kernel memory dynamically (only <code>GFP_ATOMIC</code> though, remember we are in softirq/BH context), or implement our own mechanism to share data with userspace, just for fun.</p>
<p>Do we <em>really</em> need all this flexibility? I'll leave that to you.</p>
<p>I'm not taking sides as this was mostly an excuse to play with Rust LKMs, you can either have portable, safe kernel code that runs on pretty much every Linux version without even recompiling the bytecode, or more efficient/readable/boilerplate-free logic that can do many more things, including panicking your kernel.</p>
<p>Anyway, fun is over for real now. Hopefully writing Rust LKMs feels a bit less arcane!</p>
<h2 id="the-picture">The picture</h2>
<p><img src="https://jibi.io/blog/xdp-rust-lkm/dolomites.png" alt="Dolomites" /></p>
<p>View of the Rosengarten (Catinaccio) group in the Dolomites from the Lungotalvera promenade in Bolzano (Bozen). 10/10 chill writing spot, would recommend for extra inspiration.</p>