What happened

mainnet-node-1 (genesis) panicked on 2026-06-10 11:34:30 UTC (running release/mainnet/3.0.2 @ e30f336af) and was restarted by pm2:

Message:  attempt to subtract with overflow
Location: crates/p2p/src/rate_limiting.rs:232

Reached from inbound gossip traffic: POST /gossip/pull_data → handle_get_data_sync → DataRequestTracker::check_request → cleanup_expired_entries.

Root cause

The tracker stamps per-peer timestamps (window_start, last_request, last_cleanup) from the non-monotonic wall clock (SystemTime::now() as epoch millis) and computes elapsed time by unsigned subtraction (now - stored). Because the workspace sets overflow-checks = true in [profile.release], an underflow is a hard panic in every shipped profile.

A stored timestamp can exceed the captured now via either mechanism:

1. Concurrent insert during cleanup iteration (most likely trigger): cleanup_expired_entries captures now, then iterates the DashMap (not a snapshot). A concurrent check_request inserts a fresh record stamped a moment after the captured now; the iterator visits it and now - window_start underflows. Rare per-event, inevitable over time on a busy node.
2. Backward wall-clock movement (NTP correction, VM clock sync).

Fix

saturating_sub at all four subtraction sites (is_window_expired, is_duplicate_request, cleanup_expired_entries, cleanup_if_needed). An out-of-order timestamp now reads as elapsed 0 — "just happened / not yet aged / not expired" — which is the safe, intended behavior at each call site. No rate-limiting thresholds, windows, or signatures change.

Regression tests were written first and confirmed to panic with attempt to subtract with overflow at the exact production lines on the unfixed code; they pass after the fix.

Note: this is a surgical hotfix — the proper fix is PR #1402

#1402 (fix: clock drift/monotonic timers) is the proper fix: it rewrites this tracker to use the monotonic std::time::Instant (whose arithmetic saturates by design), eliminating the wall-clock dependency entirely. That PR is open against master and should be revived and landed there. This PR is the minimal, low-risk hotfix appropriate for the release line so deployed nodes stop panicking now; #1402 supersedes it on master when it lands.

Per the shared-code hotfix process, this fix originates on release/3.x and must be cherry-picked up to master after landing (or arrive there via #1402's rewrite).

Verification

• cargo test -p irys-p2p --lib rate_limiting — 9/9 pass on this branch (new tests red before the fix)
• cargo build -p irys-p2p --profile debug-release — the overflow-checks profile builds clean
• cargo fmt / cargo clippy -p irys-p2p --tests --all-targets — clean

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• Bug Fixes

  • Improved rate limiting stability to safely handle edge cases when system timestamps are unusual or adjusted.

• Tests

  • Added test coverage for rate limiting behavior under backward clock and future timestamp scenarios.