[Problem RFC] Node Routing Table Wipeout #794

Hello, I have been working on the evaluation task and would like to propose a new benchmark problem.

This problem is inspired by an incident at Datadog in 2023, where an unintended OS update caused systemd-networkd to wipe out all routing table rules. Since the CNI adds these rules only during the startup of pods on the node, they cannot be restored automatically. This leaves the pods on the affected node isolated from the rest of the cluster.
Reference: https://www.datadoghq.com/blog/engineering/2023-03-08-deep-dive-into-platform-level-impact/

This failure is interesting because standard kubectl tools show a very limited amount of information. In my testing with the social-network app, only two pods seemed unresponsive to the readinessProbe and one eventually crashed, while others were seemingly fine. The main diagnostic route I discovered was the Prometheus probe_success metric. Probes are unsuccessful for pods on the node with the flushed routing table.

In my implementation, I plan to simulate the problem simply by deleting the routing table entries on a working node, without introducing the other complexities of the original incident. The expectation is for the agent to somehow restore these entries and get everything running again.

I have somewhat limited experience with these things, so I don't yet have a clear idea of the exact implementation details. I wanted to post this here to get some feedback and confirm if this problem is a viable addition to SREGym before continuing with the implementation.