Restart ServerDBInfo broadcasts after worker failures - #13988

Before this PR

The cluster controller distributes ServerDBInfo through a relay tree. A simplified version of the failure we encountered looked like this:

cluster controller
└── stateless-0 (live relay)
    └── storage-0 (failed)

The failure followed this timeline:

1. The cluster controller detects that storage-0 is unavailable.

   workerAvailabilityWatch() uses waitFailureClient() to require the failure to remain continuous for SERVER_KNOBS->WORKER_FAILURE_TIME (default: 1 second) before removing the worker. During that second, storage-0 remains in id_worker.

2. ServerDBInfo changes during that one-second window.

   dbInfoUpdater() builds the broadcast target list from id_worker. Because storage-0 has not been removed yet, the new relay tree includes it. In this example, the cluster controller sends the broadcast to stateless-0 and assigns storage-0 as one of its children.

   If the broadcast had started after WORKER_FAILURE_TIME elapsed, storage-0 would already have been removed and would not have appeared in the tree.

3. The cluster controller removes storage-0.

   The one-second failure interval expires and removeFailedWorker() removes storage-0 from id_worker. Before this PR, removal did not interrupt the active broadcast, whose target list still contained storage-0.

4. stateless-0 independently detects the same failure.

   The request from the cluster controller to stateless-0 remains healthy, so the cluster controller continues waiting for the relay. The relay must discover that its child is unreachable using its own failure monitor:

   • Its transport waits for FLOW_KNOBS->CONNECTION_MONITOR_TIMEOUT (default: 2 seconds outside simulation) before marking the connection failed.
   • getReplyUnlessFailedFor() then requires the child to remain failed for SERVER_KNOBS->DBINFO_FAILED_DELAY (default: 1 second).

5. The broadcast finishes approximately three seconds after it started.

   Only after those relay-local failure delays can stateless-0 report the failed endpoint to the cluster controller. This delayed transaction-server initialization even though the cluster controller had already removed the worker blocking the relay.

After this PR

The first part of the timeline is unchanged:

1. The cluster controller detects the failed connection.
2. storage-0 remains in id_worker during WORKER_FAILURE_TIME.
3. A ServerDBInfo broadcast that starts during this window may initially include storage-0.

The behavior changes when the one-second failure interval expires:

4. Removing storage-0 interrupts the active broadcast.

   removeFailedWorker() now triggers updateDBInfo. The existing interruption path in dbInfoUpdater():

   • Ends the cluster controller's local wait for the active round.
   • Requeues the endpoints from that round.
   • Filters endpoints belonging to removed workers.
   • Starts a replacement broadcast after SERVER_KNOBS->DBINFO_BATCH_DELAY (default: 0.1 seconds).

5. The replacement broadcast excludes storage-0.

   The cluster controller no longer waits for stateless-0 to spend approximately two seconds detecting the failed connection followed by another second of DBINFO_FAILED_DELAY.

Requests already delivered to the original relay tree are not remotely cancelled. stateless-0 may continue its old relay work, but that work no longer gates the replacement broadcast or recovery.

The steady-state path is unchanged because the new trigger only runs after the cluster controller removes a worker.

Possible downsides

Restarting a broadcast can duplicate work. Requests already delivered to relays may continue while the replacement round sends the same, idempotent ServerDBInfo update. Staggered worker failures can therefore increase network and CPU usage, and sustained worker churn can repeatedly restart the cluster controller's local tracking of the broadcast.

A worker removal that was not blocking the active broadcast can also cause an unnecessary replacement round. DBINFO_BATCH_DELAY coalesces removals that happen close together, and already-delivered requests continue running.

Testing

• fdbserver_clustercontroller_test: 33 passed, 0 failed.
• The focused Debug target rebuilt successfully.
• git diff --check passed.

No new unit test is included. A direct unit test would only assert that the new trigger wakes dbInfoUpdater; it would not reproduce the independent failure-monitor state that caused the three-second delay.