Repository navigation
test(inkless:switch): hold each switch step before injecting faults [KC-838] - #885
Open
giuseppelillo wants to merge 2 commits into
Open
giuseppelillo wants to merge 2 commits into
giuseppelillo wants to merge 2 commits into
Conversation
…KC-838] The Category B fault tests injected the fault right after the config change, so the protocol step that the fault hit was a race. They also used a degraded network and 30 partitions to stretch the steps, which made them the slowest switch tests. `test_fault_before_control_plane_init` replaces them. A network partition holds step 3 in `WaitingForReplication`, and a Postgres outage after the seal commit holds step 5 in `AwaitingMetadata`. Each run injects its fault inside a hold, asserts that the hold is still in place unless the fault ends it, and then releases it. The runs cover a follower and a leader crash, a controller failover before the seal commit, a frozen leader, and a rolling restart while Postgres is down. Category D runs also cover the remaining client and operator paths: - `controller_failover_at_phase_2` kills the leader of partition 0 while its flag request is pending. - `reassignment` moves a follower in phase 2 under load, then reads the partition through the new replica alone. - `object_storage_outage` resumes a consumer group before the switch, in phase 1, and in phase 2, and reads its committed offsets back. It checks `ListOffsets` for the latest, earliest, timestamp, and max-timestamp specs in phase 1 and through each broker in phase 2. A lowered seal that enters phase 2 needs no system test: `ClassicToDisklessSwitchLastKnownLeaderClusterTest` lowers the seal, restarts the leader, waits for phase 2, and reads every record. Assisted-by: Claude Opus 5.5
Three runs of `test_fault_before_control_plane_init` failed on records that the producer couldn't send: the replication hold with a leader SIGKILL and with a controller failover, and the control-plane hold with a rolling restart. `VerifiableProducer` sets `retries` on the client as well as the delivery timeout, so `retries=20` capped each record at 20 attempts. A broker that is down or sealed rejects a send within a second, and the backoff stays at 1 s, so a record used up its attempts long before the hold ended. The held runs now use `retries=1000`, which keeps a record alive for 1,000 attempts and 8 hours. A failed send now also reports the exception and message of the first errors from the producer output. Assisted-by: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Category B fault tests injected the fault right after the config change, so the protocol step that the fault hit was a race. To stretch the steps, they ran under a degraded network with 30 partitions. Together they took 43 minutes of the 2026-11-19 nightly, and the system test job has 300 minutes.
test_fault_before_control_plane_initreplaces them. Each run keeps the switch in one step and injects the fault there. Unless the fault itself moves the switch on, the test asserts that the switch is still in that step. Only then does it release the switch. The holds work like this:replicationholds step 3 inWaitingForReplication. A network partition cuts a follower of partition 0 off from the other brokers while a producer writes. A raisedreplica.lag.time.max.mskeeps the follower in the ISR.control_planeholds step 5 inAwaitingMetadata. The test blocks Postgres during the replication hold and then heals the network partition. The controller commits the seal, and control-plane init retries until Postgres is back.The runs cover a follower SIGKILL, a leader SIGKILL, and a controller failover before the seal commit under the replication hold. A leader SIGSTOP and a rolling restart run while Postgres is down. The producer uses a 10-minute delivery timeout, so no record expires during a hold.
Category D runs also cover these paths:
controller_failover_at_phase_2kills the leader of partition 0 while its flag request is pending, and starts it again after the failover.reassignmentmoves a follower of partition 1 in phase 2 while a producer writes. The test then reads the topic with only the new replica of that partition running.object_storage_outagehas a consumer group commit in the classic prefix before the switch, resume up to the seal in phase 1, and resume through the tail in phase 2. Each time, the test reads the committed offsets back withkafka-consumer-groups.sh. It checksListOffsetsfor earliest and latest in phase 1. In phase 2 it checks latest, earliest, and a timestamp at the start of the tail through each broker, and max-timestamp once.A lowered seal that enters phase 2 doesn't need a system test.
ClassicToDisklessSwitchLastKnownLeaderClusterTestwithloseTail=truelowers the seal and restarts the leader. It then waits for phase 2 on every broker and reads every record.Testing
python3 -m py_compilepasses on the test file.self.attribute that the test class uses defined in the class.make valedoesn't report alerts on the added lines.inkless_topic_switch_test.pyat 08245b0 in 154.5 minutes. 25 of 28 runs passed, including every Category D run and every unchanged test. Threetest_fault_before_control_plane_initruns failed on records that the producer couldn't send: the replication hold withleader_sigkillandcontroller_failover, and the control-plane hold withrolling_restart. a444fb8 lifts the cap of 20 attempts thatretries=20set on each record.test_fault_before_control_plane_initat a444fb8 is in progress.Assisted-by: Claude Opus 5.5
🤖 Generated with Claude Code