Skip to content

test(inkless:switch): hold each switch step before injecting faults [KC-838] - #885

Open
giuseppelillo wants to merge 2 commits into
mainfrom
claude/kc-838-fix-plan-12b40e
Open

giuseppelillo wants to merge 2 commits into
mainfrom
claude/kc-838-fix-plan-12b40e

Conversation

@giuseppelillo

@giuseppelillo giuseppelillo commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

The Category B fault tests injected the fault right after the config change, so the protocol step that the fault hit was a race. To stretch the steps, they ran under a degraded network with 30 partitions. Together they took 43 minutes of the 2026-11-19 nightly, and the system test job has 300 minutes.

test_fault_before_control_plane_init replaces them. Each run keeps the switch in one step and injects the fault there. Unless the fault itself moves the switch on, the test asserts that the switch is still in that step. Only then does it release the switch. The holds work like this:

  • replication holds step 3 in WaitingForReplication. A network partition cuts a follower of partition 0 off from the other brokers while a producer writes. A raised replica.lag.time.max.ms keeps the follower in the ISR.
  • control_plane holds step 5 in AwaitingMetadata. The test blocks Postgres during the replication hold and then heals the network partition. The controller commits the seal, and control-plane init retries until Postgres is back.

The runs cover a follower SIGKILL, a leader SIGKILL, and a controller failover before the seal commit under the replication hold. A leader SIGSTOP and a rolling restart run while Postgres is down. The producer uses a 10-minute delivery timeout, so no record expires during a hold.

Category D runs also cover these paths:

  • controller_failover_at_phase_2 kills the leader of partition 0 while its flag request is pending, and starts it again after the failover.
  • reassignment moves a follower of partition 1 in phase 2 while a producer writes. The test then reads the topic with only the new replica of that partition running.
  • object_storage_outage has a consumer group commit in the classic prefix before the switch, resume up to the seal in phase 1, and resume through the tail in phase 2. Each time, the test reads the committed offsets back with kafka-consumer-groups.sh. It checks ListOffsets for earliest and latest in phase 1. In phase 2 it checks latest, earliest, and a timestamp at the start of the tail through each broker, and max-timestamp once.

A lowered seal that enters phase 2 doesn't need a system test. ClassicToDisklessSwitchLastKnownLeaderClusterTest with loseTail=true lowers the seal and restarts the leader. It then waits for phase 2 on every broker and reads every record.

Testing

  • python3 -m py_compile passes on the test file.
  • An AST check finds every self. attribute that the test class uses defined in the class.
  • make vale doesn't report alerts on the added lines.
  • Run 44545244135 ran inkless_topic_switch_test.py at 08245b0 in 154.5 minutes. 25 of 28 runs passed, including every Category D run and every unchanged test. Three test_fault_before_control_plane_init runs failed on records that the producer couldn't send: the replication hold with leader_sigkill and controller_failover, and the control-plane hold with rolling_restart. a444fb8 lifts the cap of 20 attempts that retries=20 set on each record.
  • Run 44578598624 of test_fault_before_control_plane_init at a444fb8 is in progress.

Assisted-by: Claude Opus 5.5

🤖 Generated with Claude Code

…KC-838]

The Category B fault tests injected the fault right after the config
change, so the protocol step that the fault hit was a race. They also
used a degraded network and 30 partitions to stretch the steps, which
made them the slowest switch tests.

`test_fault_before_control_plane_init` replaces them. A network partition
holds step 3 in `WaitingForReplication`, and a Postgres outage after the
seal commit holds step 5 in `AwaitingMetadata`. Each run injects its
fault inside a hold, asserts that the hold is still in place unless the
fault ends it, and then releases it. The runs cover a follower and a
leader crash, a controller failover before the seal commit, a frozen
leader, and a rolling restart while Postgres is down.

Category D runs also cover the remaining client and operator paths:

- `controller_failover_at_phase_2` kills the leader of partition 0 while
  its flag request is pending.
- `reassignment` moves a follower in phase 2 under load, then reads the
  partition through the new replica alone.
- `object_storage_outage` resumes a consumer group before the switch, in
  phase 1, and in phase 2, and reads its committed offsets back. It
  checks `ListOffsets` for the latest, earliest, timestamp, and
  max-timestamp specs in phase 1 and through each broker in phase 2.

A lowered seal that enters phase 2 needs no system test:
`ClassicToDisklessSwitchLastKnownLeaderClusterTest` lowers the seal,
restarts the leader, waits for phase 2, and reads every record.

Assisted-by: Claude Opus 5.5
Three runs of `test_fault_before_control_plane_init` failed on
records that the producer couldn't send: the replication hold with a
leader SIGKILL and with a controller failover, and the control-plane
hold with a rolling restart. `VerifiableProducer` sets `retries` on the
client as well as the delivery timeout, so `retries=20` capped each
record at 20 attempts. A broker that is down or sealed rejects a send
within a second, and the backoff stays at 1 s, so a record used up its
attempts long before the hold ended.

The held runs now use `retries=1000`, which keeps a record alive for
1,000 attempts and 8 hours. A failed send now also reports the
exception and message of the first errors from the producer output.

Assisted-by: Claude Opus 5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant