Repository navigation
ci(inkless:system-tests): split the nightly ducktape suite into parallel shards [KC-838] - #886
Open
giuseppelillo wants to merge 1 commit into
Open
giuseppelillo wants to merge 1 commit into
giuseppelillo wants to merge 1 commit into
Conversation
…lel shards [KC-838] The nightly Inkless system tests ran in one job: one 11-node ducker cluster, one test at a time, under a 300-minute inner timeout and a 350-minute job timeout. The 2026-11-19 nightly spent 240.6 minutes in ducktape, so the classic-to-diskless switch tests that KC-838 adds had little room left, and GitHub's 360-minute ceiling leaves no room to raise the timeouts. The job is now a matrix of three shards, each on its own runner with its own ducker cluster, Postgres, and MinIO: - `switch-category-d`: `test_fault_between_seal_commit_and_phase_2` (about 72 minutes on that nightly). - `switch-other`: the rest of `inkless_topic_switch_test.py` (about 88 minutes). - `other`: every other file under `tests/kafkatest/tests/inkless/` (about 80 minutes). The shards use ducktape `--exclude`, so a new test lands in a shard without a workflow change. Each shard keeps the 300-minute and 350-minute timeouts and uploads its own artifact. The `test-target` input of `workflow_dispatch` is now optional. If it's empty, the full suite runs split. If it's set, the target runs as one `custom` job. Assisted-by: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The nightly Inkless system tests ran in one job: one 11-node ducker cluster, one test at a time, under a 300-minute inner timeout and a 350-minute job timeout. The 2026-11-19 nightly spent 240.6 minutes in ducktape, so the classic-to-diskless switch tests that KC-838 adds had little room left. GitHub's 360-minute ceiling for hosted runners leaves no room to raise the timeouts.
The job is now a matrix of three shards. Each shard runs on its own runner, with its own ducker cluster, Postgres, and MinIO. The split follows the per-test times from that nightly:
switch-category-dinkless_topic_switch_test.py::InklessClassicToDisklessSwitchTest.test_fault_between_seal_commit_and_phase_2switch-otherinkless_topic_switch_test.py, excluding Category Dothertests/kafkatest/tests/inkless/, excludinginkless_topic_switch_test.pyThe shards use ducktape
--exclude, so together they always cover the whole directory. A new switch test lands inswitch-other, and a new file lands inother, without a workflow change. Other behavior is unchanged:inkless-system-tests-results-<shard>artifact.fail-fastis off, so a failing shard doesn't cancel the others. If a target matches no tests, ducktape 0.13 exits 1 with "No tests to run!", and the shard fails.test-targetinput ofworkflow_dispatchis now optional. If it's empty, the full suite runs split. If it's set, the target runs as onecustomjob.Testing
test-target=tests/kafkatest/tests/inkless/inkless_produce_consume_test.py, a file where ducktape finds no tests. It ran as onecustomjob, ducktape printed "No tests to run!", and the job failed.switch-category-d: 12 passed, 69.7 minutes in ducktape.other: 22 passed and 1 ignored, 81.3 minutes.switch-other, attempt 1: 14 passed and 1 failed, 94.6 minutes.test_switch_succeeds_with_degraded_networkfailed becauseVerifiableConsumerreported "Consumed from an unexpected offset (4138, 2344)" for partition 28. That test passed in the 2026-11-19 nightly.switch-other, attempt 2: 15 passed, 88.2 minutes.make valereports no alerts on the new workflow comments.🤖 Generated with Claude Code