Skip to content

ci(inkless:system-tests): split the nightly ducktape suite into parallel shards [KC-838] - #886

Open
giuseppelillo wants to merge 1 commit into
mainfrom
ci/kc-838-split-system-tests
Open

giuseppelillo wants to merge 1 commit into
mainfrom
ci/kc-838-split-system-tests

Conversation

@giuseppelillo

Copy link
Copy Markdown
Contributor

The nightly Inkless system tests ran in one job: one 11-node ducker cluster, one test at a time, under a 300-minute inner timeout and a 350-minute job timeout. The 2026-11-19 nightly spent 240.6 minutes in ducktape, so the classic-to-diskless switch tests that KC-838 adds had little room left. GitHub's 360-minute ceiling for hosted runners leaves no room to raise the timeouts.

The job is now a matrix of three shards. Each shard runs on its own runner, with its own ducker cluster, Postgres, and MinIO. The split follows the per-test times from that nightly:

Shard Ducktape target Nightly time
switch-category-d inkless_topic_switch_test.py::InklessClassicToDisklessSwitchTest.test_fault_between_seal_commit_and_phase_2 72.5 min
switch-other inkless_topic_switch_test.py, excluding Category D 88.2 min
other tests/kafkatest/tests/inkless/, excluding inkless_topic_switch_test.py 79.8 min

The shards use ducktape --exclude, so together they always cover the whole directory. A new switch test lands in switch-other, and a new file lands in other, without a workflow change. Other behavior is unchanged:

  • Timeouts: each shard keeps the 300-minute inner timeout and the 350-minute job timeout. The workflow comment says to move tests or add a shard if a shard outgrows them.
  • Archive and cleanup: each shard keeps these steps and uploads its own inkless-system-tests-results-<shard> artifact.
  • Failure handling: fail-fast is off, so a failing shard doesn't cancel the others. If a target matches no tests, ducktape 0.13 exits 1 with "No tests to run!", and the shard fails.
  • Manual runs: the test-target input of workflow_dispatch is now optional. If it's empty, the full suite runs split. If it's set, the target runs as one custom job.

Testing

  • Run 44538947084 dispatched test-target=tests/kafkatest/tests/inkless/inkless_produce_consume_test.py, a file where ducktape finds no tests. It ran as one custom job, ducktape printed "No tests to run!", and the job failed.
  • Run 44540851724 dispatched the full suite with an empty target. It ran 50 tests across the shards, the same 50 as the 2026-11-19 nightly:
    • switch-category-d: 12 passed, 69.7 minutes in ducktape.
    • other: 22 passed and 1 ignored, 81.3 minutes.
    • switch-other, attempt 1: 14 passed and 1 failed, 94.6 minutes. test_switch_succeeds_with_degraded_network failed because VerifiableConsumer reported "Consumed from an unexpected offset (4138, 2344)" for partition 28. That test passed in the 2026-11-19 nightly.
    • switch-other, attempt 2: 15 passed, 88.2 minutes.
  • Attempt 1 took 112 minutes from start to finish. The 2026-11-19 nightly took 267 minutes.
  • make vale reports no alerts on the new workflow comments.

🤖 Generated with Claude Code

…lel shards [KC-838]

The nightly Inkless system tests ran in one job: one 11-node ducker
cluster, one test at a time, under a 300-minute inner timeout and a
350-minute job timeout. The 2026-11-19 nightly spent 240.6 minutes in
ducktape, so the classic-to-diskless switch tests that KC-838 adds had
little room left, and GitHub's 360-minute ceiling leaves no room to
raise the timeouts.

The job is now a matrix of three shards, each on its own runner with
its own ducker cluster, Postgres, and MinIO:

- `switch-category-d`: `test_fault_between_seal_commit_and_phase_2`
  (about 72 minutes on that nightly).
- `switch-other`: the rest of `inkless_topic_switch_test.py` (about 88
  minutes).
- `other`: every other file under `tests/kafkatest/tests/inkless/`
  (about 80 minutes).

The shards use ducktape `--exclude`, so a new test lands in a shard
without a workflow change. Each shard keeps the 300-minute and
350-minute timeouts and uploads its own artifact. The `test-target`
input of `workflow_dispatch` is now optional. If it's empty, the full
suite runs split. If it's set, the target runs as one `custom` job.

Assisted-by: Claude Opus 5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant