test(inkless:systest): run ducktape system tests against consolidated topics#645
Open
viktorsomogyi wants to merge 1 commit into
Open
test(inkless:systest): run ducktape system tests against consolidated topics#645viktorsomogyi wants to merge 1 commit into
viktorsomogyi wants to merge 1 commit into
Conversation
ce6483e to
99169bd
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds per-test “consolidation” mode support so Inkless ducktape system tests can exercise consolidating (diskless + remote tiered storage) topics alongside existing classic→diskless switch tests in the same run.
Changes:
- Add a per-test
KafkaService.consolidationflag (with fallback to--globals consolidation) and thread it into broker config rendering. - Introduce consolidation-specific Inkless test subclasses (produce-bench, compression, get-offset-shell) that reuse upstream test bodies with consolidation enabled.
- Extend the ducker system test container/workflow to include the Aiven tiered-storage plugin and broaden the workflow default test target to the whole Inkless test directory.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
tests/kafkatest/version.py |
Adds clarifying comments for tool-flag support checks tied to Kafka version gates. |
tests/kafkatest/tests/inkless/consolidation_produce_bench_test.py |
New consolidation variant of produce-bench; enables consolidation and adjusts producer settings; ignores transactional case. |
tests/kafkatest/tests/inkless/consolidation_get_offset_shell_test.py |
New consolidation variant of GetOffsetShell tests; constructs KafkaService with consolidation=True. |
tests/kafkatest/tests/inkless/consolidation_compression_test.py |
New consolidation variant of compression tests; flips broker/controller consolidation flags per-test. |
tests/kafkatest/tests/core/produce_bench_test.py |
Adds an overridable producer_config() hook and uses it for both benchmark variants. |
tests/kafkatest/services/kafka/templates/kafka.properties |
Adds a consolidation config block that enables diskless+remote-tiering settings. |
tests/kafkatest/services/kafka/kafka.py |
Adds consolidation ctor arg and passes consolidation into template rendering (with globals fallback). |
tests/docker/Dockerfile |
Downloads and installs the Aiven tiered-storage plugin into the ducker image. |
.github/workflows/inkless-system-tests.yml |
Renames/broadens the Inkless system test workflow and defaults to running all Inkless tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
2470f62 to
1f25fba
Compare
1f25fba to
8b105a4
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
tests/kafkatest/services/kafka/kafka.py:215
- The
KafkaService.__init__signature now includes theconsolidationparameter, but the constructor docstring doesn’t document it. This makes it harder for test authors to discover the per-test vs--globals consolidationbehavior that the code implements.
"""
:param context: test context
:param int num_nodes: the number of nodes in the service. There are 4 possibilities:
8b105a4 to
bf02116
Compare
… topics This commit adds the ability to run the ducktape system tests against consolidating topics, and a few consolidation-specific variants, in the same run as the existing diskless and classic->diskless switch tests. Per-test switch: KafkaService gains a `consolidation` flag (forwarded to the isolated controller) that renders a consolidation broker config in kafka.properties (diskless default + remote tiering via the Aiven tiered-storage plugin, S3/MinIO + Postgres backend). It falls back to the run-wide `--globals consolidation` flag when unset since theswitch tests require the classic default. Tests and infra: three thin subclasses under tests/kafkatest/tests/inkless reuse upstream test bodies with consolidation enabled (produce-bench, compression, get-offset-shell). Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
bf02116 to
fd552d5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit adds the ability to run the ducktape system tests against consolidating topics, and a few consolidation-specific variants, in the same run as the existing diskless and classic->diskless switch tests.
Per-test switch: KafkaService gains a
consolidationflag (forwarded to the isolated controller) that renders a consolidation broker config in kafka.properties (diskless default + remote tiering via the Aiven tiered-storage plugin, S3/MinIO + Postgres backend). It falls back to the run-wide--globals consolidationflag when unset since theswitch tests require the classic default.Tests and infra: three thin subclasses under tests/kafkatest/tests/inkless reuse upstream test bodies with consolidation enabled (produce-bench, compression, get-offset-shell).