test(inkless:systest): run ducktape system tests against consolidated topics [KC-153] by viktorsomogyi · Pull Request #645 · aiven/inkless

viktorsomogyi · 2026-06-12T15:45:37Z

This commit adds the ability to run the ducktape system tests against consolidating topics, and a few consolidation-specific variants, in the same run as the existing diskless and classic->diskless switch tests.

Per-test switch: KafkaService gains a consolidation flag (forwarded to the isolated controller) that renders a consolidation broker config in kafka.properties (diskless default + remote tiering via the Aiven tiered-storage plugin, S3/MinIO + Postgres backend). It falls back to the run-wide --globals consolidation flag when unset since theswitch tests require the classic default.

Tests and infra: three thin subclasses under tests/kafkatest/tests/inkless reuse upstream test bodies with consolidation enabled (produce-bench, compression, get-offset-shell).

Copilot

Pull request overview

Adds per-test “consolidation” mode support so Inkless ducktape system tests can exercise consolidating (diskless + remote tiered storage) topics alongside existing classic→diskless switch tests in the same run.

Changes:

Add a per-test KafkaService.consolidation flag (with fallback to --globals consolidation) and thread it into broker config rendering.
Introduce consolidation-specific Inkless test subclasses (produce-bench, compression, get-offset-shell) that reuse upstream test bodies with consolidation enabled.
Extend the ducker system test container/workflow to include the Aiven tiered-storage plugin and broaden the workflow default test target to the whole Inkless test directory.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`tests/kafkatest/version.py`	Adds clarifying comments for tool-flag support checks tied to Kafka version gates.
`tests/kafkatest/tests/inkless/consolidation_produce_bench_test.py`	New consolidation variant of produce-bench; enables consolidation and adjusts producer settings; ignores transactional case.
`tests/kafkatest/tests/inkless/consolidation_get_offset_shell_test.py`	New consolidation variant of GetOffsetShell tests; constructs KafkaService with `consolidation=True`.
`tests/kafkatest/tests/inkless/consolidation_compression_test.py`	New consolidation variant of compression tests; flips broker/controller consolidation flags per-test.
`tests/kafkatest/tests/core/produce_bench_test.py`	Adds an overridable `producer_config()` hook and uses it for both benchmark variants.
`tests/kafkatest/services/kafka/templates/kafka.properties`	Adds a consolidation config block that enables diskless+remote-tiering settings.
`tests/kafkatest/services/kafka/kafka.py`	Adds `consolidation` ctor arg and passes consolidation into template rendering (with globals fallback).
`tests/docker/Dockerfile`	Downloads and installs the Aiven tiered-storage plugin into the ducker image.
`.github/workflows/inkless-system-tests.yml`	Renames/broadens the Inkless system test workflow and defaults to running all Inkless tests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tests/kafkatest/services/kafka/kafka.py:215

The KafkaService.__init__ signature now includes the consolidation parameter, but the constructor docstring doesn’t document it. This makes it harder for test authors to discover the per-test vs --globals consolidation behavior that the code implements.

        """
        :param context: test context
        :param int num_nodes: the number of nodes in the service.  There are 4 possibilities:

… topics This commit adds the ability to run the ducktape system tests against consolidating topics, and a few consolidation-specific variants, in the same run as the existing diskless and classic->diskless switch tests. Per-test switch: KafkaService gains a `consolidation` flag (forwarded to the isolated controller) that renders a consolidation broker config in kafka.properties (diskless default + remote tiering via the Aiven tiered-storage plugin, S3/MinIO + Postgres backend). It falls back to the run-wide `--globals consolidation` flag when unset since theswitch tests require the classic default. Tests and infra: three thin subclasses under tests/kafkatest/tests/inkless reuse upstream test bodies with consolidation enabled (produce-bench, compression, get-offset-shell). Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>

giuseppelillo

LGTM, just a minor comment

giuseppelillo · 2026-06-22T13:49:01Z

+# Aiven Tiered Storage plugin (for consolidation/remote storage in system tests).
+# Download explicitly with curl (-f fails on HTTP errors, -L follows the GitHub
+# release redirect, --retry tolerates transient network hiccups).
+ARG ts_plugin_version=1.1.1
+ARG ts_plugin_base_url=https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/releases/download/v${ts_plugin_version}
+RUN mkdir -p /opt/tiered-storage-plugin/core /opt/tiered-storage-plugin/s3 && \
+    curl -fsSL --retry 5 --retry-delay 5 "${ts_plugin_base_url}/core-${ts_plugin_version}.tgz" -o /tmp/core.tgz && \
+    curl -fsSL --retry 5 --retry-delay 5 "${ts_plugin_base_url}/s3-${ts_plugin_version}.tgz" -o /tmp/s3.tgz && \
+    tar xzf /tmp/core.tgz -C /opt/tiered-storage-plugin/core --strip-components=1 && \
+    tar xzf /tmp/s3.tgz -C /opt/tiered-storage-plugin/s3 --strip-components=1 && \
+    rm /tmp/*.tgz
+


For the diskless switch tests I used the LocalTieredStorage implementation. I guess now we can refactor those tests as well to use the Aiven TS plugin.

Yes, we could if you think it makes sense. Do you mind if I did that in a follow-up to limit the scope for this one?

giuseppelillo · 2026-06-22T13:49:35Z

+
+{% if consolidation %}
+# Diskless/consolidation feature toggles
+log.diskless.enable=true


Do we really need this? Is it for allowing also the already existing tests to run?

Okay I see it's mainly for ConsolidationProduceBenchTest

Yes, we just use this for our tests. Classic tests won't use this, unless they construct the Kafka service this way (as ConsolidationProduceBenchTest does).

giuseppelillo · 2026-06-22T13:55:00Z

+    def test_produce_bench_transactions(self, metadata_quorum):
+        # Transactions are not supported on consolidating (diskless) topics, so
+        # the inherited transactional variant is intentionally ignored.
+        pass


Maybe we can add a log if it's possible?

But also, this method is tagged with @ignore so I'm not sure if the logging is displayed when called.

Ran it, it looks like it won't log. Do you think we should still add it regardless?

Not worth it

… topics (#645) This commit adds the ability to run the ducktape system tests against consolidating topics, and a few consolidation-specific variants, in the same run as the existing diskless and classic->diskless switch tests. Per-test switch: KafkaService gains a `consolidation` flag (forwarded to the isolated controller) that renders a consolidation broker config in kafka.properties (diskless default + remote tiering via the Aiven tiered-storage plugin, S3/MinIO + Postgres backend). It falls back to the run-wide `--globals consolidation` flag when unset since theswitch tests require the classic default. Tests and infra: three thin subclasses under tests/kafkatest/tests/inkless reuse upstream test bodies with consolidation enabled (produce-bench, compression, get-offset-shell). Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>

viktorsomogyi force-pushed the svv/ts-unification-systest branch from ce6483e to 99169bd Compare June 12, 2026 15:49

viktorsomogyi requested a review from Copilot June 12, 2026 16:01

Copilot started reviewing on behalf of viktorsomogyi June 12, 2026 16:02 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

Comment thread tests/kafkatest/services/kafka/templates/kafka.properties

Comment thread tests/docker/Dockerfile Outdated

viktorsomogyi force-pushed the svv/ts-unification-systest branch 2 times, most recently from 2470f62 to 1f25fba Compare June 12, 2026 16:20

viktorsomogyi marked this pull request as ready for review June 12, 2026 16:21

viktorsomogyi requested a review from Copilot June 15, 2026 09:20

Copilot started reviewing on behalf of viktorsomogyi June 15, 2026 09:20 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

Comment thread tests/kafkatest/services/kafka/templates/kafka.properties

viktorsomogyi force-pushed the svv/ts-unification-systest branch from 1f25fba to 8b105a4 Compare June 15, 2026 10:50

viktorsomogyi requested a review from Copilot June 15, 2026 10:56

Copilot started reviewing on behalf of viktorsomogyi June 15, 2026 10:57 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

viktorsomogyi force-pushed the svv/ts-unification-systest branch from 8b105a4 to bf02116 Compare June 15, 2026 13:45

viktorsomogyi changed the base branch from main to svv/ts-unification-leader-epoch June 15, 2026 13:46

viktorsomogyi requested review from EelisK, giuseppelillo and jeqo June 16, 2026 12:08

Base automatically changed from svv/ts-unification-leader-epoch to main June 16, 2026 14:46

viktorsomogyi force-pushed the svv/ts-unification-systest branch from bf02116 to fd552d5 Compare June 16, 2026 15:01

giuseppelillo reviewed Jun 22, 2026

View reviewed changes

viktorsomogyi changed the title ~~test(inkless:systest): run ducktape system tests against consolidated topics~~ test(inkless:systest): run ducktape system tests against consolidated topics [KC-153] Jun 22, 2026

viktorsomogyi requested a review from giuseppelillo June 22, 2026 15:18

giuseppelillo approved these changes Jun 22, 2026

View reviewed changes

giuseppelillo merged commit ce0ac57 into main Jun 22, 2026
15 of 16 checks passed

giuseppelillo deleted the svv/ts-unification-systest branch June 22, 2026 15:42

Uh oh!

Conversation

viktorsomogyi commented Jun 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

giuseppelillo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viktorsomogyi Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

viktorsomogyi Jun 22, 2026 •

edited

Loading