test(inkless:consolidation): add born-consolidated pipeline + WAL-prune system test#646
Draft
viktorsomogyi wants to merge 1 commit into
Draft
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an end-to-end ducktape system test that exercises the “born-consolidated” inkless pipeline (WAL → local log → remote tiered storage) and verifies that WAL pruning occurs, using new helper tooling to query JMX, MinIO, and the Postgres control plane.
Changes:
- Add
ConsolidationPipelineTestto validate consolidation gauges, tiered object movement, WAL pruning, and post-prune consumability. - Add
ConsolidationVerifierhelper to query consolidation state via broker JMX, MinIO (mc), and Postgres (psql). - Extend the ducker image to include
postgresql-clientand MinIOmc.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| tests/kafkatest/tests/inkless/consolidation_pipeline_test.py | New system test covering consolidation pipeline + WAL prune behavior end-to-end. |
| tests/kafkatest/services/inkless/consolidation_verifier.py | New helper for JMX/MinIO/Postgres assertions used by consolidation tests. |
| tests/kafkatest/services/inkless/init.py | Introduces the inkless services package init file. |
| tests/docker/Dockerfile | Installs psql and downloads mc to support consolidation system tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
5cb8ce0 to
c2d1d6e
Compare
c2d1d6e to
e51509b
Compare
1f25fba to
8b105a4
Compare
e51509b to
b7173a9
Compare
b7173a9 to
d7fcdb1
Compare
8b105a4 to
bf02116
Compare
dae5934 to
f897e44
Compare
bf02116 to
fd552d5
Compare
…ne system test Add ConsolidationPipelineTest for a born-consolidated topic, asserting the full WAL -> local log -> remote pipeline: consolidation JMX gauges are exported, the MinIO tiered-storage prefix grows, the Postgres control plane prunes the WAL, and every acked record is still consumable. Consolidation is enabled per-test via KafkaService(consolidation=True) (also flips the isolated controller), so it runs in the unified inkless ducktape suite without a run-wide --globals flag. JmxTool runs on brokers only since the controller never exposes the consolidation MBeans. Add ConsolidationVerifier helper (JMX gauges, MinIO via mc, control plane via psql) and install mc + postgresql-client in the ducker image.
f897e44 to
864e013
Compare
| target += "/" + prefix | ||
| cmd = "mc ls --recursive --json %s" % target | ||
| keys = [] | ||
| for line in node.account.ssh_capture(cmd, allow_fail=True): |
Comment on lines
+150
to
+161
| # Read every acked record back (from remote). | ||
| consumer = VerifiableConsumer(self.test_context, num_nodes=1, kafka=self.kafka, | ||
| topic=self.TOPIC, group_id="consolidation-pipeline-group") | ||
| consumer.start() | ||
| wait_until(lambda: consumer.total_consumed() >= acked or consumer.worker_errors, | ||
| timeout_sec=180, backoff_sec=1, | ||
| err_msg="Consumer failed to read back all %d acked records." % acked) | ||
| assert not consumer.worker_errors, "Consumer errors: %s" % consumer.worker_errors | ||
| consumer.stop() | ||
| total_consumed = consumer.total_consumed() | ||
| consumer.free() | ||
| self.logger.info("Read back %d records after consolidation + prune" % total_consumed) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add ConsolidationPipelineTest for a born-consolidated topic, asserting the full WAL -> local log -> remote pipeline: consolidation JMX gauges are exported, the MinIO tiered-storage prefix grows, the Postgres control plane prunes the WAL, and every acked record is still consumable.
Consolidation is enabled per-test via KafkaService(consolidation=True) (also flips the isolated controller), so it runs in the unified inkless ducktape suite without a run-wide --globals flag. JmxTool runs on brokers only since the controller never exposes the consolidation MBeans.
Add ConsolidationVerifier helper (JMX gauges, MinIO via mc, control plane via psql) and install mc + postgresql-client in the ducker image.