feat(inkless:switch): implement repair tool for diskless switch - #686
Conversation
8bced5c to
fb663ea
Compare
4fc4109 to
f9a977d
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds an operator-facing “repair” flow to reconcile diskless control-plane log entries with the seal offset committed in KRaft, exposed both as a kafka-topic-switch subcommand and as a new broker Admin API (REPAIR_DISKLESS_LOG, apiKey 502). It also wires the broker-side implementation through ReplicaManager into the control plane (Postgres + in-memory) and introduces the required DB migration + regenerated jOOQ bindings.
Changes:
- Add
kafka-topic-switch.sh repaircommand and unit tests. - Add new broker Admin API (502) and client-side Admin plumbing (
Admin#repairDisklessLog, request/response types, serialization, interception). - Add control-plane operation
repairDisklessLog(Postgres job + SQL migration + metrics), plus tests.
Reviewed changes
Copilot reviewed 31 out of 149 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/src/test/java/org/apache/kafka/tools/TopicSwitchCommandTest.java | Adds unit tests covering repair command behavior. |
| tools/src/main/java/org/apache/kafka/tools/TopicSwitchCommand.java | Implements repair CLI subcommand and argument parsing. |
| streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/TestingMetricsInterceptingAdminClient.java | Forwards the new Admin API call through the metrics intercepting wrapper. |
| storage/inkless/src/test/java/io/aiven/inkless/control_plane/AbstractControlPlaneTest.java | Adds control-plane tests for insert/update behavior of repairDisklessLog. |
| storage/inkless/src/main/resources/db/migration/V18__Repair_diskless_log.sql | Adds DB types + repair_diskless_log_v1 function for upsert repair. |
| storage/inkless/src/main/jooq/org/jooq/generated/UDTs.java | jOOQ generated: registers new repair request/response UDTs + schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/RepairDisklessLogResponseV1.java | jOOQ generated: response UDT type for repair function. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/RepairDisklessLogRequestV1.java | jOOQ generated: request UDT type for repair function. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/RepairDisklessLogResponseV1Record.java | jOOQ generated: response record type. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/RepairDisklessLogRequestV1Record.java | jOOQ generated: request record type. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/PruneBatchesBelowHighestTieredOffsetResponseV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/PruneBatchesBelowHighestTieredOffsetRequestV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/ListOffsetsResponseV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/ListOffsetsRequestV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/InitDisklessLogResponseV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/InitDisklessLogRequestV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/InitDisklessLogProducerStateV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/FindBatchesResponseV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/FindBatchesRequestV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/EnforceRetentionResponseV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/EnforceRetentionRequestV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/DeleteRecordsResponseV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/DeleteRecordsRequestV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/CommitBatchResponseV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/CommitBatchRequestV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/BatchMetadataV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/BatchInfoV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/AdvanceCrossTierLogStartResponseV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/records/AdvanceCrossTierLogStartRequestV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/PruneBatchesBelowHighestTieredOffsetResponseV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/PruneBatchesBelowHighestTieredOffsetRequestV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/RepairDisklessLogResponseV1Path.java | jOOQ generated: UDT path for response fields. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/RepairDisklessLogRequestV1Path.java | jOOQ generated: UDT path for request fields. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/PruneBatchesBelowHighestTieredOffsetResponseV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/PruneBatchesBelowHighestTieredOffsetRequestV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/ListOffsetsResponseV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/ListOffsetsRequestV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/InitDisklessLogResponseV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/InitDisklessLogRequestV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/InitDisklessLogProducerStateV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/FindBatchesResponseV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/FindBatchesRequestV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/EnforceRetentionResponseV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/EnforceRetentionRequestV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/DeleteRecordsResponseV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/DeleteRecordsRequestV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/CommitBatchResponseV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/CommitBatchRequestV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/BatchMetadataV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/BatchInfoV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/AdvanceCrossTierLogStartResponseV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/AdvanceCrossTierLogStartRequestV1Path.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/ListOffsetsResponseV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/ListOffsetsRequestV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/InitDisklessLogResponseV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/InitDisklessLogRequestV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/InitDisklessLogProducerStateV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/FindBatchesResponseV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/FindBatchesRequestV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/EnforceRetentionResponseV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/EnforceRetentionRequestV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/DeleteRecordsResponseV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/DeleteRecordsRequestV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/CommitBatchResponseV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/CommitBatchRequestV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/BatchMetadataV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/BatchInfoV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/AdvanceCrossTierLogStartResponseV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/udt/AdvanceCrossTierLogStartRequestV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/RepairDisklessLogV1.java | jOOQ generated: table-valued function wrapper for repair function. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/RepairDisklessLogV1Record.java | jOOQ generated: record type for function result wrapper. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/PruneBatchesBelowHighestTieredOffsetV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/ProducerStateRecord.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/LogsRecord.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/ListOffsetsV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/InitDisklessLogV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/FindBatchesV2Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/FindBatchesV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/FilesRecord.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/EnforceRetentionV2Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/EnforceRetentionV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/DeleteRecordsV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/CommitFileV2Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/CommitFileV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/BatchesRecord.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/records/AdvanceCrossTierLogStartV1Record.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/PruneBatchesBelowHighestTieredOffsetV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/ProducerState.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/Logs.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/ListOffsetsV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/InitDisklessLogV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/FindBatchesV2.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/FindBatchesV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/Files.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/EnforceRetentionV2.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/EnforceRetentionV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/DeleteRecordsV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/CommitFileV2.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/CommitFileV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/Batches.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/tables/AdvanceCrossTierLogStartV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/Tables.java | jOOQ generated: registers repair_diskless_log_v1 table function. |
| storage/inkless/src/main/jooq/org/jooq/generated/routines/MarkFileToDeleteV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/routines/FlushCommitRunV2.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/routines/DeleteTopicV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/routines/DeleteFilesV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/routines/DeleteBatchV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/routines/BatchTimestamp.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/Routines.java | jOOQ generated: adds repairDisklessLogV1 routine accessors. |
| storage/inkless/src/main/jooq/org/jooq/generated/Keys.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/Indexes.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/enums/PruneBatchesBelowHighestTieredOffsetErrorV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/enums/ListOffsetsResponseErrorV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/enums/InitDisklessLogErrorV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/enums/FindBatchesResponseErrorV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/enums/FileStateT.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/enums/FileReasonT.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/enums/EnforceRetentionResponseErrorV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/enums/DeleteRecordsResponseErrorV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/enums/CommitBatchResponseErrorV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/enums/AdvanceCrossTierLogStartResponseErrorV1.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/Domains.java | jOOQ generated: schema version bump. |
| storage/inkless/src/main/jooq/org/jooq/generated/DefaultSchema.java | jOOQ generated: registers new repair UDTs/table function in default schema. |
| storage/inkless/src/main/java/io/aiven/inkless/control_plane/RepairDisklessLogResponse.java | Adds control-plane response DTO for repair operation. |
| storage/inkless/src/main/java/io/aiven/inkless/control_plane/RepairDisklessLogRequest.java | Adds control-plane request DTO with validation (disklessStartOffset >= logStartOffset). |
| storage/inkless/src/main/java/io/aiven/inkless/control_plane/postgres/RepairDisklessLogJob.java | Implements Postgres-backed repair job invoking repair_diskless_log_v1. |
| storage/inkless/src/main/java/io/aiven/inkless/control_plane/postgres/PostgresControlPlaneMetrics.java | Adds metrics tracking for repair query execution. |
| storage/inkless/src/main/java/io/aiven/inkless/control_plane/postgres/PostgresControlPlane.java | Exposes repairDisklessLog via Postgres control-plane implementation. |
| storage/inkless/src/main/java/io/aiven/inkless/control_plane/InMemoryControlPlane.java | Adds in-memory implementation of repairDisklessLog for tests/dev. |
| storage/inkless/src/main/java/io/aiven/inkless/control_plane/ControlPlane.java | Extends control-plane interface with repairDisklessLog. |
| server/src/main/java/org/apache/kafka/network/RequestConvertToJson.java | Adds JSON conversion support for the new request/response. |
| docs/inkless/metrics.rst | Documents new RepairDisklessLog query metrics. |
| docs/inkless/CLASSIC_TO_DISKLESS_SWITCH.md | Documents why/when to run repair and how it works. |
| core/src/main/scala/kafka/server/ReplicaManager.scala | Adds broker-side repair implementation that writes to the control plane. |
| core/src/main/scala/kafka/server/KafkaApis.scala | Adds broker request handler for REPAIR_DISKLESS_LOG. |
| clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java | Adds request/response round-trip coverage for apiKey 502. |
| clients/src/test/java/org/apache/kafka/clients/admin/MockAdminClient.java | Adds stub override for the new Admin API in the mock client. |
| clients/src/main/resources/common/message/RepairDisklessLogResponse.json | Defines protocol schema for RepairDisklessLogResponse (v0). |
| clients/src/main/resources/common/message/RepairDisklessLogRequest.json | Defines protocol schema for RepairDisklessLogRequest (v0). |
| clients/src/main/java/org/apache/kafka/common/requests/RepairDisklessLogResponse.java | Adds client-side response type. |
| clients/src/main/java/org/apache/kafka/common/requests/RepairDisklessLogRequest.java | Adds client-side request type + error response mapping. |
| clients/src/main/java/org/apache/kafka/common/requests/AbstractResponse.java | Registers response parsing for apiKey 502. |
| clients/src/main/java/org/apache/kafka/common/requests/AbstractRequest.java | Registers request parsing for apiKey 502. |
| clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java | Adds REPAIR_DISKLESS_LOG to ApiKeys enum. |
| clients/src/main/java/org/apache/kafka/clients/admin/RepairDisklessLogResult.java | Adds Admin result wrapper for repair call. |
| clients/src/main/java/org/apache/kafka/clients/admin/RepairDisklessLogOptions.java | Adds Admin options type for repair call. |
| clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java | Implements repairDisklessLog call routing to a specific brokerId. |
| clients/src/main/java/org/apache/kafka/clients/admin/ForwardingAdmin.java | Forwards the new Admin API call to the delegate. |
| clients/src/main/java/org/apache/kafka/clients/admin/Admin.java | Adds public Admin interface methods for repairDisklessLog. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ccd6fc9 to
555b7b8
Compare
7804e90 to
5efd38f
Compare
5efd38f to
d5c3c86
Compare
d5c3c86 to
4a59f30
Compare
giuseppelillo
left a comment
There was a problem hiding this comment.
LGTM, just a couple of comments
| throw new RuntimeException("Command \"" + command + "\" not implemented"); | ||
| try (Admin adminClient = Admin.create(properties)) { | ||
| repairCommand(System.out, adminClient, topic, | ||
| Optional.ofNullable(namespace.getInt("partition"))); | ||
| } | ||
| break; |
There was a problem hiding this comment.
This command now requires to be sent to a broker, therefore we need to make it accept only --bootstrap-server and not --bootstrap-controller
* feat(inkless): add repairDisklessLog control-plane operation * feat(inkless:tools): implement repair command for topic type switch * fixup! feat(inkless:tools): implement repair command for topic type switch
* feat(inkless): add repairDisklessLog control-plane operation * feat(inkless:tools): implement repair command for topic type switch * fixup! feat(inkless:tools): implement repair command for topic type switch
This PR introduces repair tooling for diskless partitions.
This includes:
repairDisklessLogREPAIR_DISKLESS_LOG (502)The tool reconciles the diskless control plane with with KRaft,
by writing the seal offsets to the control plane.