Repository navigation
feat(inkless:consolidation): expose local time lag - #826
Closed
viktorsomogyi wants to merge 3 commits into
Closed
viktorsomogyi wants to merge 3 commits into
viktorsomogyi wants to merge 3 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The operational performance guide still observes and tunes find_batches_v2, although production now exclusively calls V3.
1 open finding
What changed in this PR
Adds a consolidation backlog time-lag gauge backed by timestamp boundaries returned from find_batches_v3.
Changes:
- Adds timestamp-aware control-plane request and response types.
- Propagates timestamps through fetch handling into consolidation metrics.
- Adds unit, integration, benchmark, and system-test coverage.
| File | Description |
|---|---|
tests/kafkatest/tests/inkless/consolidation_read_from_remote_after_prune_test.py |
Checks JMX metric export. |
tests/kafkatest/tests/inkless/consolidation_pipeline_test.py |
Checks pipeline metric export. |
tests/kafkatest/services/inkless/consolidation_verifier.py |
Defines the metric name. |
storage/inkless/src/test/java/io/aiven/inkless/control_plane/postgres/FindBatchesScanDepthBenchmarkTest.java |
Updates benchmarks for V3. |
storage/inkless/src/test/java/io/aiven/inkless/control_plane/postgres/FindBatchesJobTest.java |
Tests PostgreSQL timestamp boundaries. |
storage/inkless/src/test/java/io/aiven/inkless/control_plane/AbstractControlPlaneTest.java |
Tests shared control-plane behavior. |
storage/inkless/src/test/java/io/aiven/inkless/consume/FindBatchesJobTest.java |
Tests consolidation request flags. |
storage/inkless/src/test/java/io/aiven/inkless/consume/FetchCompleterTest.java |
Tests timestamp propagation. |
storage/inkless/src/main/resources/db/migration/V32__Find_batches_high_watermark_timestamp.sql |
Adds V3 database types and function. |
storage/inkless/src/main/jooq/org/jooq/generated/UDTs.java |
Registers generated V2 UDTs. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/RepairDisklessLogResponseV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/RepairDisklessLogRequestV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/RepairDisklessLogResponseV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/RepairDisklessLogRequestV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/PurgeDeletedLogsResponseV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/PruneBatchesBelowHighestTieredOffsetResponseV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/PruneBatchesBelowHighestTieredOffsetRequestV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/ListOffsetsResponseV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/ListOffsetsRequestV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/InitDisklessLogResponseV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/InitDisklessLogRequestV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/InitDisklessLogProducerStateV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/FindBatchesResponseV2Record.java |
Models the V2 response record. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/FindBatchesResponseV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/FindBatchesRequestV2Record.java |
Models the V2 request record. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/FindBatchesRequestV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/EnforceRetentionResponseV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/EnforceRetentionRequestV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/DeleteRecordsResponseV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/DeleteRecordsRequestV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/CommitBatchResponseV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/CommitBatchRequestV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/BatchMetadataV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/BatchInfoV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/AdvanceCrossTierLogStartResponseV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/AdvanceCrossTierLogStartRequestV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/PurgeDeletedLogsResponseV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/PruneBatchesBelowHighestTieredOffsetResponseV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/PruneBatchesBelowHighestTieredOffsetRequestV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/RepairDisklessLogResponseV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/RepairDisklessLogRequestV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/PurgeDeletedLogsResponseV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/PruneBatchesBelowHighestTieredOffsetResponseV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/PruneBatchesBelowHighestTieredOffsetRequestV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/ListOffsetsResponseV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/ListOffsetsRequestV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/InitDisklessLogResponseV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/InitDisklessLogRequestV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/InitDisklessLogProducerStateV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/FindBatchesResponseV2Path.java |
Adds generated V2 response paths. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/FindBatchesResponseV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/FindBatchesRequestV2Path.java |
Adds generated V2 request paths. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/FindBatchesRequestV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/EnforceRetentionResponseV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/EnforceRetentionRequestV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/DeleteRecordsResponseV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/DeleteRecordsRequestV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/CommitBatchResponseV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/CommitBatchRequestV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/BatchMetadataV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/BatchInfoV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/AdvanceCrossTierLogStartResponseV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/AdvanceCrossTierLogStartRequestV1Path.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/ListOffsetsResponseV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/ListOffsetsRequestV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/InitDisklessLogResponseV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/InitDisklessLogRequestV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/InitDisklessLogProducerStateV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/FindBatchesResponseV2.java |
Defines generated V2 response fields. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/FindBatchesResponseV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/FindBatchesRequestV2.java |
Defines generated V2 request fields. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/FindBatchesRequestV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/EnforceRetentionResponseV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/EnforceRetentionRequestV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/DeleteRecordsResponseV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/DeleteRecordsRequestV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/CommitBatchResponseV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/CommitBatchRequestV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/BatchMetadataV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/BatchInfoV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/AdvanceCrossTierLogStartResponseV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/udt/AdvanceCrossTierLogStartRequestV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/RepairDisklessLogV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/RepairDisklessLogV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/PurgeDeletedLogsV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/PruneBatchesBelowHighestTieredOffsetV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/ProducerStateRecord.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/LogsRecord.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/ListOffsetsV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/InitDisklessLogV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/FindBatchesV3Record.java |
Adds the generated V3 result record. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/FindBatchesV2Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/FindBatchesV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/FilesRecord.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/EnforceRetentionV2Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/EnforceRetentionV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/DeleteRecordsV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/CommitFileV2Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/CommitFileV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/BatchesRecord.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/AdvanceCrossTierLogStartV1Record.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/PurgeDeletedLogsV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/PruneBatchesBelowHighestTieredOffsetV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/ProducerState.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/Logs.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/ListOffsetsV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/InitDisklessLogV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/FindBatchesV3.java |
Defines the generated V3 function table. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/FindBatchesV2.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/FindBatchesV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/Files.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/EnforceRetentionV2.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/EnforceRetentionV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/DeleteRecordsV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/CommitFileV2.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/CommitFileV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/Batches.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/tables/AdvanceCrossTierLogStartV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/Tables.java |
Exposes the generated V3 function. |
storage/inkless/src/main/jooq/org/jooq/generated/routines/MarkFileToDeleteV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/routines/FlushCommitRunV2.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/routines/DeleteTopicV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/routines/DeleteFilesV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/routines/DeleteBatchV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/routines/BatchTimestamp.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/Routines.java |
Exposes generated V3 routine calls. |
storage/inkless/src/main/jooq/org/jooq/generated/Keys.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/Indexes.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/enums/PruneBatchesBelowHighestTieredOffsetErrorV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/enums/ListOffsetsResponseErrorV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/enums/InitDisklessLogErrorV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/enums/FindBatchesResponseErrorV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/enums/FileStateT.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/enums/FileReasonT.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/enums/EnforceRetentionResponseErrorV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/enums/DeleteRecordsResponseErrorV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/enums/CommitBatchResponseErrorV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/enums/AdvanceCrossTierLogStartResponseErrorV1.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/Domains.java |
Updates generated schema metadata. |
storage/inkless/src/main/jooq/org/jooq/generated/DefaultSchema.java |
Registers V3 and its UDTs. |
storage/inkless/src/main/java/io/aiven/inkless/control_plane/postgres/FindBatchesJob.java |
Calls V3 and maps timestamps. |
storage/inkless/src/main/java/io/aiven/inkless/control_plane/InMemoryControlPlane.java |
Implements timestamp lookup in memory. |
storage/inkless/src/main/java/io/aiven/inkless/control_plane/FindBatchResponse.java |
Adds timestamp response fields. |
storage/inkless/src/main/java/io/aiven/inkless/control_plane/FindBatchRequest.java |
Adds the timestamp request flag. |
storage/inkless/src/main/java/io/aiven/inkless/consume/Reader.java |
Enables timestamps for consolidation. |
storage/inkless/src/main/java/io/aiven/inkless/consume/InklessFetchPartitionData.java |
Carries timestamp metadata. |
storage/inkless/src/main/java/io/aiven/inkless/consume/FindBatchesJob.java |
Propagates the request flag. |
storage/inkless/src/main/java/io/aiven/inkless/consume/FetchCompleter.java |
Returns timestamp-aware fetch data. |
docs/inkless/DISKLESS_CONSOLIDATION.md |
Documents the new gauge. |
core/src/test/scala/io/aiven/inkless/consolidation/DisklessLeaderEndPointTest.scala |
Tests sample lifecycle. |
core/src/test/scala/io/aiven/inkless/consolidation/ConsolidationFetcherThreadTest.scala |
Tests metric behavior and aggregation. |
core/src/main/scala/io/aiven/inkless/consolidation/DisklessLeaderEndPoint.scala |
Captures timestamp samples. |
core/src/main/scala/io/aiven/inkless/consolidation/ConsolidationMetrics.scala |
Implements the new gauges. |
core/src/main/scala/io/aiven/inkless/consolidation/ConsolidationFetcherThread.scala |
Reports samples after fetch processing. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
viktorsomogyi
force-pushed
the
svv/consolidation-time-lag
branch
from
October 13, 2026 13:42
e986ebe to
32804eb
Compare
viktorsomogyi
force-pushed
the
svv/consolidation-time-lag
branch
3 times, most recently
from
October 13, 2026 16:39
2833d27 to
0b64381
Compare
viktorsomogyi
marked this pull request as ready for review
October 15, 2026 08:08
viktorsomogyi
marked this pull request as draft
October 15, 2026 08:57
viktorsomogyi
force-pushed
the
svv/consolidation-time-lag
branch
3 times, most recently
from
October 16, 2026 14:39
fea0014 to
eb8331a
Compare
viktorsomogyi
force-pushed
the
svv/consolidation-time-lag
branch
from
October 16, 2026 14:45
eb8331a to
db83a80
Compare
viktorsomogyi
force-pushed
the
svv/consolidation-time-lag
branch
3 times, most recently
from
October 19, 2026 08:54
d33b8cd to
f2efe2b
Compare
Consolidation time lag is the timestamp span of diskless coordinates still absent from the local log. find_batches_v3 returns both ends when a consolidation fetch asks for them. Each end is that coordinate's aggregate timestamp. batch_timestamp() chooses batch_max_timestamp for CreateTime and log_append_timestamp for LogAppendTime. A coalesced coordinate contributes one timestamp, the greatest physical timestamp stored on that row. That estimate is the intended precision at minute scale. The head timestamp is the last coordinate below the diskless high watermark. The start timestamp is the first coordinate this fetch does not admit: the rejected coordinate already in hand on a byte-budget exit, the next coordinate after the last returned coordinate on a batch-cap exit, and the first pending coordinate when the response is empty. A fully admitted backlog leaves the start timestamp empty. Consumer fetches leave both timestamps unset and skip the extra lookups. A consolidation fetch adds a high-watermark lookup, and a start lookup when the loop does not admit a coordinate. That is up to three statements per partition. The ordered scan joins batches to files. When object storage returns a prefix of an admitted coalesced coordinate, the unpublished start is that coordinate's aggregate timestamp. V31 adds the backlog timestamps on the newer request, response, and routine, and keeps the ordered scan's early exit. The previous batch-lookup types and routines stay unchanged. Testing: - ./gradlew :storage:inkless:generateJooqClasses - ./gradlew :storage:inkless:test --tests io.aiven.inkless.control_plane.postgres.FindBatchesJobTest --tests io.aiven.inkless.control_plane.postgres.CommitFileJobCoalescingTest --tests io.aiven.inkless.consume.FetchCompleterTest - make vale - git diff --check Assisted-by: GPT-5.6 Assisted-by: Muse 1.3 Assisted-by: Claude Opus 5.5 Assisted-by: GPT-5.6 Luna Max Assisted-by: Composer 2.5 Fast Assisted-by: GPT-5.6 Sol High Assisted-by: Grok 4.7
Offset lag does not say how long unconsolidated data has been waiting. ConsolidationLocalTimeLag publishes the timestamp span of diskless coordinates still absent from the local log after this fetch, in milliseconds. The span uses each coordinate's aggregate timestamp and runs from the first pending coordinate to the last coordinate below the diskless high watermark. A coalesced coordinate contributes one timestamp, the greatest physical timestamp stored on that row. That estimate is the intended precision. One remaining batch publishes 0. An empty start publishes 0 when the fetched records reach the diskless end. A missing, NO_TIMESTAMP, or negative boundary keeps the previous published value. Registration and a caught-up partition publish 0. The broker aggregate is the maximum partition value. Offset lag stays a sum. selectRecordsWithinSegmentBudget keeps the batches that fit the segment byte budget and carries the timestamp of the first batch that does not. A prefix that does not land starts the span at that batch. The fetcher takes this fetch's timestamps before the append, clears them for that fetch offset, and publishes with the post-append local log end. A batch the follower rejects as larger than the segment still publishes before the retriable error. The span starts at the first fetched batch, the one that did not land, and the backlog counts as not consumed. While the partition is parked, the gauge keeps that span instead of the last successful value. Testing: - ./gradlew :core:test --tests io.aiven.inkless.consolidation.DisklessLeaderEndPointTest --tests io.aiven.inkless.consolidation.ConsolidationFetcherThreadTest - make vale Assisted-by: GPT-5.6 Assisted-by: Muse 1.3 Assisted-by: Claude Opus 5.5 Assisted-by: GPT-5.6 Luna Max Assisted-by: Composer 2.5 Fast Assisted-by: GPT-5.6 Sol High Assisted-by: Grok 4.7
Pin ConsolidationLocalTimeLag in the consolidation system tests. Document the span from the first pending coordinate after this fetch to the last coordinate below disklessLEO. The span uses each coordinate's aggregate timestamp. A coalesced coordinate is one timestamp, the greatest physical timestamp stored on that row. The gauge reads 0 after registration, when caught up, and when one batch remains. It retains its published value when a sample is incomplete. The broker aggregate is the maximum. The two system tests share one aggregate-export check. They assert that the gauge is exported. Testing: - python3 -m py_compile tests/kafkatest/services/inkless/consolidation_verifier.py tests/kafkatest/tests/inkless/consolidation_pipeline_test.py tests/kafkatest/tests/inkless/consolidation_read_from_remote_after_prune_test.py - make vale Assisted-by: GPT-5.6 Assisted-by: Muse 1.3 Assisted-by: Claude Opus 5.5 Assisted-by: GPT-5.6 Luna Max Assisted-by: Composer 2.5 Fast Assisted-by: Grok 4.7
viktorsomogyi
force-pushed
the
svv/consolidation-time-lag
branch
from
October 19, 2026 13:54
f2efe2b to
60636c4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Offset lag does not say how long unconsolidated WAL data has been waiting.
ConsolidationLocalTimeLagpublishes that span in milliseconds, using each batch coordinate's aggregate timestamp. The gauge is not age against the current time.A coalesced batch coordinate stores one timestamp, the greatest physical timestamp on the row.
batch_timestamp()selectsbatch_max_timestampforCreateTimeandlog_append_timestampforLogAppendTime.One remaining batch publishes 0. An empty start publishes 0 when the fetched records reach the diskless end. A missing,
NO_TIMESTAMP, or negative sample keeps the previous value. Registration and a caught-up partition publish 0. Offset lag stays a sum.Evidence
ConsolidationLocalLagcounts offsets. A coalesced batch coordinate has no separate first and last physical timestamp.After:
FindBatchesJobTest.flagOnUsesAggregateTimestampFromACoalescedRowexpects head1000.flagOnUsesAggregateTimestampWhenBudgetRejectsACoalescedRowandFetchCompleterTest.testPartialCoalescedReadStartsAtCoordinateAggregateTimestampexpect start300.FindBatchesJobTest,CommitFileJobCoalescingTest, andFetchCompleterTestpassed.make valereported no alerts.Merge Danger
Door: two-way
Reverting the broker removes the gauge. After
V31runs, Flyway keepsfind_batches_v3and the V2 request and response types. A database that already applied an earlierV31from this branch must be recreated.Blast Radius: coordinator
Every diskless fetch calls
find_batches_v3. A consumer fetch runs the scan and leaves both timestamps empty. A consolidation fetch adds a high-watermark lookup, and a start lookup when the scan does not admit a batch coordinate: up to three statements per partition.