Skip to content

feat(inkless:consolidation): expose local time lag - #826

Closed
viktorsomogyi wants to merge 3 commits into
mainfrom
svv/consolidation-time-lag
Closed

viktorsomogyi wants to merge 3 commits into
mainfrom
svv/consolidation-time-lag

Conversation

@viktorsomogyi

@viktorsomogyi viktorsomogyi commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Offset lag does not say how long unconsolidated WAL data has been waiting. ConsolidationLocalTimeLag publishes that span in milliseconds, using each batch coordinate's aggregate timestamp. The gauge is not age against the current time.

head  = last batch coordinate below disklessLEO
start = first batch coordinate this fetch does not land
lag   = max(0, head.timestamp - start.timestamp)

A coalesced batch coordinate stores one timestamp, the greatest physical timestamp on the row. batch_timestamp() selects batch_max_timestamp for CreateTime and log_append_timestamp for LogAppendTime.

find_batches_v3
  scan batch coordinates by last_offset
  consolidation fetch
    head from the last coordinate below the diskless high watermark
    start from the coordinate the fetch does not admit
  consumer fetch
    leave both timestamps empty
ConsolidationFetcherThread
  publish after the append
  broker aggregate = maximum partition value

One remaining batch publishes 0. An empty start publishes 0 when the fetched records reach the diskless end. A missing, NO_TIMESTAMP, or negative sample keeps the previous value. Registration and a caught-up partition publish 0. Offset lag stays a sum.

Evidence

  • Before: ConsolidationLocalLag counts offsets. A coalesced batch coordinate has no separate first and last physical timestamp.
    After: FindBatchesJobTest.flagOnUsesAggregateTimestampFromACoalescedRow expects head 1000. flagOnUsesAggregateTimestampWhenBudgetRejectsACoalescedRow and FetchCompleterTest.testPartialCoalescedReadStartsAtCoordinateAggregateTimestamp expect start 300. FindBatchesJobTest, CommitFileJobCoalescingTest, and FetchCompleterTest passed. make vale reported no alerts.

Merge Danger

Door: two-way

Reverting the broker removes the gauge. After V31 runs, Flyway keeps find_batches_v3 and the V2 request and response types. A database that already applied an earlier V31 from this branch must be recreated.

Blast Radius: coordinator

Every diskless fetch calls find_batches_v3. A consumer fetch runs the scan and leaves both timestamps empty. A consolidation fetch adds a high-watermark lookup, and a start lookup when the scan does not admit a batch coordinate: up to three statements per partition.

@viktorsomogyi
viktorsomogyi marked this pull request as draft October 13, 2026 11:39
@viktorsomogyi
viktorsomogyi requested a balanced review from Copilot October 13, 2026 13:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The operational performance guide still observes and tunes find_batches_v2, although production now exclusively calls V3.

1 open finding
What changed in this PR

Adds a consolidation backlog time-lag gauge backed by timestamp boundaries returned from find_batches_v3.

Changes:

  • Adds timestamp-aware control-plane request and response types.
  • Propagates timestamps through fetch handling into consolidation metrics.
  • Adds unit, integration, benchmark, and system-test coverage.
File Description
tests/​kafkatest/​tests/​inkless/​consolidation_read_from_remote_after_prune_test.py Checks JMX metric export.
tests/​kafkatest/​tests/​inkless/​consolidation_pipeline_test.py Checks pipeline metric export.
tests/​kafkatest/​services/​inkless/​consolidation_verifier.py Defines the metric name.
storage/​inkless/​src/​test/​java/​io/​aiven/​inkless/​control_plane/​postgres/​FindBatchesScanDepthBenchmarkTest.java Updates benchmarks for V3.
storage/​inkless/​src/​test/​java/​io/​aiven/​inkless/​control_plane/​postgres/​FindBatchesJobTest.java Tests PostgreSQL timestamp boundaries.
storage/​inkless/​src/​test/​java/​io/​aiven/​inkless/​control_plane/​AbstractControlPlaneTest.java Tests shared control-plane behavior.
storage/​inkless/​src/​test/​java/​io/​aiven/​inkless/​consume/​FindBatchesJobTest.java Tests consolidation request flags.
storage/​inkless/​src/​test/​java/​io/​aiven/​inkless/​consume/​FetchCompleterTest.java Tests timestamp propagation.
storage/​inkless/​src/​main/​resources/​db/​migration/​V32__Find_batches_high_watermark_timestamp.sql Adds V3 database types and function.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​UDTs.java Registers generated V2 UDTs.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​RepairDisklessLogResponseV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​RepairDisklessLogRequestV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​RepairDisklessLogResponseV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​RepairDisklessLogRequestV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​PurgeDeletedLogsResponseV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​PruneBatchesBelowHighestTieredOffsetResponseV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​PruneBatchesBelowHighestTieredOffsetRequestV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​ListOffsetsResponseV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​ListOffsetsRequestV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​InitDisklessLogResponseV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​InitDisklessLogRequestV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​InitDisklessLogProducerStateV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​FindBatchesResponseV2Record.java Models the V2 response record.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​FindBatchesResponseV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​FindBatchesRequestV2Record.java Models the V2 request record.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​FindBatchesRequestV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​EnforceRetentionResponseV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​EnforceRetentionRequestV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​DeleteRecordsResponseV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​DeleteRecordsRequestV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​CommitBatchResponseV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​CommitBatchRequestV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​BatchMetadataV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​BatchInfoV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​AdvanceCrossTierLogStartResponseV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​records/​AdvanceCrossTierLogStartRequestV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​PurgeDeletedLogsResponseV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​PruneBatchesBelowHighestTieredOffsetResponseV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​PruneBatchesBelowHighestTieredOffsetRequestV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​RepairDisklessLogResponseV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​RepairDisklessLogRequestV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​PurgeDeletedLogsResponseV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​PruneBatchesBelowHighestTieredOffsetResponseV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​PruneBatchesBelowHighestTieredOffsetRequestV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​ListOffsetsResponseV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​ListOffsetsRequestV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​InitDisklessLogResponseV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​InitDisklessLogRequestV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​InitDisklessLogProducerStateV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​FindBatchesResponseV2Path.java Adds generated V2 response paths.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​FindBatchesResponseV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​FindBatchesRequestV2Path.java Adds generated V2 request paths.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​FindBatchesRequestV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​EnforceRetentionResponseV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​EnforceRetentionRequestV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​DeleteRecordsResponseV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​DeleteRecordsRequestV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​CommitBatchResponseV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​CommitBatchRequestV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​BatchMetadataV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​BatchInfoV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​AdvanceCrossTierLogStartResponseV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​paths/​AdvanceCrossTierLogStartRequestV1Path.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​ListOffsetsResponseV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​ListOffsetsRequestV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​InitDisklessLogResponseV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​InitDisklessLogRequestV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​InitDisklessLogProducerStateV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​FindBatchesResponseV2.java Defines generated V2 response fields.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​FindBatchesResponseV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​FindBatchesRequestV2.java Defines generated V2 request fields.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​FindBatchesRequestV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​EnforceRetentionResponseV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​EnforceRetentionRequestV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​DeleteRecordsResponseV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​DeleteRecordsRequestV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​CommitBatchResponseV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​CommitBatchRequestV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​BatchMetadataV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​BatchInfoV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​AdvanceCrossTierLogStartResponseV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​udt/​AdvanceCrossTierLogStartRequestV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​RepairDisklessLogV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​RepairDisklessLogV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​PurgeDeletedLogsV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​PruneBatchesBelowHighestTieredOffsetV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​ProducerStateRecord.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​LogsRecord.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​ListOffsetsV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​InitDisklessLogV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​FindBatchesV3Record.java Adds the generated V3 result record.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​FindBatchesV2Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​FindBatchesV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​FilesRecord.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​EnforceRetentionV2Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​EnforceRetentionV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​DeleteRecordsV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​CommitFileV2Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​CommitFileV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​BatchesRecord.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​records/​AdvanceCrossTierLogStartV1Record.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​PurgeDeletedLogsV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​PruneBatchesBelowHighestTieredOffsetV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​ProducerState.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​Logs.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​ListOffsetsV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​InitDisklessLogV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​FindBatchesV3.java Defines the generated V3 function table.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​FindBatchesV2.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​FindBatchesV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​Files.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​EnforceRetentionV2.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​EnforceRetentionV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​DeleteRecordsV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​CommitFileV2.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​CommitFileV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​Batches.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​tables/​AdvanceCrossTierLogStartV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​Tables.java Exposes the generated V3 function.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​routines/​MarkFileToDeleteV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​routines/​FlushCommitRunV2.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​routines/​DeleteTopicV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​routines/​DeleteFilesV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​routines/​DeleteBatchV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​routines/​BatchTimestamp.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​Routines.java Exposes generated V3 routine calls.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​Keys.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​Indexes.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​enums/​PruneBatchesBelowHighestTieredOffsetErrorV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​enums/​ListOffsetsResponseErrorV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​enums/​InitDisklessLogErrorV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​enums/​FindBatchesResponseErrorV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​enums/​FileStateT.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​enums/​FileReasonT.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​enums/​EnforceRetentionResponseErrorV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​enums/​DeleteRecordsResponseErrorV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​enums/​CommitBatchResponseErrorV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​enums/​AdvanceCrossTierLogStartResponseErrorV1.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​Domains.java Updates generated schema metadata.
storage/​inkless/​src/​main/​jooq/​org/​jooq/​generated/​DefaultSchema.java Registers V3 and its UDTs.
storage/​inkless/​src/​main/​java/​io/​aiven/​inkless/​control_plane/​postgres/​FindBatchesJob.java Calls V3 and maps timestamps.
storage/​inkless/​src/​main/​java/​io/​aiven/​inkless/​control_plane/​InMemoryControlPlane.java Implements timestamp lookup in memory.
storage/​inkless/​src/​main/​java/​io/​aiven/​inkless/​control_plane/​FindBatchResponse.java Adds timestamp response fields.
storage/​inkless/​src/​main/​java/​io/​aiven/​inkless/​control_plane/​FindBatchRequest.java Adds the timestamp request flag.
storage/​inkless/​src/​main/​java/​io/​aiven/​inkless/​consume/​Reader.java Enables timestamps for consolidation.
storage/​inkless/​src/​main/​java/​io/​aiven/​inkless/​consume/​InklessFetchPartitionData.java Carries timestamp metadata.
storage/​inkless/​src/​main/​java/​io/​aiven/​inkless/​consume/​FindBatchesJob.java Propagates the request flag.
storage/​inkless/​src/​main/​java/​io/​aiven/​inkless/​consume/​FetchCompleter.java Returns timestamp-aware fetch data.
docs/​inkless/​DISKLESS_CONSOLIDATION.md Documents the new gauge.
core/​src/​test/​scala/​io/​aiven/​inkless/​consolidation/​DisklessLeaderEndPointTest.scala Tests sample lifecycle.
core/​src/​test/​scala/​io/​aiven/​inkless/​consolidation/​ConsolidationFetcherThreadTest.scala Tests metric behavior and aggregation.
core/​src/​main/​scala/​io/​aiven/​inkless/​consolidation/​DisklessLeaderEndPoint.scala Captures timestamp samples.
core/​src/​main/​scala/​io/​aiven/​inkless/​consolidation/​ConsolidationMetrics.scala Implements the new gauges.
core/​src/​main/​scala/​io/​aiven/​inkless/​consolidation/​ConsolidationFetcherThread.scala Reports samples after fetch processing.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The lag calculation must treat Kafka’s no-timestamp sentinel as an incomplete boundary to avoid publishing epoch-scale values.

1 open finding
1 resolved since last review

🧠 Review effort: Balanced

@viktorsomogyi
viktorsomogyi force-pushed the svv/consolidation-time-lag branch 3 times, most recently from 2833d27 to 0b64381 Compare October 13, 2026 16:39
@viktorsomogyi
viktorsomogyi requested a balanced review from Copilot October 13, 2026 16:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The coordinated database migration, fetch-path metadata propagation, and operational metric semantics warrant final human validation.

0 open findings

1 resolved since last review

🧠 Review effort: Balanced

@viktorsomogyi
viktorsomogyi marked this pull request as ready for review October 15, 2026 08:08
@viktorsomogyi
viktorsomogyi requested a review from jeqo October 15, 2026 08:38
@viktorsomogyi
viktorsomogyi marked this pull request as draft October 15, 2026 08:57
@viktorsomogyi
viktorsomogyi removed the request for review from jeqo October 15, 2026 08:57
@viktorsomogyi
viktorsomogyi force-pushed the svv/consolidation-time-lag branch 3 times, most recently from fea0014 to eb8331a Compare October 16, 2026 14:39
@viktorsomogyi
viktorsomogyi requested a balanced review from Copilot October 16, 2026 14:40
@viktorsomogyi
viktorsomogyi force-pushed the svv/consolidation-time-lag branch from eb8331a to db83a80 Compare October 16, 2026 14:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Coalesced batch timestamps can overstate or hide the pending time span, making the new metric inaccurate.

1 open finding

🧠 Review effort: Balanced

@viktorsomogyi
viktorsomogyi force-pushed the svv/consolidation-time-lag branch 3 times, most recently from d33b8cd to f2efe2b Compare October 19, 2026 08:54
Consolidation time lag is the timestamp span of diskless coordinates still absent from the local log. find_batches_v3 returns both ends when a consolidation fetch asks for them. Each end is that coordinate's aggregate timestamp. batch_timestamp() chooses batch_max_timestamp for CreateTime and log_append_timestamp for LogAppendTime. A coalesced coordinate contributes one timestamp, the greatest physical timestamp stored on that row. That estimate is the intended precision at minute scale.

The head timestamp is the last coordinate below the diskless high watermark. The start timestamp is the first coordinate this fetch does not admit: the rejected coordinate already in hand on a byte-budget exit, the next coordinate after the last returned coordinate on a batch-cap exit, and the first pending coordinate when the response is empty. A fully admitted backlog leaves the start timestamp empty. Consumer fetches leave both timestamps unset and skip the extra lookups.

A consolidation fetch adds a high-watermark lookup, and a start lookup when the loop does not admit a coordinate. That is up to three statements per partition. The ordered scan joins batches to files. When object storage returns a prefix of an admitted coalesced coordinate, the unpublished start is that coordinate's aggregate timestamp.

V31 adds the backlog timestamps on the newer request, response, and routine, and keeps the ordered scan's early exit. The previous batch-lookup types and routines stay unchanged.

Testing:
- ./gradlew :storage:inkless:generateJooqClasses
- ./gradlew :storage:inkless:test --tests io.aiven.inkless.control_plane.postgres.FindBatchesJobTest --tests io.aiven.inkless.control_plane.postgres.CommitFileJobCoalescingTest --tests io.aiven.inkless.consume.FetchCompleterTest
- make vale
- git diff --check

Assisted-by: GPT-5.6
Assisted-by: Muse 1.3
Assisted-by: Claude Opus 5.5
Assisted-by: GPT-5.6 Luna Max
Assisted-by: Composer 2.5 Fast
Assisted-by: GPT-5.6 Sol High
Assisted-by: Grok 4.7
Offset lag does not say how long unconsolidated data has been waiting. ConsolidationLocalTimeLag publishes the timestamp span of diskless coordinates still absent from the local log after this fetch, in milliseconds. The span uses each coordinate's aggregate timestamp and runs from the first pending coordinate to the last coordinate below the diskless high watermark. A coalesced coordinate contributes one timestamp, the greatest physical timestamp stored on that row. That estimate is the intended precision. One remaining batch publishes 0. An empty start publishes 0 when the fetched records reach the diskless end. A missing, NO_TIMESTAMP, or negative boundary keeps the previous published value. Registration and a caught-up partition publish 0. The broker aggregate is the maximum partition value. Offset lag stays a sum.

selectRecordsWithinSegmentBudget keeps the batches that fit the segment byte budget and carries the timestamp of the first batch that does not. A prefix that does not land starts the span at that batch. The fetcher takes this fetch's timestamps before the append, clears them for that fetch offset, and publishes with the post-append local log end.

A batch the follower rejects as larger than the segment still publishes before the retriable error. The span starts at the first fetched batch, the one that did not land, and the backlog counts as not consumed. While the partition is parked, the gauge keeps that span instead of the last successful value.

Testing:
- ./gradlew :core:test --tests io.aiven.inkless.consolidation.DisklessLeaderEndPointTest --tests io.aiven.inkless.consolidation.ConsolidationFetcherThreadTest
- make vale

Assisted-by: GPT-5.6
Assisted-by: Muse 1.3
Assisted-by: Claude Opus 5.5
Assisted-by: GPT-5.6 Luna Max
Assisted-by: Composer 2.5 Fast
Assisted-by: GPT-5.6 Sol High
Assisted-by: Grok 4.7
Pin ConsolidationLocalTimeLag in the consolidation system tests. Document the span from the first pending coordinate after this fetch to the last coordinate below disklessLEO. The span uses each coordinate's aggregate timestamp. A coalesced coordinate is one timestamp, the greatest physical timestamp stored on that row. The gauge reads 0 after registration, when caught up, and when one batch remains. It retains its published value when a sample is incomplete. The broker aggregate is the maximum.

The two system tests share one aggregate-export check. They assert that the gauge is exported.

Testing:
- python3 -m py_compile tests/kafkatest/services/inkless/consolidation_verifier.py tests/kafkatest/tests/inkless/consolidation_pipeline_test.py tests/kafkatest/tests/inkless/consolidation_read_from_remote_after_prune_test.py
- make vale

Assisted-by: GPT-5.6
Assisted-by: Muse 1.3
Assisted-by: Claude Opus 5.5
Assisted-by: GPT-5.6 Luna Max
Assisted-by: Composer 2.5 Fast
Assisted-by: Grok 4.7
@viktorsomogyi
viktorsomogyi force-pushed the svv/consolidation-time-lag branch from f2efe2b to 60636c4 Compare October 19, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants