Skip to content

feat(inkless:consolidation): [KC-298] add JMX metrics for the cross-tier log start offset #682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jeqo merged 1 commit into from
Jul 8, 2026
Merged

feat(inkless:consolidation): [KC-298] add JMX metrics for the cross-tier log start offset #682

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 64 additions & 32 deletions docs/inkless/metrics.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,21 @@ RetentionEnforcementTotalTime Total time spent on a retention enforc
======================================== =================================================================


CrossTierLogStartReporter metrics
==================================

io.aiven.inkless.delete:type=CrossTierLogStartReporter
------------------------------------------------------

=================== ================================================================================================
Attribute name Description
=================== ================================================================================================
PartitionsReported Total number of partition cross-tier log start offsets accepted by the control plane
PendingPartitions Current number of partitions with a buffered cross-tier log start offset awaiting the next flush
ReportErrors Total number of errors while reporting cross-tier log start offsets to the control plane
=================== ================================================================================================


ClientAzAwareness metrics
==================================

Expand All @@ -259,38 +274,40 @@ PostgresControlPlane metrics
io.aiven.inkless.control_plane.postgres:type=PostgresControlPlane
-----------------------------------------------------------------

============================= ==================================================================
Attribute name Description
============================= ==================================================================
CommitFileQueryRate Total number of CommitFile queries executed
CommitFileQueryTime Time spent executing the CommitFile query in milliseconds
DeleteRecordsQueryRate Total number of DeleteRecords queries executed
DeleteRecordsQueryTime Time spent executing the DeleteRecords query in milliseconds
EnforceRetentionQueryRate Total number of EnforceRetention queries executed
EnforceRetentionQueryTime Time spent executing the EnforceRetention query in milliseconds
FilesDeleteQueryRate Total number of FilesDelete queries executed
FilesDeleteQueryTime Time spent executing the FilesDelete query in milliseconds
FindBatchesQueryRate Total number of FindBatches queries executed
FindBatchesQueryTime Time spent executing the FindBatches query in milliseconds
GetFilesToDeleteQueryRate Total number of GetFilesToDelete queries executed
GetFilesToDeleteQueryTime Time spent executing the GetFilesToDelete query in milliseconds
GetLogInfoQueryRate Total number of GetLogInfo queries executed
GetLogInfoQueryTime Time spent executing the GetLogInfo query in milliseconds
GetLogsQueryRate Total number of GetLogs queries executed
GetLogsQueryTime Time spent executing the GetLogs query in milliseconds
GetProducerStateQueryRate Total number of GetProducerState queries executed
GetProducerStateQueryTime Time spent executing the GetProducerState query in milliseconds
InitDisklessLogQueryRate Total number of InitDisklessLog queries executed
InitDisklessLogQueryTime Time spent executing the InitDisklessLog query in milliseconds
ListOffsetsQueryRate Total number of ListOffsets queries executed
ListOffsetsQueryTime Time spent executing the ListOffsets query in milliseconds
SafeDeleteFileCheckQueryRate Total number of SafeDeleteFileCheck queries executed
SafeDeleteFileCheckQueryTime Time spent executing the SafeDeleteFileCheck query in milliseconds
TopicCreateQueryRate Total number of TopicCreate queries executed
TopicCreateQueryTime Time spent executing the TopicCreate query in milliseconds
TopicDeleteQueryRate Total number of TopicDelete queries executed
TopicDeleteQueryTime Time spent executing the TopicDelete query in milliseconds
============================= ==================================================================
================================== =======================================================================
Attribute name Description
================================== =======================================================================
AdvanceCrossTierLogStartQueryRate Total number of AdvanceCrossTierLogStart queries executed
AdvanceCrossTierLogStartQueryTime Time spent executing the AdvanceCrossTierLogStart query in milliseconds
CommitFileQueryRate Total number of CommitFile queries executed
CommitFileQueryTime Time spent executing the CommitFile query in milliseconds
DeleteRecordsQueryRate Total number of DeleteRecords queries executed
DeleteRecordsQueryTime Time spent executing the DeleteRecords query in milliseconds
EnforceRetentionQueryRate Total number of EnforceRetention queries executed
EnforceRetentionQueryTime Time spent executing the EnforceRetention query in milliseconds
FilesDeleteQueryRate Total number of FilesDelete queries executed
FilesDeleteQueryTime Time spent executing the FilesDelete query in milliseconds
FindBatchesQueryRate Total number of FindBatches queries executed
FindBatchesQueryTime Time spent executing the FindBatches query in milliseconds
GetFilesToDeleteQueryRate Total number of GetFilesToDelete queries executed
GetFilesToDeleteQueryTime Time spent executing the GetFilesToDelete query in milliseconds
GetLogInfoQueryRate Total number of GetLogInfo queries executed
GetLogInfoQueryTime Time spent executing the GetLogInfo query in milliseconds
GetLogsQueryRate Total number of GetLogs queries executed
GetLogsQueryTime Time spent executing the GetLogs query in milliseconds
GetProducerStateQueryRate Total number of GetProducerState queries executed
GetProducerStateQueryTime Time spent executing the GetProducerState query in milliseconds
InitDisklessLogQueryRate Total number of InitDisklessLog queries executed
InitDisklessLogQueryTime Time spent executing the InitDisklessLog query in milliseconds
ListOffsetsQueryRate Total number of ListOffsets queries executed
ListOffsetsQueryTime Time spent executing the ListOffsets query in milliseconds
SafeDeleteFileCheckQueryRate Total number of SafeDeleteFileCheck queries executed
SafeDeleteFileCheckQueryTime Time spent executing the SafeDeleteFileCheck query in milliseconds
TopicCreateQueryRate Total number of TopicCreate queries executed
TopicCreateQueryTime Time spent executing the TopicCreate query in milliseconds
TopicDeleteQueryRate Total number of TopicDelete queries executed
TopicDeleteQueryTime Time spent executing the TopicDelete query in milliseconds
================================== =======================================================================


PostgresConnectionPool metrics
Expand Down Expand Up @@ -332,6 +349,21 @@ CacheSize Current number of entries in the batch coordinate cache
===================== ================================================================================


CrossTierLogStartCache metrics
==================================

io.aiven.inkless.cache:type=CrossTierLogStartCache
--------------------------------------------------

=============== ========================================================================================
Attribute name Description
=============== ========================================================================================
CacheHits Total number of cross-tier log start cache hits served without a control-plane call
CacheMisses Total number of cross-tier log start cache misses that fell through to the control plane
CacheSize Current number of partitions cached for the cross-tier log start offset
=============== ========================================================================================


CaffeineCache metrics
==================================

Expand Down
11 changes: 10 additions & 1 deletion storage/inkless/src/main/java/io/aiven/inkless/cache/CaffeineCrossTierLogStartCache.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
public class CaffeineCrossTierLogStartCache implements CrossTierLogStartCache {

private final Cache<TopicIdPartition, Long> cache;
private final CrossTierLogStartCacheMetrics metrics;

public CaffeineCrossTierLogStartCache(final Duration ttl) {
this(ttl, Ticker.systemTicker());
Expand All @@ -47,11 +48,18 @@ public CaffeineCrossTierLogStartCache(final Duration ttl) {
.expireAfterWrite(ttl)
.ticker(ticker)
.build();
this.metrics = new CrossTierLogStartCacheMetrics(cache::estimatedSize);
}

@Override
public Long get(final TopicIdPartition topicIdPartition) {
return cache.getIfPresent(topicIdPartition);
final Long offset = cache.getIfPresent(topicIdPartition);
if (offset == null) {
metrics.recordCacheMiss();
} else {
metrics.recordCacheHit();
}
return offset;
}

@Override
Expand All @@ -66,5 +74,6 @@ public void put(final TopicIdPartition topicIdPartition, final long offset) {
public void close() throws IOException {
cache.invalidateAll();
cache.cleanUp();
metrics.close();
}
}
83 changes: 83 additions & 0 deletions storage/inkless/src/main/java/io/aiven/inkless/cache/CrossTierLogStartCacheMetrics.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
/*
* Inkless
* Copyright (C) 2024 - 2026 Aiven OY
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
package io.aiven.inkless.cache;

import org.apache.kafka.common.MetricNameTemplate;
import org.apache.kafka.server.metrics.KafkaMetricsGroup;

import java.io.Closeable;
import java.util.List;
import java.util.concurrent.atomic.LongAdder;
import java.util.function.Supplier;

/**
* JMX metrics for the read-through {@link CrossTierLogStartCache}. A hit is served entirely from the
* cache without a control-plane {@code listOffsets} call; a miss falls through to the control plane
* and (on success) populates the cache.
*/
public final class CrossTierLogStartCacheMetrics implements Closeable {
private static final String GROUP = CrossTierLogStartCache.class.getSimpleName();

static final String CACHE_HITS = "CacheHits";
private static final String CACHE_HITS_DOC =
"Total number of cross-tier log start cache hits served without a control-plane call";
static final String CACHE_MISSES = "CacheMisses";
private static final String CACHE_MISSES_DOC =
"Total number of cross-tier log start cache misses that fell through to the control plane";
static final String CACHE_SIZE = "CacheSize";
private static final String CACHE_SIZE_DOC =
"Current number of partitions cached for the cross-tier log start offset";

/**
* This method returns a list of all the metric name templates for the CrossTierLogStartCacheMetrics class.
* This is used for documentation purposes only.
*/
public static List<MetricNameTemplate> all() {
return List.of(
new MetricNameTemplate(CACHE_HITS, GROUP, CACHE_HITS_DOC),
new MetricNameTemplate(CACHE_MISSES, GROUP, CACHE_MISSES_DOC),
new MetricNameTemplate(CACHE_SIZE, GROUP, CACHE_SIZE_DOC)
);
}

private final KafkaMetricsGroup metricsGroup = new KafkaMetricsGroup(
CrossTierLogStartCache.class.getPackageName(), CrossTierLogStartCache.class.getSimpleName());
private final LongAdder cacheHits = new LongAdder();
private final LongAdder cacheMisses = new LongAdder();

public CrossTierLogStartCacheMetrics(final Supplier<Long> sizeSupplier) {
metricsGroup.newGauge(CACHE_HITS, cacheHits::intValue);
metricsGroup.newGauge(CACHE_MISSES, cacheMisses::intValue);
metricsGroup.newGauge(CACHE_SIZE, () -> sizeSupplier.get().intValue());
}

public void recordCacheHit() {
cacheHits.increment();
}

public void recordCacheMiss() {
cacheMisses.increment();
}

@Override
public void close() {
metricsGroup.removeMetric(CACHE_HITS);
metricsGroup.removeMetric(CACHE_MISSES);
metricsGroup.removeMetric(CACHE_SIZE);
}
}
1 change: 1 addition & 0 deletions storage/inkless/src/main/java/io/aiven/inkless/common/SharedState.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ public void close() throws IOException {
Utils.closeQuietly(storageMetrics, "storageMetrics");
Utils.closeQuietly(batchCoordinateCache, "batchCoordinateCache");
Utils.closeQuietly(crossTierLogStartCache, "crossTierLogStartCache");
Utils.closeQuietly(crossTierLogStartReporter, "crossTierLogStartReporter");
Utils.closeQuietly(cache, "objectCache");
}

Expand Down
2 changes: 1 addition & 1 deletion ...e/inkless/src/main/java/io/aiven/inkless/control_plane/postgres/PostgresControlPlane.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ public List<EnforceRetentionResponse> enforceRetention(final List<EnforceRetenti
public List<AdvanceCrossTierLogStartOffsetResponse> advanceCrossTierLogStartOffset(final List<AdvanceCrossTierLogStartOffsetRequest> requests) {
try {
final AdvanceCrossTierLogStartOffsetJob job = new AdvanceCrossTierLogStartOffsetJob(
time, writeJooqCtx, requests, duration -> { });
time, writeJooqCtx, requests, pgMetrics::onAdvanceCrossTierLogStartCompleted);
return job.call();
} catch (final Exception e) {
throw new ControlPlaneException("Failed to advance cross-tier log start offset", e);
Expand Down
9 changes: 8 additions & 1 deletion ...ss/src/main/java/io/aiven/inkless/control_plane/postgres/PostgresControlPlaneMetrics.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ public class PostgresControlPlaneMetrics implements Closeable {
"TopicCreate", "TopicDelete", "FilesDelete", "ListOffsets",
"DeleteRecords", "EnforceRetention", "GetFilesToDelete",
"SafeDeleteFileCheck",
"GetLogInfo", "InitDisklessLog", "GetProducerState"
"GetLogInfo", "InitDisklessLog", "GetProducerState",
"AdvanceCrossTierLogStart"
);

/**
Expand Down Expand Up @@ -75,6 +76,7 @@ public static List<MetricNameTemplate> all() {
private final QueryMetrics initDisklessLogMetrics = new QueryMetrics("InitDisklessLog");
private final QueryMetrics getProducerStateMetrics = new QueryMetrics("GetProducerState");
private final QueryMetrics pruneDisklessLogsMetrics = new QueryMetrics("PruneDisklessLogs");
private final QueryMetrics advanceCrossTierLogStartMetrics = new QueryMetrics("AdvanceCrossTierLogStart");

public PostgresControlPlaneMetrics(Time time) {
this.time = Objects.requireNonNull(time, "time cannot be null");
Expand Down Expand Up @@ -140,6 +142,10 @@ public void onPruneDisklessLogsCompleted(Long duration) {
pruneDisklessLogsMetrics.record(duration);
}

public void onAdvanceCrossTierLogStartCompleted(Long duration) {
advanceCrossTierLogStartMetrics.record(duration);
}

@Override
public void close() {
findBatchesMetrics.remove();
Expand All @@ -157,6 +163,7 @@ public void close() {
initDisklessLogMetrics.remove();
getProducerStateMetrics.remove();
pruneDisklessLogsMetrics.remove();
advanceCrossTierLogStartMetrics.remove();
}

private class QueryMetrics {
Expand Down
Loading
Loading