fix(inkless:switch): clear under-replicated partitions after classic-to-diskless switch - #697
Open
giuseppelillo wants to merge 1 commit into
Open
fix(inkless:switch): clear under-replicated partitions after classic-to-diskless switch#697giuseppelillo wants to merge 1 commit into
giuseppelillo wants to merge 1 commit into
Conversation
giuseppelillo
force-pushed
the
glillo/resolve-urp-switch
branch
from
July 14, 2026 13:54
a9a968f to
8fb0f9e
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes an operational issue where ReplicaManager’s UnderReplicatedPartitions aggregate could remain non-zero (or otherwise not reflect live ISR state) after switching a topic from classic to diskless, by ensuring the leader continues to observe the follower’s fetch state until the replica is back in ISR.
Changes:
- Update follower fetch routing to treat
fetchOffset == classicToDisklessStartOffsetas still eligible for unified-log reads for follower requests, enabling the leader to observe fetch progress at the seal offset. - Ensure diskless followers schedule a fetch even when caught up to the seal but currently out of ISR, so the leader can expand ISR again.
- Add a system test that stops a follower after switch, asserts URP rises, then restarts the follower and asserts URP clears after ISR fully recovers.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| tests/kafkatest/tests/inkless/inkless_topic_switch_test.py | Adds URP JMX scraping + a new test to verify URP rises on replica loss and clears after replica recovery post-switch. |
| core/src/main/scala/kafka/server/ReplicaManager.scala | Adjusts diskless follower fetch eligibility at the seal offset and ensures fetching is scheduled when out of ISR even if already caught up. |
| core/src/main/scala/kafka/server/ReplicaFetcherThread.scala | Prevents evicting switched partitions from the replica fetcher until the replica is back in ISR. |
giuseppelillo
force-pushed
the
glillo/resolve-urp-switch
branch
from
July 15, 2026 12:57
8fb0f9e to
012e6d5
Compare
giuseppelillo
marked this pull request as ready for review
August 6, 2026 14:53
…to-diskless switch A follower that dropped out of ISR and recovered after a classic-to-diskless switch stayed under-replicated forever, keeping UnderReplicatedPartitions stuck. - Leader fetch handler: record a switched follower's fetch state at the seal so ISR can re-expand, gated on a leader-epoch check (no diskless data read locally). - makeFollower: give a switched, at-seal, out-of-ISR follower a catch-up fetcher. - ReplicaFetcherThread: don't self-evict a switched partition until it's in ISR. Adds unit tests for the epoch-gated seal fetch and a URP-recovery system test.
giuseppelillo
force-pushed
the
glillo/resolve-urp-switch
branch
from
August 7, 2026 13:08
012e6d5 to
acdcb9e
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
Suppressed comments (3)
tests/kafkatest/tests/inkless/inkless_topic_switch_test.py:409
- _live_cluster_jmx_sum can return 0 even when one or more live brokers failed to report the metric (e.g., JMX read failed or no samples yet). That can make wait-for-zero checks pass prematurely and mask a real UnderReplicatedPartitions > 0 on an unsampled broker. Consider treating partial observations as a scrape miss and returning None unless every live broker produced a value for the requested gauge.
if time_to_stats:
latest = max(time_to_stats.keys())
total += time_to_stats[latest].get(key, 0)
observed = True
return int(total) if observed else None
core/src/test/scala/unit/kafka/server/ReplicaManagerInklessTest.scala:7169
- The comment describes the follower's leader epoch as "STALE (ahead-of-leader)", but an epoch greater than the leader's is not stale; it's mismatched/ahead-of-leader. Tightening the wording makes the test intent clearer.
// Follower fetches at the seal, but carries a STALE (ahead-of-leader) leader epoch. This mirrors
// the classic read path, which validates the request epoch before touching follower state.
core/src/main/scala/kafka/server/ReplicaFetcherThread.scala:174
- The eviction comment says the fetcher stops only once the replica is in ISR, but the condition also allows eviction for consolidating topics even if not in ISR. Updating the comment avoids a mismatch between documentation and behavior.
// Stop fetching after the switch from classic to diskless is completed: once the controller
// has committed a classicToDisklessStartOffset for this partition, our local LEO has reached it,
// and this replica is in ISR, the follower is fully caught up to the leader's frozen classic log
// and must not keep fetching.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A follower that dropped out of ISR and recovered after a classic-to-diskless switch stayed under-replicated forever, keeping UnderReplicatedPartitions stuck.
Adds unit tests for the epoch-gated seal fetch and a URP-recovery system test.