Skip to content

Fix flaky RealmInvalidationClusterTest by waiting for cluster cache sync - #55349

Merged
ahus1 merged 1 commit into
keycloak:mainfrom
ruchikajha95:fix-55346-cluster-cache-sync
Nov 19, 2026
Merged

ahus1 merged 1 commit into
keycloak:mainfrom
ruchikajha95:fix-55346-cluster-cache-sync

Conversation

@ruchikajha95

@ruchikajha95 ruchikajha95 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Closes #55346

After failback() brings killed nodes back online, the test immediately proceeds without waiting for Infinispan cluster rebalancing to complete, causing race conditions when cache invalidation events have not fully propagated during split-brain recovery.

Decisions and assumptions

  • Poll-based approach rather than fixed sleep allows early return once cluster is synchronized (typically < 1 second in healthy scenarios)
  • Check ALL backend nodes (not just survivors) to ensure the restarted node has rejoined the cluster and received cache invalidations before proceeding
  • 10-second timeout is consistent with similar cluster tests and allows time for split-brain recovery
  • 100ms polling interval balances responsiveness with avoiding excessive reads during rebalancing
  • Use TimeUnit.MILLISECONDS.sleep() to avoid adding external dependencies
  • Final assertion on timeout checks all nodes and provides detailed diagnostics showing which nodes are inconsistent and what fields differ

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The failback readiness check excludes the restarted node, so it can return while that node is still isolated and leave the next update exposed to the same race.

1 open issue
What changed in this PR

This PR aims to reduce the flaky realm invalidation failure reported in #55346 by adding a post-failback consistency wait to the shared cluster-test base.

Changes:

  • Adds entity-state polling with a 10-second timeout and 100ms interval.
  • Calls the wait after failback, before rotating the failure node.
  • Logs synchronization progress and performs a final consistency assertion on timeout.
File Description
testsuite/​integration-arquillian/​tests/​base/​src/​test/​java/​org/​keycloak/​testsuite/​cluster/​AbstractInvalidationClusterTest.java Adds post-failback consistency polling and timeout diagnostics.

Review effort: Balanced

Replace blind sleep after failback() with condition-based polling that
actively verifies cache consistency across all backend nodes before
proceeding. This prevents race conditions during split-brain recovery
where cache invalidation events have not fully propagated.

The new waitForClusterSync() method:
- Polls ALL backend nodes (not just survivors) for entity consistency
- Ensures the restarted node has rejoined and received invalidations
- Returns immediately once synchronized (typically < 1 second)
- Has a 10-second timeout with detailed failure diagnostics
- Uses TimeUnit.MILLISECONDS.sleep() for polling (no external deps)

This addresses the root cause of flaky failures where the test would
immediately check cache consistency after failback without waiting for
Infinispan cluster rebalancing to complete.

Closes keycloak#55346

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Ruchika <Ruchika.Jha1@ibm.com>
@ahus1
ahus1 marked this pull request as ready for review November 19, 2026 15:43
@ahus1
ahus1 requested review from a team as code owners November 19, 2026 15:43
Copilot AI balanced review requested due to automatic review settings November 19, 2026 15:43
@ahus1
ahus1 merged commit b65e433 into keycloak:main Nov 19, 2026
157 of 159 checks passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The recovery wait can return before cluster membership is restored because matching values may come from the shared database, and the initial failback path still updates entities without waiting, leaving invalidation races unresolved.

2 open issues
1 resolved since last review

Review effort: Balanced

Comment on lines +180 to +184
if (allNodesConsistent) {
long elapsed = System.currentTimeMillis() - startTime;
log.info(String.format("Cluster cache synchronized successfully across all %d nodes after %dms",
getClusterSize(), elapsed));
return;
Comment on lines +236 to +237
// Wait for cluster rebalancing to complete after failback
waitForClusterSync(testEntity);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test: org.keycloak.testsuite.cluster.RealmInvalidationClusterTest#crudWithFailover

3 participants