Skip to content

Ignore a left coordinator in the JDBC_PING2 health check until it writes again - #55248

Draft
ahus1 wants to merge 2 commits into
keycloak:mainfrom
ahus1:is-55236-jdbc-ping-remove-left-coordinator
Draft

ahus1 wants to merge 2 commits into
keycloak:mainfrom
ahus1:is-55236-jdbc-ping-remove-left-coordinator

Conversation

@ahus1

@ahus1 ahus1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Closes #55236

When the coordinator of a small cluster dies, its row stays in the table with coord = true until it is stale, and with the lower address it wins the split-brain decision against the survivor. Ignoring that row lets the survivor stay ready.

Decisions and assumptions

  • Each node remembers when a member left its view. The health check ignores the coord = true row of such a member until the row has been updated after it left. Only a member itself writes its own coordinator row, so an updated row shows that it is alive again, and the usual tie-break applies (the older node with the lower address wins).
  • The row is ignored instead of deleted. Deleting it from the other side races with a live member rewriting its own row, which can leave a live coordinator without a row for up to one info-writer interval and invert the tie-break.
  • The check compares the row's last_update with the local clock, as the staleness check already does. It relies on synchronized clocks, and the resolution is one second: a live member that rewrites its row within the same second as it left is counted again with its next write.
  • Remembered entries older than the staleness timeout are dropped whenever members leave, as the rows would be ignored as stale anyway.
  • A real partition has a short window where the other side's row is ignored and both sides may look healthy, until that side writes again. Split-brain detection is polling-based anyway, and a crash is more common than a partition.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Deleting a live former coordinator’s row can temporarily select the smaller partition as the winner, requiring prompt republication and a live-partition regression test rather than reliance on periodic refresh.

1 open issue
What changed in this PR

This PR addresses #55236 by removing a departed coordinator’s JDBC_PING2 row, allowing surviving nodes to recover readiness without waiting for that row to become stale.

Changes:

  • Deletes the previous coordinator’s row when the local node is coordinator and the previous coordinator is absent from the new view.
  • Adds a regression test verifying removal while preserving an unrelated coordinator’s row.
  • Adds a separate design draft for consolidating database-backed session handling.
File Description
model/​infinispan/​src/​test/​java/​org/​keycloak/​jgroups/​protocol/​JdbcPing2Test.java Tests departed-coordinator cleanup and preservation of unrelated rows.
model/​infinispan/​src/​main/​java/​org/​keycloak/​jgroups/​protocol/​KEYCLOAK_JDBC_PING2.java Removes the departed coordinator’s database row during view changes.
model/​infinispan/​session-storage-refactoring.md Documents a separate session-storage proposal and its transaction, isolation, and validation requirements.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The proposal relies on unavailable reference documents and incorrectly treats embedded-session binary-storage isolation as an existing guarantee.

2 open issues
1 resolved since last review

Review effort: Balanced

Closes keycloak#55236

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ahus1
ahus1 force-pushed the is-55236-jdbc-ping-remove-left-coordinator branch from 5b3b57a to f6e31fa Compare November 16, 2026 20:13
@ahus1
ahus1 requested a balanced review from Copilot November 16, 2026 20:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the departed-coordinator database-row cleanup change.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@keycloak-github-bot keycloak-github-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot

Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.model.infinispan.EmbeddedInfinispanSplitBrainTest#testRevokedTokensReloadedOnMergeEvent

Keycloak CI - Store Model Tests

java.lang.AssertionError: Execution didn't complete: java.lang.RuntimeException: org.awaitility.core.ConditionTimeoutException: Condition with alias 'Persisted revocation was not restored after the view merge' didn't complete within 30 seconds because condition with Lambda expression in org.keycloak.testsuite.model.infinispan.EmbeddedInfinispanSplitBrainTest was not fulfilled.
	at org.junit.Assert.fail(Assert.java:89)
	at org.keycloak.testsuite.model.KeycloakModelTest.inIndependentFactories(KeycloakModelTest.java:448)
	at org.keycloak.testsuite.model.infinispan.EmbeddedInfinispanSplitBrainTest.testRevokedTokensReloadedOnMergeEvent(EmbeddedInfinispanSplitBrainTest.java:125)

Report flaky test

…again

Instead of deleting the row of the previous coordinator, ignore its row in the split-brain decision until the row is updated after it left the view.

Closes keycloak#55236

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ahus1 ahus1 changed the title Remove the previous coordinator from JDBC_PING2 on view change Ignore a left coordinator in the JDBC_PING2 health check until it writes again Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two-node cluster: killing the JDBC_PING2 coordinator makes the survivor report split-brain and stay unready for ~40–70 s

2 participants