You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Raise generated cache-transport RSA keys to 3072 bits and allow ECDSA - #55224
The generated cache-transport certificate follows BSI TR-02102-2, which asks for RSA keys of at least 3000 bits and deprecates PKCS#1 v1.5 signatures.
Decisions and assumptions
The default is RSA 3072 signed with RSASSA-PSS. Only the JGroups certificate opts into PSS; realm keys, client credentials and other callers of the shared certificate utilities keep PKCS#1 v1.5 to avoid breaking relying parties.
The SPI options algorithm (RSA default, or ECDSA), key-size (default 3072, minimum 2048) and curve (secp256r1 default, secp384r1, secp521r1) are added to the jgroups-mtls SPI. Invalid values fail at startup. The CLI options are expected to be removed in a future version, so this is meant to be backported to allow a graceful migration.
A node that starts with a different key than the stored certificate first joins the cluster with the stored certificate, which the running nodes trust. On its first view it replaces the certificate if it still mismatches the configured key, and sends the existing reload notification. The replacement is conditional on the mismatch still existing, so concurrent nodes replace it once. Nodes of older versions handle that notification, so a rolling upgrade does not split the cluster. A stored certificate that is expired or cannot be loaded is replaced at startup, as a node could not join with it. Until the join, the node keeps using the weaker key.
No additional certificate is trusted, so an outdated certificate stays unusable.
A running node still presents the old certificate if it misses the notification. A joining node rejects it before presenting its own certificate, so the running node only sees a failed handshake. The JDK asks the trust manager for the accepted issuers on every incoming handshake, so the node reloads its certificate from there, off the handshake thread, and drops reload requests while one is running. This costs one database lookup per incoming connection. Reading and applying a certificate happen under one lock, so a delayed reload cannot reinstall a superseded certificate.
The private key is now stored as PKCS#8, as the existing PEM utilities write EC keys as SEC1 and cannot read them back.
The Arquillian FIPS integration tests now run with clustered caches (--cache=ispn) so that they exercise certificate generation in FIPS mode; the Arquillian local mode is left in place and can be removed in a follow-up. The new-framework FIPS suites stay on local caches, as tests like LoginTest assume cookies without a cluster route.
The clustered ECDSA test only asserts that two nodes form a cluster
The migration is tested in the new test framework in tests/clustering against a two-node cluster. JGroupsCertificateRotationClusterTest is migrated from the Arquillian testsuite for this, and the Arquillian version is removed. To run code on a specific node, run-on-server now works with the clustered server: the containers load the test classes from the host via host.testcontainers.internal, configured through the test-class-url option of the run-on-server provider, and the load balancer selects the node. Stopping the coordinator needs its own cluster, so it is a separate test class. They run in the "Clustering IT" job through ClusterTestSuite, next to the Arquillian cluster tests that they will eventually replace. The tests skip on stateless clusters, as those nodes do not form one cluster.
The upgrade note is in changes-27_0_0.adoc; a backport needs to move it to the matching version file.
The mixed-version job only runs the org.keycloak.tests.compatibility package, selected through MixedClusterTestSuite; JdbcPingCustomSchemaTest therefore no longer runs there, only in the current-build run. Its old node is an older patch release that does not know the new key options, so the certificate tests must not run there.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Database-backed startup migration tests are missing for larger, smaller, and unchanged keys and concurrent startups, leaving committed-certificate convergence and alias preservation unverified.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The rotation tests fail in stateless and mixed-version CI configurations, and their cleanup can leave shared-cluster certificates inconsistent with the server configuration.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Certificate replacement can block rolling joins without triggering existing nodes to reload, while migrated rotation and failover tests have no CI execution path.
When a joining node replaces the certificate, it trusts only the new certificate and rejects an existing node's old server certificate before presenting its own client certificate. The existing node receives a TLS alert without invoking its trust-failure reload, so retries can remain blocked until a separate reload event. Retain transitional trust or provide a reload mechanism that reaches existing nodes before the join, and add a live rolling-join test.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Synchronous TLS handshaking inside accept() occurs before a read timeout is configured, allowing a silent TCP client to block acceptance of all other cluster peers indefinitely.
If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.
java.lang.AssertionError: Execution didn't complete: java.lang.RuntimeException: org.awaitility.core.ConditionTimeoutException: Condition with alias 'Persisted revocation was not restored after the view merge' didn't complete within 30 seconds because condition with Lambda expression in org.keycloak.testsuite.model.infinispan.EmbeddedInfinispanSplitBrainTest was not fulfilled.
at org.junit.Assert.fail(Assert.java:89)
at org.keycloak.testsuite.model.KeycloakModelTest.inIndependentFactories(KeycloakModelTest.java:448)
at org.keycloak.testsuite.model.infinispan.EmbeddedInfinispanSplitBrainTest.testRevokedTokensReloadedOnMergeEvent(EmbeddedInfinispanSplitBrainTest.java:125)
Adds the startup check, logs the active certificate, and adds docs and release notes.
Closeskeycloak#54496
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
…ration tests
Run the test on the new test framework by letting run-on-server reach the
containers of a clustered server, and cover the certificate replacement on
startup for larger, smaller and different keys, and concurrent startups.
Relates to keycloak#54496
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Relates to keycloak#54496
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Critical reload races can restore superseded certificates, while startup migration can leave rolling-upgrade joins blocked because older coordinators lack a compatible reload trigger.
A node that starts with a changed key replaces the stored certificate.
Running nodes still present the old one, and the connecting node aborts the handshake before it presents its own certificate.
The running node therefore never sees an untrusted certificate, but the JDK asks its trust manager for the accepted issuers on every incoming handshake.
Trigger a reload from there, outside of the handshake thread, and drop requests while a reload is running.
Relates to keycloak#54496
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A critical race allows an asynchronous reload to reinstall a stale certificate after rotation because database reads and installation are not serialized, potentially causing the node to reject peers.
A node that starts with a different key now joins with the stored certificate that the running nodes trust, and replaces it after its first view.
It then notifies the nodes to reload it, which nodes of older versions also handle, so a rolling upgrade does not split the cluster.
An expired or unusable stored certificate is still replaced at startup.
Relates to keycloak#54496
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Certificate handling bypasses validity and loadability checks, while exhausted migration retries can leave stable clusters using a mismatching key until normal rotation.
var stored = fromJson(storage.loadOrCreate(CERTIFICATE_ID, this::generateSelfSignedCertificate));
if (matchesKeySpec(stored)) {
return stored;
}
if (isUsable(stored)) {
// The running nodes trust only the stored certificate, so join with it and replace it afterwards in migrateCertificate().
logger.infof("Stored JGroups certificate does not match the configured key (%s). Replacing it once this node has joined the cluster.", keySpec.description());
migrationPending = true;
return stored;
}
// Replace only if still mismatching, so concurrently starting nodes with the same configuration regenerate it once.
// Running nodes reload the new certificate when the trust manager rejects the connection of this node.
logger.infof("Stored JGroups certificate does not match the configured key (%s) and is unusable. Regenerating it.", keySpec.description());
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #54496
The generated cache-transport certificate follows BSI TR-02102-2, which asks for RSA keys of at least 3000 bits and deprecates PKCS#1 v1.5 signatures.
Decisions and assumptions
algorithm(RSAdefault, orECDSA),key-size(default 3072, minimum 2048) andcurve(secp256r1default,secp384r1,secp521r1) are added to thejgroups-mtlsSPI. Invalid values fail at startup. The CLI options are expected to be removed in a future version, so this is meant to be backported to allow a graceful migration.--cache=ispn) so that they exercise certificate generation in FIPS mode; the Arquillianlocalmode is left in place and can be removed in a follow-up. The new-framework FIPS suites stay on local caches, as tests likeLoginTestassume cookies without a cluster route.tests/clusteringagainst a two-node cluster.JGroupsCertificateRotationClusterTestis migrated from the Arquillian testsuite for this, and the Arquillian version is removed. To run code on a specific node, run-on-server now works with the clustered server: the containers load the test classes from the host viahost.testcontainers.internal, configured through thetest-class-urloption of the run-on-server provider, and the load balancer selects the node. Stopping the coordinator needs its own cluster, so it is a separate test class. They run in the "Clustering IT" job throughClusterTestSuite, next to the Arquillian cluster tests that they will eventually replace. The tests skip on stateless clusters, as those nodes do not form one cluster.changes-27_0_0.adoc; a backport needs to move it to the matching version file.org.keycloak.tests.compatibilitypackage, selected throughMixedClusterTestSuite;JdbcPingCustomSchemaTesttherefore no longer runs there, only in the current-build run. Its old node is an older patch release that does not know the new key options, so the certificate tests must not run there.