Skip to content

Raise generated cache-transport RSA keys to 3072 bits and allow ECDSA - #55224

Draft
ahus1 wants to merge 9 commits into
keycloak:mainfrom
ahus1:is-54496-jgroups-rsa-key-size
Draft

ahus1 wants to merge 9 commits into
keycloak:mainfrom
ahus1:is-54496-jgroups-rsa-key-size

Conversation

@ahus1

@ahus1 ahus1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Closes #54496

The generated cache-transport certificate follows BSI TR-02102-2, which asks for RSA keys of at least 3000 bits and deprecates PKCS#1 v1.5 signatures.

Decisions and assumptions

  • The default is RSA 3072 signed with RSASSA-PSS. Only the JGroups certificate opts into PSS; realm keys, client credentials and other callers of the shared certificate utilities keep PKCS#1 v1.5 to avoid breaking relying parties.
  • The SPI options algorithm (RSA default, or ECDSA), key-size (default 3072, minimum 2048) and curve (secp256r1 default, secp384r1, secp521r1) are added to the jgroups-mtls SPI. Invalid values fail at startup. The CLI options are expected to be removed in a future version, so this is meant to be backported to allow a graceful migration.
  • A node that starts with a different key than the stored certificate first joins the cluster with the stored certificate, which the running nodes trust. On its first view it replaces the certificate if it still mismatches the configured key, and sends the existing reload notification. The replacement is conditional on the mismatch still existing, so concurrent nodes replace it once. Nodes of older versions handle that notification, so a rolling upgrade does not split the cluster. A stored certificate that is expired or cannot be loaded is replaced at startup, as a node could not join with it. Until the join, the node keeps using the weaker key.
  • No additional certificate is trusted, so an outdated certificate stays unusable.
  • A running node still presents the old certificate if it misses the notification. A joining node rejects it before presenting its own certificate, so the running node only sees a failed handshake. The JDK asks the trust manager for the accepted issuers on every incoming handshake, so the node reloads its certificate from there, off the handshake thread, and drops reload requests while one is running. This costs one database lookup per incoming connection. Reading and applying a certificate happen under one lock, so a delayed reload cannot reinstall a superseded certificate.
  • The private key is now stored as PKCS#8, as the existing PEM utilities write EC keys as SEC1 and cannot read them back.
  • The Arquillian FIPS integration tests now run with clustered caches (--cache=ispn) so that they exercise certificate generation in FIPS mode; the Arquillian local mode is left in place and can be removed in a follow-up. The new-framework FIPS suites stay on local caches, as tests like LoginTest assume cookies without a cluster route.
  • The clustered ECDSA test only asserts that two nodes form a cluster
  • The migration is tested in the new test framework in tests/clustering against a two-node cluster. JGroupsCertificateRotationClusterTest is migrated from the Arquillian testsuite for this, and the Arquillian version is removed. To run code on a specific node, run-on-server now works with the clustered server: the containers load the test classes from the host via host.testcontainers.internal, configured through the test-class-url option of the run-on-server provider, and the load balancer selects the node. Stopping the coordinator needs its own cluster, so it is a separate test class. They run in the "Clustering IT" job through ClusterTestSuite, next to the Arquillian cluster tests that they will eventually replace. The tests skip on stateless clusters, as those nodes do not form one cluster.
  • The upgrade note is in changes-27_0_0.adoc; a backport needs to move it to the matching version file.
  • The mixed-version job only runs the org.keycloak.tests.compatibility package, selected through MixedClusterTestSuite; JdbcPingCustomSchemaTest therefore no longer runs there, only in the current-build run. Its old node is an older patch release that does not know the new key options, so the certificate tests must not run there.

Closes keycloak#54496

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Closes keycloak#54496

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ahus1
ahus1 force-pushed the is-54496-jgroups-rsa-key-size branch 3 times, most recently from 98b4314 to 3966750 Compare November 16, 2026 14:34
@ahus1
ahus1 requested a balanced review from Copilot November 16, 2026 15:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Database-backed startup migration tests are missing for larger, smaller, and unchanged keys and concurrent startups, leaving committed-certificate convergence and alias preservation unverified.

1 open issue
What changed in this PR

Strengthens Keycloak’s automatically generated cache-transport certificates for #54496, without changing realm keys or user-supplied certificates.

Changes:

  • Defaults to RSA-3072 with RSASSA-PSS, adds RSA/ECDSA configuration, and replaces stored certificates on startup when their keys do not match.
  • Adds opt-in PSS signing to shared certificate utilities and stores transport private keys as PKCS#8.
  • Updates documentation, adds cryptographic and cluster-formation tests, and enables clustered caches in Arquillian FIPS tests.
File Description
testsuite/​integration-arquillian/​tests/​base/​src/​test/​java/​org/​keycloak/​testsuite/​adapter/​servlet/​SAMLServletAdapterTest.java Removes cluster routes before decoding session cookies.
testsuite/​integration-arquillian/​tests/​base/​src/​main/​java/​org/​keycloak/​testsuite/​arquillian/​containers/​AbstractQuarkusDeployableContainer.java Supports default clustered-cache configuration.
tests/​clustering/​src/​test/​java/​org/​keycloak/​tests/​clustering/​EcdsaCacheTransportCertificateTest.java Adds ECDSA cluster-formation coverage.
model/​infinispan/​src/​test/​java/​org/​keycloak/​jgroups/​certificates/​UtilsTest.java Tests generation, key matching, and legacy private-key loading.
model/​infinispan/​src/​main/​java/​org/​keycloak/​jgroups/​certificates/​Utils.java Generates configured transport keys and requests PSS signing.
model/​infinispan/​src/​main/​java/​org/​keycloak/​jgroups/​certificates/​KeySpec.java Defines RSA/ECDSA configuration, validation, and matching.
model/​infinispan/​src/​main/​java/​org/​keycloak/​jgroups/​certificates/​JGroupsCertificate.java Stores private keys as PKCS#8.
model/​infinispan/​src/​main/​java/​org/​keycloak/​jgroups/​certificates/​DefaultJGroupsCertificateProviderFactory.java Exposes and reads certificate provider options.
model/​infinispan/​src/​main/​java/​org/​keycloak/​jgroups/​certificates/​DatabaseJGroupsCertificateProvider.java Replaces mismatched stored certificates during startup.
model/​infinispan/​pom.xml Adds the default crypto provider for tests.
docs/​guides/​server/​caching.adoc Documents stronger defaults and configuration behavior.
docs/​documentation/​upgrading/​topics/​changes/​changes-27_0_0.adoc Explains migration, retries, and configuration options.
docs/​documentation/​release_notes/​topics/​27_0_0.adoc Announces stronger cache-transport certificates.
crypto/​fips1402/​src/​test/​java/​org/​keycloak/​crypto/​fips/​test/​BCFIPSCertificateUtilsProviderTest.java Tests PSS and ECDSA certificate signing.
crypto/​fips1402/​src/​main/​java/​org/​keycloak/​crypto/​fips/​BCFIPSCertificateUtilsProvider.java Adds opt-in PSS signing for the FIPS provider.
crypto/​elytron/​src/​main/​java/​org/​keycloak/​crypto/​elytron/​ElytronCertificateUtilsProvider.java Adds PSS signature selection for Elytron.
crypto/​default/​src/​test/​java/​org/​keycloak/​crypto/​def/​test/​BCCertificateUtilsProviderTest.java Tests PSS and preserved PKCS#1 signing behavior.
crypto/​default/​src/​main/​java/​org/​keycloak/​crypto/​def/​BCCertificateUtilsProvider.java Adds opt-in PSS signing for the default provider.
common/​src/​main/​java/​org/​keycloak/​common/​util/​CertificateUtils.java Exposes the optional PSS signing overload.
common/​src/​main/​java/​org/​keycloak/​common/​crypto/​CertificateUtilsProvider.java Extends the certificate-provider signing contract.
.github/​scripts/​run-fips-it.sh Enables clustered caches for Arquillian FIPS tests.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The rotation tests fail in stateless and mixed-version CI configurations, and their cleanup can leave shared-cluster certificates inconsistent with the server configuration.

3 open issues
1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Certificate replacement can block rolling joins without triggering existing nodes to reload, while migrated rotation and failover tests have no CI execution path.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

High severity Preserve transitional trust during rolling node joins

model/​infinispan/​src/​main/​java/​org/​keycloak/​jgroups/​certificates/​DatabaseJGroupsCertificateProvider.java:205

When a joining node replaces the certificate, it trusts only the new certificate and rejects an existing node's old server certificate before presenting its own client certificate. The existing node receives a TLS alert without invoking its trust-failure reload, so retries can remain blocked until a separate reload event. Retain transitional trust or provide a reload mechanism that reaches existing nodes before the join, and add a live rolling-join test.

3 resolved since last review

Review effort: Balanced

Comment thread .github/workflows/ci.yml Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Synchronous TLS handshaking inside accept() occurs before a read timeout is configured, allowing a silent TCP client to block acceptance of all other cluster peers indefinitely.

1 open issue
1 resolved since last review

Review effort: Balanced

@keycloak-github-bot keycloak-github-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot

Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.model.infinispan.EmbeddedInfinispanSplitBrainTest#testRevokedTokensReloadedOnMergeEvent

Keycloak CI - Store Model Tests

java.lang.AssertionError: Execution didn't complete: java.lang.RuntimeException: org.awaitility.core.ConditionTimeoutException: Condition with alias 'Persisted revocation was not restored after the view merge' didn't complete within 30 seconds because condition with Lambda expression in org.keycloak.testsuite.model.infinispan.EmbeddedInfinispanSplitBrainTest was not fulfilled.
	at org.junit.Assert.fail(Assert.java:89)
	at org.keycloak.testsuite.model.KeycloakModelTest.inIndependentFactories(KeycloakModelTest.java:448)
	at org.keycloak.testsuite.model.infinispan.EmbeddedInfinispanSplitBrainTest.testRevokedTokensReloadedOnMergeEvent(EmbeddedInfinispanSplitBrainTest.java:125)

Report flaky test

@ahus1
ahus1 force-pushed the is-54496-jgroups-rsa-key-size branch 2 times, most recently from 14bb83d to 10753bb Compare November 17, 2026 17:18
ahus1 and others added 5 commits November 17, 2026 18:48
Adds the startup check, logs the active certificate, and adds docs and release notes.

Closes keycloak#54496

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Closes keycloak#54496

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Closes keycloak#54496

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
…ration tests

Run the test on the new test framework by letting run-on-server reach the
containers of a clustered server, and cover the certificate replacement on
startup for larger, smaller and different keys, and concurrent startups.

Relates to keycloak#54496

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Relates to keycloak#54496

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ahus1
ahus1 force-pushed the is-54496-jgroups-rsa-key-size branch from 10753bb to d1b695e Compare November 17, 2026 17:49
@ahus1
ahus1 requested a balanced review from Copilot November 17, 2026 17:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical reload races can restore superseded certificates, while startup migration can leave rolling-upgrade joins blocked because older coordinators lack a compatible reload trigger.

3 open issues

Review effort: Balanced

A node that starts with a changed key replaces the stored certificate.
Running nodes still present the old one, and the connecting node aborts the handshake before it presents its own certificate.
The running node therefore never sees an untrusted certificate, but the JDK asks its trust manager for the accepted issuers on every incoming handshake.
Trigger a reload from there, outside of the handshake thread, and drop requests while a reload is running.

Relates to keycloak#54496

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A critical race allows an asynchronous reload to reinstall a stale certificate after rotation because database reads and installation are not serialized, potentially causing the node to reject peers.

3 open issues

Review effort: Balanced

A node that starts with a different key now joins with the stored certificate that the running nodes trust, and replaces it after its first view.
It then notifies the nodes to reload it, which nodes of older versions also handle, so a rolling upgrade does not split the cluster.
An expired or unusable stored certificate is still replaced at startup.

Relates to keycloak#54496

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ahus1
ahus1 force-pushed the is-54496-jgroups-rsa-key-size branch from d1b695e to c9945a4 Compare November 17, 2026 18:19
@ahus1
ahus1 requested a balanced review from Copilot November 17, 2026 18:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Certificate handling bypasses validity and loadability checks, while exhausted migration retries can leave stable clusters using a mismatching key until normal rotation.

2 open issues
3 resolved since last review

Review effort: Balanced

Comment on lines +140 to +151
private void migrateCertificate() {
lock.lock();
try (lock) {
if (KeycloakModelUtils.runJobInTransactionWithResult(sessionFactory, CertificateReloadManager::migrateCertificateInTransaction)) {
logger.info("Replaced the JGroups certificate to match the configured key");
sendReloadNotification();
}
} catch (RuntimeException e) {
// the migration stays pending and is retried on the next view
logger.warn("Failed to replace the JGroups certificate", e);
}
}
Comment on lines +255 to +273
var stored = fromJson(storage.loadOrCreate(CERTIFICATE_ID, this::generateSelfSignedCertificate));
if (matchesKeySpec(stored)) {
return stored;
}
if (isUsable(stored)) {
// The running nodes trust only the stored certificate, so join with it and replace it afterwards in migrateCertificate().
logger.infof("Stored JGroups certificate does not match the configured key (%s). Replacing it once this node has joined the cluster.", keySpec.description());
migrationPending = true;
return stored;
}
// Replace only if still mismatching, so concurrently starting nodes with the same configuration regenerate it once.
// Running nodes reload the new certificate when the trust manager rejects the connection of this node.
logger.infof("Stored JGroups certificate does not match the configured key (%s) and is unusable. Regenerating it.", keySpec.description());
storage.replace(CERTIFICATE_ID, json -> !matchesKeySpec(fromJson(json)), this::generateSelfSignedCertificate);
return fromJson(storage.find(CERTIFICATE_ID).orElseThrow());
}

private static boolean isUsable(JGroupsCertificate certificate) {
try {
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Raise generated cache-transport RSA keys to 3072 bits

2 participants