From 4b2c9e48682711d16fc7dd7fccb85968f73a12fa Mon Sep 17 00:00:00 2001 From: Ruchika Date: Thu, 30 Jul 2026 17:05:56 +0100 Subject: [PATCH 1/3] Migration guide from multi-cluster Closes #50866 Signed-off-by: Ruchika --- .../multi-cluster-v2/introduction.adoc | 1 + .../migrate-from-v1-to-v2.adoc | 325 ++++++++++++++++++ docs/guides/high-availability/pinned-guides | 1 + 3 files changed, 327 insertions(+) create mode 100644 docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc diff --git a/docs/guides/high-availability/multi-cluster-v2/introduction.adoc b/docs/guides/high-availability/multi-cluster-v2/introduction.adoc index 40b5c84c1f43..7ab1b89c37bd 100644 --- a/docs/guides/high-availability/multi-cluster-v2/introduction.adoc +++ b/docs/guides/high-availability/multi-cluster-v2/introduction.adoc @@ -99,6 +99,7 @@ Additional performance tuning and security hardening are still recommended when == Operational procedures * <@links.ha id="multi-cluster-v2-upgrades" /> +* <@links.ha id="multi-cluster-v2-migrate-from-v1-to-v2" /> diff --git a/docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc b/docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc new file mode 100644 index 000000000000..100fd9b8ce5b --- /dev/null +++ b/docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc @@ -0,0 +1,325 @@ +<#import "/templates/guide.adoc" as tmpl> +<#import "/templates/links.adoc" as links> + +<@tmpl.guide +title="Migrating from multi-cluster v1 to v2" +summary="Migrate an existing multi-cluster v1 deployment to v2 (stateless mode)." +tileVisible="false" +preview="true" +previewDiscussionLink="https://github.com/keycloak/keycloak/discussions/50640" > + +This {section} describes how to migrate an existing <@links.ha id="multi-cluster-introduction" /> deployment to <@links.ha id="multi-cluster-v2-introduction" />. + +Multi-cluster v2 uses the `stateless` feature to store volatile data in the database instead of an external {jdgserver_name} cluster. +This simplifies the deployment architecture by removing {jdgserver_name}, its cross-site replication, the fencing automation, and {jdgserver_name}-specific monitoring. + +== Overview of changes + +The migration involves the following changes: + +.Components removed +[%autowidth] +|=== +|Component |Reason + +|External {jdgserver_name} cluster +|Volatile data (authentication sessions, action tokens, login failure counters) is stored in the database instead. + +|{jdgserver_name} cross-site replication +|No inter-site cache replication is needed; the database handles all cross-site data. + +|{jdgserver_name} credentials and TLS secrets +|No longer required without the external {jdgserver_name} cluster. + +|Fencing automation (for example, AWS Lambda) +|Split-brain detection and fencing for {jdgserver_name} are no longer necessary. + +|{jdgserver_name}-specific monitoring and alerts +|Prometheus alerts for {jdgserver_name} cache health and cross-site connectivity are no longer needed. +|=== + +.Components retained +[%autowidth] +|=== +|Component |Notes + +|Database +|The existing synchronously replicated database is retained. Plan for approximately double the CPU usage and write IOPS compared to v1. + +|Load balancer +|The external load balancer is retained. +Remove any health checks that target {jdgserver_name} endpoints. +The load balancer should probe the {project_name} `/lb-check` endpoint. +|=== + +.Configuration changes to {project_name} +[%autowidth] +|=== +|Change |Details + +|Enable the `stateless` feature +|Replaces the `multi-site` feature. + +|Set a unique cluster name per site +|Required for cross-cluster cache invalidation via the database outbox. + +|Increase the database connection pool size +|Volatile data is now stored in the database, increasing load. + +|Remove external {jdgserver_name} connection options +|Options such as `cache-remote-host`, `cache-remote-port`, `cache-remote-username`, and `cache-remote-password` are no longer used. +|=== + +== Prerequisites + +* A running multi-cluster v1 deployment as described in the <@links.ha id="multi-cluster-introduction" /> {section}. +* Familiarity with the v2 architecture, its requirements, and limitations as described in the <@links.ha id="multi-cluster-v2-introduction" /> {section} and the <@links.ha id="multi-cluster-v2-concepts" /> {section}. +* A database sized for the increased load. +With v2, database CPU usage and write IOPS can roughly double. +Review and adjust the database instance size, storage IOPS, and connection pool sizing accordingly. +See the <@links.ha id="multi-cluster-concepts-database-connections" /> {section} for details. +* Network latency between database nodes that meets the requirements. ++ +include::../partials/network-latency.adoc[] + +== Impact on volatile data + +Since user sessions are already stored in the database in multi-cluster v1, users remain logged in after migrating to v2. + +The following volatile data is lost during the migration: + +In-progress authentication sessions:: +Users who are mid-login need to restart their login flow. + +Login failure counters:: +Brute-force detection counters are reset. +Any ongoing lockouts are cleared. + +Action tokens in the {jdgserver_name} cache:: +Email verification links, password reset tokens, and OAuth codes that were stored in {jdgserver_name} are lost. +Users with pending email verifications or password resets need to request new links. + +[IMPORTANT] +==== +To minimize user impact, perform the migration during a maintenance window with low traffic. +==== + +== Migrating on {kubernetes} + +This procedure assumes the {project_name} Operator is installed in each {kubernetes} cluster. + +Perform the following steps in each {kubernetes} cluster, unless stated otherwise: + +. Scale down {project_name} in all sites. ++ +Scaling down ensures a clean transition. +No requests should be served during the migration. ++ +[source,bash] +---- +kubectl --namespace keycloak scale keycloak keycloak --replicas=0 +---- ++ +Wait until all {project_name} Pods are terminated in all sites before proceeding. + +. Prepare the updated {project_name} CR. ++ +The following changes are required: ++ +-- +.. Replace the `multi-site` feature with `stateless`: ++ +[source,yaml] +---- + features: + enabled: + - stateless # <1> +---- +<1> Replaces the `multi-site` feature from v1. + +.. Add a unique cluster name for each site: ++ +[source,yaml] +---- + additionalOptions: + - name: spi-cache-embedded--default--cluster-name + value: CLUSTER1 # <1> +---- +<1> Use a distinct name for each site (for example, `CLUSTER1` and `CLUSTER2`). No two sites may share the same name. + +.. Remove the external {jdgserver_name} connection options `cache-remote-host`, `cache-remote-port`, `cache-remote-username`, and `cache-remote-password` from `additionalOptions`. + +.. Increase the database connection pool size to handle the additional load: ++ +[source,yaml] +---- + db: + poolMinSize: 30 # <1> + poolInitialSize: 30 + poolMaxSize: 30 +---- +<1> Adjust this value based on your workload. +With the `stateless` feature, authentication sessions and action tokens are stored in the database, which increases database load. +See the <@links.ha id="multi-cluster-concepts-database-connections" /> {section} for details. +-- ++ +See the <@links.ha id="multi-cluster-v2-deploy-keycloak-kubernetes" /> {section} for a complete example of the v2 {project_name} CR. + +. Remove the {jdgserver_name} deployment. ++ +-- +.. Delete the {jdgserver_name} `Cache` CRs: ++ +[source,bash] +---- +kubectl --namespace keycloak delete cache actiontokens authenticationssessions loginfailures work +---- + +.. Delete the `Infinispan` CR: ++ +[source,bash] +---- +kubectl --namespace keycloak delete infinispan infinispan +---- + +.. Remove the {jdgserver_name} credentials and cross-site secrets: ++ +[source,bash] +---- +kubectl --namespace keycloak delete secret remote-store-secret +kubectl --namespace keycloak delete secret ispn-xsite-sa-token +kubectl --namespace keycloak delete secret xsite-keystore-secret +kubectl --namespace keycloak delete secret xsite-truststore-secret +---- ++ +Adjust the secret names to match your deployment. + +.. Optionally, uninstall the {jdgserver_name} Operator if it is no longer needed for other purposes. +-- + +. Remove the fencing automation. ++ +If using the AWS Lambda-based fencing as described in the <@links.ha id="multi-cluster-deploy-aws-accelerator-fencing-lambda" /> {section}: ++ +-- +.. Delete the AWS Lambda function. +.. Remove the Prometheus `AlertmanagerConfig` and `PrometheusRule` CRs that triggered the Lambda. +.. Remove the SNS topic and any related IAM roles. +-- ++ +Adjust these steps based on your fencing implementation. + +. Remove {jdgserver_name}-specific monitoring. ++ +Remove any Prometheus `ServiceMonitor` or `PodMonitor` resources that target the {jdgserver_name} deployment, as well as any Grafana dashboards for {jdgserver_name} cache metrics. + +. Deploy the updated {project_name} CR to all sites: ++ +[source,bash] +---- +kubectl --namespace keycloak apply -f keycloak.yaml +---- + +. Verify the deployment. ++ +Confirm that the {project_name} deployment is ready: ++ +[source,bash] +---- +kubectl wait --for=condition=Ready keycloaks.k8s.keycloak.org/keycloak +kubectl wait --for=condition=RollingUpdate=False keycloaks.k8s.keycloak.org/keycloak +---- ++ +Afterward, verify that the external load balancer routes traffic to all sites. + +== Migrating on bare metal or virtual machines + +This procedure assumes an existing multi-cluster v1 deployment with an external {jdgserver_name} cluster. + +Perform the following steps on each {project_name} node at each site, unless stated otherwise: + +. Stop {project_name} on all nodes in all sites. ++ +Ensure that no {project_name} instances are running before proceeding. + +. Update the {project_name} startup command. ++ +The following changes are required: ++ +-- +.. Add `--features=stateless` to enable the stateless feature. +.. Add `--spi-cache-embedded--default--cluster-name=` with a unique name for each site. +.. Remove any `--cache-remote-*` options (`--cache-remote-host`, `--cache-remote-port`, `--cache-remote-username`, `--cache-remote-password`). +.. Remove `--features=multi-site` if it was previously set. +.. Increase the database connection pool size (`--db-pool-initial-size`, `--db-pool-min-size`, `--db-pool-max-size`). +-- ++ +The updated command should resemble: ++ +[source,bash] +---- +bin/kc.sh start \ + --db=postgres \ + --db-url=jdbc:postgresql://:5432/keycloak \ + --db-pool-initial-size=30 \ + --db-pool-min-size=30 \ + --db-pool-max-size=30 \ + --features=stateless \ + --spi-cache-embedded--default--cluster-name= \ + --hostname= \ + --https-certificate-file=/path/to/tls.crt \ + --https-certificate-key-file=/path/to/tls.key +---- ++ +See the <@links.ha id="multi-cluster-v2-deploy-keycloak-bare-metal" /> {section} for a complete example. + +. Decommission the external {jdgserver_name} cluster. ++ +Stop all {jdgserver_name} nodes and remove the {jdgserver_name} installation. +Remove any cross-site replication configuration, TLS certificates, and service accounts that were used for the {jdgserver_name} deployment. + +. Remove fencing automation and {jdgserver_name} monitoring. ++ +Remove any scripts, cron jobs, or monitoring integrations that handled {jdgserver_name}-related split-brain detection, failover, or health checks. + +. Start {project_name} on all nodes in all sites. + +. Verify the deployment. ++ +Confirm that each {project_name} node is ready by checking the health endpoint on the management port: ++ +[source,bash] +---- +curl -sk -w '\n%{http_code}\n' https://localhost:9000/health/ready +---- ++ +A healthy node returns HTTP 200 with an overall status of `UP`. +Repeat this check for each node in each site. ++ +Verify that `/lb-check` per site returns `200 OK` with status `UP`: ++ +[source,bash] +---- +curl -sk -w '\n%{http_code}\n' https:///lb-check +---- ++ +Finally, verify that the external load balancer routes traffic to all sites. + +== Changes to operational procedures + +After migrating to v2, several operational procedures from v1 are no longer applicable: + +{jdgserver_name} synchronization:: +The procedure described in the <@links.ha id="multi-cluster-operate-synchronize" /> {section} is no longer needed, as there is no external {jdgserver_name} to synchronize. + +Site offline/online for {jdgserver_name}:: +The {jdgserver_name}-specific steps in the <@links.ha id="multi-cluster-operate-site-offline" /> and <@links.ha id="multi-cluster-operate-site-online" /> {sections} are replaced by load balancer configuration changes. +To take a site offline, remove it from the load balancer. +To bring it back online, add it to the load balancer. + +{jdgserver_name} health checks:: +The {jdgserver_name} health checks described in the <@links.ha id="multi-cluster-health-checks" /> {section} are no longer applicable. + +For upgrade procedures, see the <@links.ha id="multi-cluster-v2-upgrades" /> {section}. + + diff --git a/docs/guides/high-availability/pinned-guides b/docs/guides/high-availability/pinned-guides index 22dc63af880f..dbe2f940d441 100644 --- a/docs/guides/high-availability/pinned-guides +++ b/docs/guides/high-availability/pinned-guides @@ -34,3 +34,4 @@ multi-cluster-v2/concepts multi-cluster-v2/deploy-keycloak-kubernetes multi-cluster-v2/deploy-keycloak-bare-metal multi-cluster-v2/upgrades +multi-cluster-v2/migrate-from-v1-to-v2 From 69b7f6d5476c9a958aa97d813df545f371a0e93d Mon Sep 17 00:00:00 2001 From: Ruchika Date: Tue, 4 Aug 2026 13:50:11 +0100 Subject: [PATCH 2/3] Fixed the copilot comments Closes #50866 Signed-off-by: Ruchika --- .../multi-cluster-v2/migrate-from-v1-to-v2.adoc | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc b/docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc index 100fd9b8ce5b..40b797771941 100644 --- a/docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc +++ b/docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc @@ -136,6 +136,10 @@ The following changes are required: - stateless # <1> ---- <1> Replaces the `multi-site` feature from v1. ++ +NOTE: If you are using an optimized custom image, the `stateless` feature is a build-time option. +You need to rebuild the image with `--features=stateless` included in the build command. +See <@links.operator id="customizing-keycloak" /> for details. .. Add a unique cluster name for each site: + @@ -265,6 +269,7 @@ bin/kc.sh start \ --db-pool-min-size=30 \ --db-pool-max-size=30 \ --features=stateless \ + --health-enabled=true \ --spi-cache-embedded--default--cluster-name= \ --hostname= \ --https-certificate-file=/path/to/tls.crt \ From 9bbfb1b2e09d81572d693d4cbdf9cb5201fc51b1 Mon Sep 17 00:00:00 2001 From: Ruchika Date: Wed, 5 Aug 2026 13:24:03 +0100 Subject: [PATCH 3/3] Fixed the migration guide pr review comments Closes #50866 Signed-off-by: Ruchika --- .../migrate-from-v1-to-v2.adoc | 93 ++----------------- 1 file changed, 6 insertions(+), 87 deletions(-) diff --git a/docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc b/docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc index 40b797771941..f4c799f74a39 100644 --- a/docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc +++ b/docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc @@ -23,7 +23,7 @@ The migration involves the following changes: |Component |Reason |External {jdgserver_name} cluster -|Volatile data (authentication sessions, action tokens, login failure counters) is stored in the database instead. +|Authentication sessions, action tokens and login failure counters are stored in the database instead. |{jdgserver_name} cross-site replication |No inter-site cache replication is needed; the database handles all cross-site data. @@ -64,7 +64,7 @@ The load balancer should probe the {project_name} `/lb-check` endpoint. |Required for cross-cluster cache invalidation via the database outbox. |Increase the database connection pool size -|Volatile data is now stored in the database, increasing load. +|Some data is now stored in database instead of Infinispan. |Remove external {jdgserver_name} connection options |Options such as `cache-remote-host`, `cache-remote-port`, `cache-remote-username`, and `cache-remote-password` are no longer used. @@ -93,7 +93,7 @@ Users who are mid-login need to restart their login flow. Login failure counters:: Brute-force detection counters are reset. -Any ongoing lockouts are cleared. +Any ongoing temporary lockouts are cleared. Action tokens in the {jdgserver_name} cache:: Email verification links, password reset tokens, and OAuth codes that were stored in {jdgserver_name} are lost. @@ -137,8 +137,8 @@ The following changes are required: ---- <1> Replaces the `multi-site` feature from v1. + -NOTE: If you are using an optimized custom image, the `stateless` feature is a build-time option. -You need to rebuild the image with `--features=stateless` included in the build command. +IMPORTANT: If you are using an optimized custom image, the `stateless` feature is a build-time option and cannot be enabled through the CR alone. +You must rebuild the image with `--features=stateless` included in the build command before applying this CR. See <@links.operator id="customizing-keycloak" /> for details. .. Add a unique cluster name for each site: @@ -226,90 +226,9 @@ kubectl --namespace keycloak apply -f keycloak.yaml . Verify the deployment. + -Confirm that the {project_name} deployment is ready: -+ -[source,bash] ----- -kubectl wait --for=condition=Ready keycloaks.k8s.keycloak.org/keycloak -kubectl wait --for=condition=RollingUpdate=False keycloaks.k8s.keycloak.org/keycloak ----- -+ +Follow the verification steps in the <@links.ha id="multi-cluster-v2-deploy-keycloak-kubernetes" /> {section}. Afterward, verify that the external load balancer routes traffic to all sites. -== Migrating on bare metal or virtual machines - -This procedure assumes an existing multi-cluster v1 deployment with an external {jdgserver_name} cluster. - -Perform the following steps on each {project_name} node at each site, unless stated otherwise: - -. Stop {project_name} on all nodes in all sites. -+ -Ensure that no {project_name} instances are running before proceeding. - -. Update the {project_name} startup command. -+ -The following changes are required: -+ --- -.. Add `--features=stateless` to enable the stateless feature. -.. Add `--spi-cache-embedded--default--cluster-name=` with a unique name for each site. -.. Remove any `--cache-remote-*` options (`--cache-remote-host`, `--cache-remote-port`, `--cache-remote-username`, `--cache-remote-password`). -.. Remove `--features=multi-site` if it was previously set. -.. Increase the database connection pool size (`--db-pool-initial-size`, `--db-pool-min-size`, `--db-pool-max-size`). --- -+ -The updated command should resemble: -+ -[source,bash] ----- -bin/kc.sh start \ - --db=postgres \ - --db-url=jdbc:postgresql://:5432/keycloak \ - --db-pool-initial-size=30 \ - --db-pool-min-size=30 \ - --db-pool-max-size=30 \ - --features=stateless \ - --health-enabled=true \ - --spi-cache-embedded--default--cluster-name= \ - --hostname= \ - --https-certificate-file=/path/to/tls.crt \ - --https-certificate-key-file=/path/to/tls.key ----- -+ -See the <@links.ha id="multi-cluster-v2-deploy-keycloak-bare-metal" /> {section} for a complete example. - -. Decommission the external {jdgserver_name} cluster. -+ -Stop all {jdgserver_name} nodes and remove the {jdgserver_name} installation. -Remove any cross-site replication configuration, TLS certificates, and service accounts that were used for the {jdgserver_name} deployment. - -. Remove fencing automation and {jdgserver_name} monitoring. -+ -Remove any scripts, cron jobs, or monitoring integrations that handled {jdgserver_name}-related split-brain detection, failover, or health checks. - -. Start {project_name} on all nodes in all sites. - -. Verify the deployment. -+ -Confirm that each {project_name} node is ready by checking the health endpoint on the management port: -+ -[source,bash] ----- -curl -sk -w '\n%{http_code}\n' https://localhost:9000/health/ready ----- -+ -A healthy node returns HTTP 200 with an overall status of `UP`. -Repeat this check for each node in each site. -+ -Verify that `/lb-check` per site returns `200 OK` with status `UP`: -+ -[source,bash] ----- -curl -sk -w '\n%{http_code}\n' https:///lb-check ----- -+ -Finally, verify that the external load balancer routes traffic to all sites. - == Changes to operational procedures After migrating to v2, several operational procedures from v1 are no longer applicable: