Skip to content

Migration guide from multi-cluster #51316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ruchikajha95 wants to merge 2 commits into
base: main
Choose a base branch
from
+332 −0
Draft

Migration guide from multi-cluster #51316

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4b2c9e4
Migration guide from multi-cluster
ruchikajha95 Jul 30, 2026
69b7f6d
Fixed the copilot comments
ruchikajha95 Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/guides/high-availability/multi-cluster-v2/introduction.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ Additional performance tuning and security hardening are still recommended when
== Operational procedures

* <@links.ha id="multi-cluster-v2-upgrades" />
* <@links.ha id="multi-cluster-v2-migrate-from-v1-to-v2" />

</@profile.ifCommunity>

Expand Down
330 changes: 330 additions & 0 deletions docs/guides/high-availability/multi-cluster-v2/migrate-from-v1-to-v2.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,330 @@
<#import "/templates/guide.adoc" as tmpl>
<#import "/templates/links.adoc" as links>

<@tmpl.guide
title="Migrating from multi-cluster v1 to v2"
summary="Migrate an existing multi-cluster v1 deployment to v2 (stateless mode)."
tileVisible="false"
preview="true"
previewDiscussionLink="https://github.com/keycloak/keycloak/discussions/50640" >

This {section} describes how to migrate an existing <@links.ha id="multi-cluster-introduction" /> deployment to <@links.ha id="multi-cluster-v2-introduction" />.

Multi-cluster v2 uses the `stateless` feature to store volatile data in the database instead of an external {jdgserver_name} cluster.
This simplifies the deployment architecture by removing {jdgserver_name}, its cross-site replication, the fencing automation, and {jdgserver_name}-specific monitoring.

== Overview of changes

The migration involves the following changes:

.Components removed
[%autowidth]
|===
|Component |Reason

|External {jdgserver_name} cluster
|Volatile data (authentication sessions, action tokens, login failure counters) is stored in the database instead.

|{jdgserver_name} cross-site replication
|No inter-site cache replication is needed; the database handles all cross-site data.

|{jdgserver_name} credentials and TLS secrets
|No longer required without the external {jdgserver_name} cluster.

|Fencing automation (for example, AWS Lambda)
|Split-brain detection and fencing for {jdgserver_name} are no longer necessary.

|{jdgserver_name}-specific monitoring and alerts
|Prometheus alerts for {jdgserver_name} cache health and cross-site connectivity are no longer needed.
|===

.Components retained
[%autowidth]
|===
|Component |Notes

|Database
|The existing synchronously replicated database is retained. Plan for approximately double the CPU usage and write IOPS compared to v1.

|Load balancer
|The external load balancer is retained.
Remove any health checks that target {jdgserver_name} endpoints.
The load balancer should probe the {project_name} `/lb-check` endpoint.
|===

.Configuration changes to {project_name}
[%autowidth]
|===
|Change |Details

|Enable the `stateless` feature
|Replaces the `multi-site` feature.

|Set a unique cluster name per site
|Required for cross-cluster cache invalidation via the database outbox.

|Increase the database connection pool size
|Volatile data is now stored in the database, increasing load.

|Remove external {jdgserver_name} connection options
|Options such as `cache-remote-host`, `cache-remote-port`, `cache-remote-username`, and `cache-remote-password` are no longer used.
|===

== Prerequisites

* A running multi-cluster v1 deployment as described in the <@links.ha id="multi-cluster-introduction" /> {section}.
* Familiarity with the v2 architecture, its requirements, and limitations as described in the <@links.ha id="multi-cluster-v2-introduction" /> {section} and the <@links.ha id="multi-cluster-v2-concepts" /> {section}.
* A database sized for the increased load.
With v2, database CPU usage and write IOPS can roughly double.
Review and adjust the database instance size, storage IOPS, and connection pool sizing accordingly.
See the <@links.ha id="multi-cluster-concepts-database-connections" /> {section} for details.
* Network latency between database nodes that meets the requirements.
+
include::../partials/network-latency.adoc[]

== Impact on volatile data

Since user sessions are already stored in the database in multi-cluster v1, users remain logged in after migrating to v2.

The following volatile data is lost during the migration:

In-progress authentication sessions::
Users who are mid-login need to restart their login flow.

Login failure counters::
Brute-force detection counters are reset.
Any ongoing lockouts are cleared.

Action tokens in the {jdgserver_name} cache::
Email verification links, password reset tokens, and OAuth codes that were stored in {jdgserver_name} are lost.
Users with pending email verifications or password resets need to request new links.

[IMPORTANT]
====
To minimize user impact, perform the migration during a maintenance window with low traffic.
====

== Migrating on {kubernetes}

This procedure assumes the {project_name} Operator is installed in each {kubernetes} cluster.

Perform the following steps in each {kubernetes} cluster, unless stated otherwise:

. Scale down {project_name} in all sites.
+
Scaling down ensures a clean transition.
No requests should be served during the migration.
+
[source,bash]
----
kubectl --namespace keycloak scale keycloak keycloak --replicas=0
----
+
Wait until all {project_name} Pods are terminated in all sites before proceeding.

. Prepare the updated {project_name} CR.
+
The following changes are required:
+
--
.. Replace the `multi-site` feature with `stateless`:
+
[source,yaml]
----
features:
enabled:
- stateless # <1>
----
<1> Replaces the `multi-site` feature from v1.
+
NOTE: If you are using an optimized custom image, the `stateless` feature is a build-time option.
You need to rebuild the image with `--features=stateless` included in the build command.
See <@links.operator id="customizing-keycloak" /> for details.

.. Add a unique cluster name for each site:
+
[source,yaml]
----
additionalOptions:
- name: spi-cache-embedded--default--cluster-name
value: CLUSTER1 # <1>
----
<1> Use a distinct name for each site (for example, `CLUSTER1` and `CLUSTER2`). No two sites may share the same name.

.. Remove the external {jdgserver_name} connection options `cache-remote-host`, `cache-remote-port`, `cache-remote-username`, and `cache-remote-password` from `additionalOptions`.

.. Increase the database connection pool size to handle the additional load:
+
[source,yaml]
----
db:
poolMinSize: 30 # <1>
poolInitialSize: 30
poolMaxSize: 30
----
<1> Adjust this value based on your workload.
With the `stateless` feature, authentication sessions and action tokens are stored in the database, which increases database load.
See the <@links.ha id="multi-cluster-concepts-database-connections" /> {section} for details.
--
+
See the <@links.ha id="multi-cluster-v2-deploy-keycloak-kubernetes" /> {section} for a complete example of the v2 {project_name} CR.

. Remove the {jdgserver_name} deployment.
+
--
.. Delete the {jdgserver_name} `Cache` CRs:
+
[source,bash]
----
kubectl --namespace keycloak delete cache actiontokens authenticationssessions loginfailures work
----

.. Delete the `Infinispan` CR:
+
[source,bash]
----
kubectl --namespace keycloak delete infinispan infinispan
----

.. Remove the {jdgserver_name} credentials and cross-site secrets:
+
[source,bash]
----
kubectl --namespace keycloak delete secret remote-store-secret
kubectl --namespace keycloak delete secret ispn-xsite-sa-token
kubectl --namespace keycloak delete secret xsite-keystore-secret
kubectl --namespace keycloak delete secret xsite-truststore-secret
----
+
Adjust the secret names to match your deployment.

.. Optionally, uninstall the {jdgserver_name} Operator if it is no longer needed for other purposes.
--

. Remove the fencing automation.
+
If using the AWS Lambda-based fencing as described in the <@links.ha id="multi-cluster-deploy-aws-accelerator-fencing-lambda" /> {section}:
+
--
.. Delete the AWS Lambda function.
.. Remove the Prometheus `AlertmanagerConfig` and `PrometheusRule` CRs that triggered the Lambda.
.. Remove the SNS topic and any related IAM roles.
--
+
Adjust these steps based on your fencing implementation.

. Remove {jdgserver_name}-specific monitoring.
+
Remove any Prometheus `ServiceMonitor` or `PodMonitor` resources that target the {jdgserver_name} deployment, as well as any Grafana dashboards for {jdgserver_name} cache metrics.

. Deploy the updated {project_name} CR to all sites:
+
[source,bash]
----
kubectl --namespace keycloak apply -f keycloak.yaml
----

. Verify the deployment.
+
Confirm that the {project_name} deployment is ready:
+
[source,bash]
----
kubectl wait --for=condition=Ready keycloaks.k8s.keycloak.org/keycloak
kubectl wait --for=condition=RollingUpdate=False keycloaks.k8s.keycloak.org/keycloak
----
+
Afterward, verify that the external load balancer routes traffic to all sites.

== Migrating on bare metal or virtual machines

This procedure assumes an existing multi-cluster v1 deployment with an external {jdgserver_name} cluster.

Perform the following steps on each {project_name} node at each site, unless stated otherwise:

. Stop {project_name} on all nodes in all sites.
+
Ensure that no {project_name} instances are running before proceeding.

. Update the {project_name} startup command.
+
The following changes are required:
+
--
.. Add `--features=stateless` to enable the stateless feature.
.. Add `--spi-cache-embedded--default--cluster-name=<CLUSTER_NAME>` with a unique name for each site.
.. Remove any `--cache-remote-*` options (`--cache-remote-host`, `--cache-remote-port`, `--cache-remote-username`, `--cache-remote-password`).
.. Remove `--features=multi-site` if it was previously set.
.. Increase the database connection pool size (`--db-pool-initial-size`, `--db-pool-min-size`, `--db-pool-max-size`).
--
+
The updated command should resemble:
+
[source,bash]
----
bin/kc.sh start \
--db=postgres \
--db-url=jdbc:postgresql://<DATABASE_URL>:5432/keycloak \
--db-pool-initial-size=30 \
--db-pool-min-size=30 \
--db-pool-max-size=30 \
--features=stateless \
--health-enabled=true \
--spi-cache-embedded--default--cluster-name=<CLUSTER_NAME> \
--hostname=<KEYCLOAK_HOSTNAME> \
--https-certificate-file=/path/to/tls.crt \
--https-certificate-key-file=/path/to/tls.key
----
+
See the <@links.ha id="multi-cluster-v2-deploy-keycloak-bare-metal" /> {section} for a complete example.

. Decommission the external {jdgserver_name} cluster.
+
Stop all {jdgserver_name} nodes and remove the {jdgserver_name} installation.
Remove any cross-site replication configuration, TLS certificates, and service accounts that were used for the {jdgserver_name} deployment.

. Remove fencing automation and {jdgserver_name} monitoring.
+
Remove any scripts, cron jobs, or monitoring integrations that handled {jdgserver_name}-related split-brain detection, failover, or health checks.

. Start {project_name} on all nodes in all sites.

. Verify the deployment.
+
Confirm that each {project_name} node is ready by checking the health endpoint on the management port:
+
[source,bash]
----
curl -sk -w '\n%{http_code}\n' https://localhost:9000/health/ready
----
+
A healthy node returns HTTP 200 with an overall status of `UP`.
Repeat this check for each node in each site.
+
Verify that `/lb-check` per site returns `200 OK` with status `UP`:
+
[source,bash]
----
curl -sk -w '\n%{http_code}\n' https://<SITE_LB_HOSTNAME>/lb-check
----
+
Finally, verify that the external load balancer routes traffic to all sites.

== Changes to operational procedures

After migrating to v2, several operational procedures from v1 are no longer applicable:

{jdgserver_name} synchronization::
The procedure described in the <@links.ha id="multi-cluster-operate-synchronize" /> {section} is no longer needed, as there is no external {jdgserver_name} to synchronize.

Site offline/online for {jdgserver_name}::
The {jdgserver_name}-specific steps in the <@links.ha id="multi-cluster-operate-site-offline" /> and <@links.ha id="multi-cluster-operate-site-online" /> {sections} are replaced by load balancer configuration changes.
To take a site offline, remove it from the load balancer.
To bring it back online, add it to the load balancer.

{jdgserver_name} health checks::
The {jdgserver_name} health checks described in the <@links.ha id="multi-cluster-health-checks" /> {section} are no longer applicable.

For upgrade procedures, see the <@links.ha id="multi-cluster-v2-upgrades" /> {section}.

</@tmpl.guide>
1 change: 1 addition & 0 deletions docs/guides/high-availability/pinned-guides
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,4 @@ multi-cluster-v2/concepts
multi-cluster-v2/deploy-keycloak-kubernetes
multi-cluster-v2/deploy-keycloak-bare-metal
multi-cluster-v2/upgrades
multi-cluster-v2/migrate-from-v1-to-v2
Loading