Hi @ruchikajha95 ,

I hope you are doing well.

Could you please clarify one point?

When we enable replication for the COMPONENT_CONFIG table (i.e., we no longer exclude it), we encounter the following replication error on both sides of the cross-data-center replication:

Last_Errno: 1032
Last_Error: Could not execute Delete_rows_v1 event on table moi_bc_keycloak.COMPONENT_CONFIG;
Can't find record in 'COMPONENT_CONFIG', Error_code: 1032;
handler error HA_ERR_KEY_NOT_FOUND;
the event's master log mysql-bin.017738, end_log_pos 0

Have you encountered this type of replication issue before? If so, could you advise what might be causing it and how it should be handled in a supported Keycloak HA deployment?

Thank you.

Hi @DimaShaar,

I set up a local cluster lab to reproduce this exact scenario end-to-end (simulating an active-active setup, excluding COMPONENT_CONFIG, generating data drift, re-enabling replication, and fixing the resulting error).

Here is a breakdown of why you are seeing Error 1032 (HA_ERR_KEY_NOT_FOUND) and how to safely resolve it:

Why Error 1032 is Happening
Data Drift: While COMPONENT_CONFIG was excluded from replication, Node A and Node B maintained independent versions of that table. Any new mappers, updates, or deletions executed on Node A were never written to Node B.

When you re-enabled replication for COMPONENT_CONFIG, Node A sent a DELETE (or update) binary log event over to Node B for a record that Node A created while replication was off.

The Crash: Because Node B never received the initial INSERT statement for that record, it tried to delete a key that doesn't exist in its local database causing the replication SQL thread to fail with Error 1032.

Here is what i did :

To bring replication back online safely without losing configuration data, you need to bring the secondary database back in sync with your primary source of truth:

1.Stop the failing replication SQL thread on the secondary node.

2.Resynchronize COMPONENT_CONFIG from the primary database:
Dump the COMPONENT_CONFIG table from your primary node (the source of truth) and import it directly into the secondary node to align the primary keys and missing rows.

3.Skip the orphaned event and restart replication:
Since the table data on the secondary node is now identical to the primary node, advance the replication skip counter past the failing event and resume replication.

4.Verify replication health:
Check that both threads are running.

Once replication is restored, keep COMPONENT_CONFIG (and all other relational tables) 100% replicated synchronously across all nodes. As outlined in the Keycloak Multi-Cluster High Availability documentation, Keycloak relies on a unified, consistent database state to prevent split-brain cache states and synchronization issues.

Hope this helps your team get replication back in sync!

Hi @ruchikajha95 ,

We are running into a recurring MariaDB Error 1032 (HA_ERR_KEY_NOT_FOUND) across our Data Centers and would appreciate your guidance on how Keycloak handles or avoids concurrent updates to COMPONENT_CONFIG.

Our Scenario & Observed Behavior

1. Data Centers almost immediately hits Error 1032 on COMPONENT_CONFIG due to a concurrent Delete_rows_v1 event triggered on both sides.
2. Within the same single cluster, everything works without issues. The error only surfaces during cross-DC replication/sync.
   3.Sometime Even when Keycloak is completely shut down / not running in DC2, replication still breaks on DC1. This happens because a DELETE statement previously executed or logged on DC2 gets replicated over to DC1, attempting to delete a record on DC1 that either no longer exists or was modified. ( is it possible that LDAP or any other part of keyclock is updating this table even keyclock serviec is down?)

We currently exclude transient/session tables from database replication to avoid similar conflicts :

• OFFLINE_USER_SESSION
• EVENT_ENTITY
• ADMIN_EVENT_ENTITY
• OFFLINE_CLIENT_SESSION

However, ignoring COMPONENT_CONFIG is not an option for us because excluding it breaks LDAP federation, user attribute mappers, and component configurations across sites.

Questions

1. Is there any configuration or mechanism in Keycloak to prevent Keycloak instances in DC2 from issuing concurrent COMPONENT_CONFIG updates/deletes at the exact same time when both DC are running?
2. Is COMPONENT_CONFIG expected to be updated automatically on node startup or cache invalidation, and if so, how can we serialize or delay these updates across multi-cluster deployments?
3. What is the recommended operational process when restoring/syncing a Keycloak DB snapshot to a secondary DC to prevent both sides from modifying COMPONENT_CONFIG simultaneously?

Any insights on preventing these simultaneous table updates during cross-DC sync would be greatly appreciated!

Thanks,
Smit