Site Replication

Relevant source files

• cmd/admin-handlers-idp-ldap.go
• cmd/admin-handlers-site-replication.go
• cmd/admin-handlers-users-race_test.go
• cmd/admin-handlers-users.go
• cmd/admin-handlers-users_test.go
• cmd/iam-etcd-store.go
• cmd/iam-object-store.go
• cmd/iam-store.go
• cmd/iam.go
• cmd/site-replication.go
• cmd/sts-datatypes.go
• cmd/sts-errors.go
• cmd/sts-handlers.go
• cmd/sts-handlers_test.go
• cmd/stserrorcode_string.go
• internal/config/identity/ldap/ldap.go

Purpose: Site Replication enables synchronization of Identity and Access Management (IAM) configuration and bucket metadata across multiple geographically distributed MinIO clusters. This feature ensures that users, policies, groups, service accounts, and bucket configurations remain consistent across all participating sites, allowing for disaster recovery and multi-region deployments with unified identity management.

Scope: This document covers the site replication subsystem, focusing on IAM and bucket metadata synchronization. For object-level data replication between buckets, see Bucket Replication. For the underlying IAM system being replicated, see Identity and Access Management. For peer-to-peer communication mechanisms used by site replication, see Distributed Coordination.

---

Architecture Overview

The site replication system (SiteReplicationSys) maintains a federation of MinIO clusters where IAM changes on any cluster are automatically propagated to all other clusters in near real-time. Each cluster maintains its own storage but shares a common IAM configuration.

Site Replication Flow Diagram

Sources: cmd/site-replication.go199-210 cmd/iam.go86-112

---

Core Components

SiteReplicationSys Structure

The SiteReplicationSys struct manages all site replication operations:

Field	Type	Purpose
enabled	bool	Whether site replication is active
state	srState	Current replication state including peer list
iamMetaCache	srIAMCache	Cache for IAM metadata synchronization

The state is persisted at the path defined by getSRStateFilePath() which returns minioConfigPrefix + "/site-replication/state.json".

Sources: cmd/site-replication.go199-210 cmd/site-replication.go174-176

State Persistence Format

Site Replication State Data Model

The state contains:

• Name: This cluster's name in the replication setup
• Peers: Map of deployment IDs to peer cluster information
• ServiceAccountAccessKey: Access key for the special site-replicator-0 service account
• UpdatedAt: Timestamp of last state update

Sources: cmd/site-replication.go211-229 cmd/site-replication.go54-61

---

Initialization and Peer Addition

AddPeerClusters Flow

The AddPeerClusters function orchestrates adding clusters to the site replication configuration:

Peer Cluster Addition Sequence

Key validation steps before enabling replication:

1. Deployment ID Uniqueness: Each cluster must have a unique deployment ID
2. IDP Settings Match: All clusters must use identical LDAP/OpenID configurations
3. Data Requirements: Only one cluster (typically the coordinator) may have existing data

Sources: cmd/site-replication.go396-611 cmd/site-replication.go685-712

Service Account Creation

A special service account siteReplicatorSvcAcc (value: "site-replicator-0") is created during setup:

Service Account Creation and Distribution

This service account:

• Has the same access key (site-replicator-0) on all clusters
• Has the same secret key synchronized across all clusters
• Is allowed via allowSiteReplicatorAccount flag in newServiceAccountOpts
• Is used to authenticate all cross-cluster replication API calls

Sources: cmd/site-replication.go348-351 cmd/site-replication.go479-499

---

IAM Change Synchronization

IAMChangeHook Mechanism

The IAMChangeHook function is the central mechanism for replicating IAM changes. It is called whenever an IAM entity is created, updated, or deleted:

IAM Change Propagation Architecture

Sources: cmd/site-replication.go1239-1456

SRIAMItem Structure

The madmin.SRIAMItem structure encapsulates all IAM change information:

Field	Type	Description
Type	madmin.SRItemType	Type of IAM item (user, policy, group, etc.)
Name	string	Name of the entity being changed
IAMUser	*madmin.SRIAMUser	User creation/deletion details
IAMPolicy	*madmin.SRIAMPolicy	Policy definition
PolicyMapping	*madmin.SRPolicyMapping	User/group to policy associations
GroupInfo	*madmin.SRGroupInfo	Group membership changes
SvcAccChange	*madmin.SRSvcAccChange	Service account operations
STSCredential	*madmin.SRSTSCredential	STS temporary credentials
UpdatedAt	time.Time	Timestamp of change

Example call sites:

User Deletion:

// cmd/admin-handlers-users.go:93-100
replLogIf(ctx, globalSiteReplicationSys.IAMChangeHook(ctx, madmin.SRIAMItem{
    Type: madmin.SRIAMItemIAMUser,
    IAMUser: &madmin.SRIAMUser{
        AccessKey:   accessKey,
        IsDeleteReq: true,
    },
    UpdatedAt: UTCNow(),
}))

Service Account Creation:

// cmd/admin-handlers-users.go:761-778
replLogIf(ctx, globalSiteReplicationSys.IAMChangeHook(ctx, madmin.SRIAMItem{
    Type: madmin.SRIAMItemSvcAcc,
    SvcAccChange: &madmin.SRSvcAccChange{
        Create: &madmin.SRSvcAccCreate{
            Parent:        newCred.ParentUser,
            AccessKey:     newCred.AccessKey,
            SecretKey:     newCred.SecretKey,
            Groups:        newCred.Groups,
            Claims:        opts.claims,
            SessionPolicy: madmin.SRSessionPolicy(createReq.Policy),
            Status:        auth.AccountOn,
        },
    },
    UpdatedAt: updatedAt,
}))

Sources: cmd/admin-handlers-users.go93-100 cmd/admin-handlers-users.go761-778 cmd/site-replication.go1239-1456

Replication Implementation

The IAMChangeHook function distributes changes to all peer clusters:

IAM Change Replication Sequence

The replication process:

1. Checks if site replication is enabled
2. Calls concurrentReplicateIAMItem() to fan out to all peers in parallel
3. Each peer receives the change via SRPeerReplicateIAMItem handler
4. Handler routes to specific function based on item.Type:
   • SRIAMItemIAMUser → PeerAddUserHandler() or PeerDeleteUserHandler()
   • SRIAMItemPolicy → PeerAddPolicyHandler()
   • SRIAMItemGroupInfo → PeerGroupInfoHandler()
   • SRIAMItemSvcAcc → PeerSvcAccChangeHandler()
   • SRIAMItemSTSAcc → PeerSTSAccHandler()
   • SRIAMItemPolicyMapping → PeerPolicyMappingHandler()

Sources: cmd/site-replication.go1239-1456 cmd/admin-handlers-site-replication.go148-210

---

Bucket Operations Replication

Site replication also synchronizes bucket-level operations and metadata across clusters.

Bucket Creation Hook

The MakeBucketHook function replicates bucket creation:

Bucket Creation Replication Flow

Key features:

• Bucket is created with versioning enabled by default
• Lock status is replicated if source bucket has object locking
• Bucket creation timestamp is synchronized
• Replication rules are automatically configured between all peers

Sources: cmd/site-replication.go797-893

Bucket Metadata Replication

The BucketMetaHook function replicates bucket metadata changes:

Bucket Metadata Synchronization

The metadata hook handles:

• Bucket Policies: Access control policies
• Bucket Tags: Key-value tags
• Object Lock Configuration: Retention and legal hold settings
• SSE Configuration: Server-side encryption settings
• Versioning: Versioning state changes
• Quota: Storage quotas
• Lifecycle Rules: ILM expiration and transition rules (if ReplicateILMExpiry is enabled)

Sources: cmd/site-replication.go1051-1210 cmd/admin-handlers-site-replication.go228-331

---

API Endpoints

Admin API Handlers

The site replication system exposes several admin API endpoints:

Endpoint	Method	Handler	Purpose
/minio/admin/v3/site-replication/add	POST	SiteReplicationAdd	Add peer clusters to replication
/minio/admin/v3/site-replication/join	PUT	SRPeerJoin	Internal: peer joins replication
/minio/admin/v3/site-replication/iam-item	PUT	SRPeerReplicateIAMItem	Replicate IAM change to peer
/minio/admin/v3/site-replication/bucket-ops	PUT	SRPeerBucketOps	Replicate bucket operation
/minio/admin/v3/site-replication/bucket-meta	PUT	SRPeerReplicateBucketItem	Replicate bucket metadata
/minio/admin/v3/site-replication/info	GET	SiteReplicationInfo	Get replication status
/minio/admin/v3/site-replication/status	GET	SiteReplicationStatus	Get detailed peer status
/minio/admin/v3/site-replication/edit	PUT	SiteReplicationEdit	Modify replication config
/minio/admin/v3/site-replication/remove	PUT	SiteReplicationRemove	Remove peer from replication

Sources: cmd/admin-handlers-site-replication.go39-447

Authentication for Cross-Cluster Calls

All cross-cluster API calls use the special site-replicator-0 service account:

Cross-Cluster Authentication

The getAdminClient() function creates an authenticated admin client for peer communication:

// cmd/site-replication.go:1562-1594
func (c *SiteReplicationSys) getAdminClient(ctx context.Context, deploymentID string) (*madmin.AdminClient, error) {
    // Get peer info from state
    peerInfo := c.state.Peers[deploymentID]
    
    // Use globalSiteReplicatorCred for authentication
    secretKey, err := globalSiteReplicatorCred.Get(ctx)
    
    // Create admin client
    return getAdminClient(peerInfo.Endpoint, 
                         c.state.ServiceAccountAccessKey,
                         secretKey)
}

Sources: cmd/site-replication.go1562-1594 cmd/sts-handlers.go244-255

---

Healing and Resynchronization

The site replication system includes healing mechanisms to recover from failures or synchronize newly added clusters.

Background Healing Routine

Site Replication Healing Process

The healing routine periodically:

1. Scans the local IAM system
2. Queries all peer clusters for their IAM state
3. Identifies inconsistencies (missing users, policies, groups)
4. Replicates missing items to bring peers into sync

Sources: cmd/site-replication.go232-255 cmd/site-replication.go2678-2862

Initial Sync After Adding Peers

When new clusters are added, syncToAllPeers() performs an initial synchronization:

Initial Synchronization Sequence

This ensures that a newly added cluster receives:

• All existing buckets with their configurations
• All IAM users, policies, and groups
• All service accounts (except those belonging to root user)
• All policy mappings (user-to-policy and group-to-policy associations)

Sources: cmd/site-replication.go1901-2074

---

IDP Settings Validation

All peer clusters must have identical Identity Provider (IDP) settings to ensure consistent authentication behavior.

Validated Settings

IDP Settings Validation Flow

The validation process:

1. Queries each peer via SRPeerGetIDPSettings API
2. Compares all IDP settings using reflect.DeepEqual()
3. Rejects setup if any mismatch is found

Example validation code:

// cmd/site-replication.go:705-709
for i := 1; i < len(s); i++ {
    if !reflect.DeepEqual(s[i], s[0]) {
        return errSRIAMConfigMismatch(peers[0].Name, peers[i].Name, s[0], s[i])
    }
}

Sources: cmd/site-replication.go685-712 cmd/site-replication.go669-683

---

Error Handling and Status Tracking

Error Types

The site replication system defines specific error types:

Error	Code	Description
errSRCannotJoin	ErrSiteReplicationInvalidRequest	Cluster already in replication
errSRDuplicateSites	ErrSiteReplicationInvalidRequest	Duplicate deployment IDs provided
errSRSelfNotFound	ErrSiteReplicationInvalidRequest	Current cluster not in peer list
errSRPeerNotFound	ErrSiteReplicationInvalidRequest	Specified peer not found
errSRNotEnabled	ErrSiteReplicationInvalidRequest	Site replication not configured
errSRBackendIssue	ErrSiteReplicationBackendIssue	Storage backend error
errSRIAMError	ErrSiteReplicationIAMError	IAM operation failed
errSRBucketConfigError	ErrSiteReplicationBucketConfigError	Bucket config error

Sources: cmd/site-replication.go64-173

Status Monitoring

The SiteReplicationStatus API provides detailed status information:

Site Replication Status Data Model

The status API returns:

• Overall replication status (enabled/disabled)
• Counts of users, groups, policies, and buckets per site
• Detailed synchronization status for each entity
• Mismatch information (e.g., policy differences between sites)

Sources: cmd/site-replication.go2152-2574

---

Integration Points

IAM System Integration

Every IAM mutation in the codebase includes a site replication hook:

User Operations:

• CreateUser() → IAMChangeHook() with SRIAMItemIAMUser cmd/admin-handlers-users.go538-546
• DeleteUser() → IAMChangeHook() with IsDeleteReq: true cmd/admin-handlers-users.go93-100
• SetUserStatus() → IAMChangeHook() with status change cmd/admin-handlers-users.go425-435

Policy Operations:

• SetPolicy() → IAMChangeHook() with SRIAMItemPolicy cmd/site-replication.go1456
• DeletePolicy() → IAMChangeHook() with IsDeleteReq: true
• AttachPolicy() → IAMChangeHook() with SRIAMItemPolicyMapping

Group Operations:

• AddUsersToGroup() → IAMChangeHook() with SRIAMItemGroupInfo cmd/admin-handlers-users.go297-303
• RemoveUsersFromGroup() → IAMChangeHook() with group update
• SetGroupStatus() → IAMChangeHook() with status change cmd/admin-handlers-users.go387-397

Service Account Operations:

• NewServiceAccount() → IAMChangeHook() with SRIAMItemSvcAcc (if not root account) cmd/admin-handlers-users.go761-778
• UpdateServiceAccount() → IAMChangeHook() with update details cmd/admin-handlers-users.go887-901
• DeleteServiceAccount() → IAMChangeHook() with delete request

Sources: cmd/admin-handlers-users.go93-100 cmd/admin-handlers-users.go538-546 cmd/admin-handlers-users.go761-778

STS Integration

The STS (Security Token Service) system also integrates with site replication:

// cmd/sts-handlers.go:349-361
// Call hook for site replication.
if cred.ParentUser != globalActiveCred.AccessKey {
    replLogIf(ctx, globalSiteReplicationSys.IAMChangeHook(ctx, madmin.SRIAMItem{
        Type: madmin.SRIAMItemSTSAcc,
        STSCredential: &madmin.SRSTSCredential{
            AccessKey:    cred.AccessKey,
            SecretKey:    cred.SecretKey,
            SessionToken: cred.SessionToken,
            ParentUser:   cred.ParentUser,
        },
        UpdatedAt: updatedAt,
    }))
}

Note: Root user's STS credentials are not replicated (as indicated by the check cred.ParentUser != globalActiveCred.AccessKey).

Sources: cmd/sts-handlers.go349-361 cmd/sts-handlers.go589-600

---

Key Functions and Code Paths

Core Functions Summary

Function	File	Purpose
AddPeerClusters()	cmd/site-replication.go:397-611	Add new clusters to replication setup
PeerJoinReq()	cmd/site-replication.go:614-665	Handle join request from coordinator
IAMChangeHook()	cmd/site-replication.go:1239-1456	Replicate IAM change to all peers
MakeBucketHook()	cmd/site-replication.go:797-893	Replicate bucket creation
DeleteBucketHook()	cmd/site-replication.go:895-950	Replicate bucket deletion
BucketMetaHook()	cmd/site-replication.go:1051-1210	Replicate bucket metadata changes
validateIDPSettings()	cmd/site-replication.go:685-712	Ensure IDP config matches across peers
syncToAllPeers()	cmd/site-replication.go:1901-2074	Initial sync of all entities
healIAMSystem()	cmd/site-replication.go:2678-2862	Background healing routine
getAdminClient()	cmd/site-replication.go:1562-1594	Get authenticated client for peer

Service Account Constant

// cmd/site-replication.go:348-351
const (
    // Access key of service account used for perform cluster-replication
    // operations.
    siteReplicatorSvcAcc = "site-replicator-0"
)

This constant defines the fixed access key used for all cross-cluster authentication.

Sources: cmd/site-replication.go348-351 cmd/site-replication.go397-2862

---

Configuration Example

To set up site replication between three clusters:

1. Prepare clusters: Ensure all clusters are running and have identical IDP configurations
2. Call AddPeerClusters: From one cluster (coordinator), provide peer information
3. Service account created: site-replicator-0 account is automatically created with synchronized credentials
4. Initial sync: All IAM entities and buckets are replicated to peers
5. Ongoing replication: All subsequent IAM/bucket changes are automatically replicated

The coordinator cluster should be the one with existing data (if any). All other clusters should be empty to avoid conflicts.

Sources: cmd/site-replication.go396-611