Bucket Replication

Relevant source files

• cmd/api-errors.go
• cmd/api-response.go
• cmd/api-router.go
• cmd/apierrorcode_string.go
• cmd/bucket-handlers.go
• cmd/bucket-replication.go
• cmd/encryption-v1.go
• cmd/encryption-v1_test.go
• cmd/generic-handlers.go
• cmd/generic-handlers_test.go
• cmd/handler-utils.go
• cmd/object-api-datatypes.go
• cmd/object-api-errors.go
• cmd/object-api-options.go
• cmd/object-api-utils.go
• cmd/object-api-utils_test.go
• cmd/object-handlers.go
• cmd/routers.go

Purpose and Scope

This document describes bucket-level replication in MinIO, which synchronizes objects and metadata between buckets within a single deployment or across different MinIO deployments. Bucket replication enables cross-region data protection, data locality optimization, and active-active configurations where objects can be read from any target even if they don't exist locally.

Note: For multi-site replication that synchronizes IAM policies, bucket configurations, and other system-level settings across entire MinIO deployments, see Site Replication.

---

Overview

Bucket replication in MinIO allows objects written to a source bucket to be automatically copied to one or more remote target buckets. The system supports:

• Unidirectional replication: Source → Target(s)
• Active-active replication: Bidirectional sync where both sides can accept writes
• Object replication: Complete object data and metadata
• Delete replication: Propagation of delete markers and version deletions
• Metadata-only replication: Synchronization of tags, retention settings, legal holds
• Existing object replication: Replicating objects that existed before replication was configured
• SSE-C encrypted object replication: Including encrypted objects with customer-provided keys

The replication subsystem operates asynchronously, queuing replication tasks and tracking their status through metadata stored with each object version.

Sources: cmd/bucket-replication.go1-1362

---

Architecture Overview

Replication Flow Overview

The diagram shows how replication flows through MinIO when objects are created or deleted:

1. Decision Phase: API handlers invoke mustReplicate() or checkReplicateDelete() to determine if replication is needed
2. Queuing Phase: Eligible operations are queued to globalReplicationPool
3. Execution Phase: Workers execute replication to each target via TargetClient
4. Status Tracking: Metadata is updated with replication status
5. Failure Handling: Failed replications are queued to MRF for retry

Sources: cmd/bucket-replication.go251-307 cmd/bucket-replication.go421-600 cmd/object-handlers.go405-465

---

Replication Configuration

Configuration Structure

Bucket replication is configured using an AWS S3-compatible replication configuration XML. The configuration is stored and managed by globalBucketMetadataSys and accessed via getReplicationConfig():

Target Configuration

Replication targets are configured with:

• ARN (Amazon Resource Name) identifying the target
• Endpoint URL of the remote MinIO deployment
• Credentials for authentication
• Bandwidth limits (optional)
• Health check interval
• Synchronous vs asynchronous replication mode

Sources: cmd/bucket-replication.go84-91 cmd/bucket-replication.go93-170

Validation

When a replication configuration is added or updated, validateReplicationDestination() performs several checks:

Validation Check	Purpose
Target bucket exists	Ensures destination is accessible
Object lock compatibility	Source and target must both have object lock enabled
Self-referential check	Prevents replication loops by checking deployment IDs
ARN format validation	Ensures valid replication service ARN
Connectivity check	Verifies target endpoint is reachable

Sources: cmd/bucket-replication.go93-170 cmd/bucket-replication.go172-209

---

Replication Decision Process

Decision Flow

Function: mustReplicate()

The core decision function is mustReplicate() at cmd/bucket-replication.go253-307:

Key Parameters:

• bucket: Source bucket name
• object: Object name/key
• mopts: Options including metadata, replication status, operation type

Decision Criteria:

1. Versioning: Object must be in a versioned prefix
2. Not a replica: Objects with ReplicationStatus=Replica are not re-replicated (except metadata-only)
3. Not incoming: Requests with ReplicationRequest=true flag are skipped
4. Has configuration: Bucket must have replication rules
5. Rule matching: Object must match at least one replication rule's filters (prefix, tags, etc.)

Output: ReplicateDecision containing per-target decisions with:

• Whether to replicate
• Whether replication should be synchronous
• Target ARN

Sources: cmd/bucket-replication.go253-307 cmd/bucket-replication.go211-249

---

Object Replication

Replication Workflow

Metadata Transfer

When replicating an object, the system prepares metadata using putReplicationOpts() cmd/bucket-replication.go778-896:

Included Metadata:

• User-defined metadata (x-amz-meta-*)
• Standard headers (Content-Type, Cache-Control, Content-Encoding, etc.)
• Object tags
• Storage class
• Checksums (for integrity verification)
• SSE-C encryption parameters (for encrypted objects)

Special Headers for Replication:

• X-Minio-Source-ETag: Original object ETag
• X-Minio-Source-MTime: Original modification time
• X-Minio-Replication-*: Internal replication metadata
• X-Amz-Bucket-Replication-Status: Set to "REPLICA"

Excluded Metadata:

• Internal MinIO metadata (X-Minio-Internal-*)
• Replication status from source
• Tier metadata (for transitioned objects)

Sources: cmd/bucket-replication.go778-896 cmd/bucket-replication.go715-756 cmd/handler-utils.go76-115

Replication States

Objects track replication status through metadata fields:

Status	Description
PENDING	Queued for replication, not yet started
COMPLETED	Successfully replicated to target
FAILED	Replication attempt failed
REPLICA	Object is a replica received from another site

These states are stored in the object's xl.meta under keys defined at cmd/bucket-replication.go60-82:

• ReplicationStatus: Overall replication status
• ReplicationTimestamp: Last replication attempt time
• ReplicaStatus: Marks object as a replica
• ReplicaTimestamp: When replica was received

Sources: cmd/bucket-replication.go60-82

---

Delete Replication

Delete Marker Replication

When a DELETE request is made without specifying a version ID, MinIO creates a delete marker. The system then replicates this marker to targets:

Version Purge (Permanent Delete)

When a DELETE request includes a specific version ID, it's a permanent delete:

Replication Behavior:

1. Source marks version with VersionPurgeStatus=Pending
2. Version remains on source but hidden from listings
3. Replication worker calls RemoveObject() on target with version ID
4. On success, source updates to VersionPurgeStatus=Complete
5. Source then permanently deletes the version

This ensures the version isn't permanently deleted on the source until it's confirmed deleted on all targets.

Sources: cmd/bucket-replication.go421-600 cmd/bucket-replication.go602-713 cmd/bucket-replication.go346-409

Delete Replication Metadata

Delete operations track additional metadata:

Metadata Key	Purpose
DeleteMarkerReplicationStatus	Status of delete marker replication
VersionPurgeStatus	Status of permanent version deletion
VersionPurgeStatuses	Per-target purge status (for multiple targets)

Sources: cmd/bucket-replication.go60-82

---

Active-Active Replication

Proxy Replication

MinIO supports active-active replication where clients can read objects from either site, even if the object only exists on the remote site. This is implemented through "proxy replication":

Implementation Details:

The proxy logic at cmd/object-handlers.go412-430 works as follows:

1. Detection: When GetObjectNInfo() returns ObjectNotFound, VersionNotFound, or ReadQuorum errors
2. Target Selection: getProxyTargets() identifies replication targets marked as active-active
3. Proxy Request: proxyGetToReplicationTarget() fetches from remote
4. Streaming: Object data streams directly from remote to client
5. Metrics: globalReplicationStats tracks proxy operations

HEAD Request Proxying: Similar logic at cmd/object-handlers.go824-840 proxies HEAD requests to get metadata.

Sources: cmd/object-handlers.go404-465 cmd/object-handlers.go822-840

---

Replication Status Tracking

Status Metadata Structure

Each object version stores replication state in its metadata:

Per-Target State Tracking

For multi-target replication, the system tracks state per target using replicatedInfos structure:

Structure:

replicatedInfos {
    Targets []replicatedTargetInfo {
        Arn: "arn:minio:replication::uuid:bucket/target"
        PrevReplicationStatus: "PENDING"
        ReplicationStatus: "COMPLETED"
        ReplicationTimestamp: time.Time
        Endpoint: "target-minio.example.com"
        Secure: true
    }
}

This allows tracking independent replication progress to each configured target.

Sources: cmd/bucket-replication.go60-82

Replication Timestamp Tracking

Different types of metadata changes track their own timestamps:

Timestamp Key	Tracks Changes To
ReplicationTimestamp	Object data replication
TaggingTimestamp	Object tag modifications
ObjectLockRetentionTimestamp	Retention mode/date changes
ObjectLockLegalHoldTimestamp	Legal hold status changes

These timestamps enable MinIO to determine which metadata is newer when bidirectional replication encounters conflicts.

Sources: cmd/bucket-replication.go68-78

---

Failed Replication Handling

MRF (Metadata Replication Failure) Queue

When replication fails, MinIO queues the operation for retry using the MRF system:

MRF Queue Operations:

1. Queueing: Failed replications call queueMRFSave() at cmd/bucket-replication.go482
2. Persistence: MRF queue is stored persistently to survive restarts
3. Worker Processing: Background workers periodically retry MRF entries
4. Backoff: Retry intervals increase exponentially
5. Eventual Consistency: MRF ensures eventual replication even after temporary failures

Sources: cmd/bucket-replication.go482-495 cmd/bucket-replication.go560

Replication Healing

MinIO also provides a healing mechanism for replication via QueueReplicationHeal():

Triggering Conditions:

• Object found to be missing on target during GET request
• Periodic scan detects replication status inconsistencies
• Manual healing initiated by administrator

Process:

1. Verify object should be replicated
2. Check current replication status
3. Re-queue object for replication if needed
4. Update metadata on success

Sources: cmd/object-handlers.go460 cmd/object-handlers.go495 cmd/object-handlers.go866 cmd/object-handlers.go906

---

Integration Points

IAM and Authentication

Replication targets are authenticated using:

• Credentials: Stored in target configuration
• Service Accounts: Can be used for target authentication
• IAM Policies: Control which users can configure replication

The replication worker uses the configured credentials when making requests to remote targets.

Sources: cmd/bucket-replication.go93-170

Encryption

Replication handles multiple encryption scenarios:

Encryption Type	Replication Behavior
SSE-S3	Object re-encrypted at target with target's keys
SSE-KMS	Similar to SSE-S3, target uses its own KMS
SSE-C	Encrypted data transferred with special headers, re-encrypted at target

For SSE-C objects, replication includes special headers defined at cmd/handler-utils.go87-104:

• X-Minio-Replication-Server-Side-Encryption-Sealed-Key
• X-Minio-Replication-Server-Side-Encryption-Seal-Algorithm
• X-Minio-Replication-Server-Side-Encryption-Iv
• X-Minio-Replication-Encrypted-Multipart
• X-Minio-Replication-Actual-Object-Size
• X-Minio-Replication-Ssec-Crc (checksum of encrypted data)

Sources: cmd/bucket-replication.go778-896 cmd/handler-utils.go87-104 cmd/bucket-replication.go80-82

Lifecycle Management

Replication interacts with lifecycle (ILM) rules:

Lifecycle Before Replication: When getting an object, lifecycle rules are evaluated first. If an object should expire, it may be deleted before replication status can be checked cmd/object-handlers.go472-493

Replication Configuration in Lifecycle: Lifecycle rules can check replication status before applying transitions or expiration.

Sources: cmd/object-handlers.go472-493 cmd/object-handlers.go884-905

Versioning

Replication requires versioning to be enabled on both source and target buckets. The check at cmd/bucket-replication.go261 ensures objects in non-versioned prefixes are not replicated.

Versioning enables:

• Tracking which specific version needs replication
• Preserving version history on targets
• Delete marker and version purge replication

Sources: cmd/bucket-replication.go261-263

Object Lock

When both source and target have Object Lock enabled, replication includes:

• Retention Mode: Governance or Compliance mode
• Retention Date: When retention expires
• Legal Hold: Whether object has legal hold set

Validation at cmd/bucket-replication.go128-138 ensures targets with Object Lock can receive locked objects.

Sources: cmd/bucket-replication.go128-138

Event Notifications

Replication generates events:

Event Type	When Generated
ObjectReplicationComplete	Successful replication
ObjectReplicationFailed	Replication failure
ObjectReplicationNotTracked	Replication not configured or error

These events are sent via the event notification system to configured targets (webhook, SQS, etc.).

Sources: cmd/bucket-replication.go444-456 cmd/bucket-replication.go515-527 cmd/bucket-replication.go557-599

Bandwidth Management

Replication respects bandwidth limits configured per target. The globalBucketTargetSys manages these limits and throttles replication traffic accordingly.

Sources: cmd/bucket-replication.go44

---

Error Handling

Replication Errors

Specific error codes for replication failures are defined in cmd/api-errors.go:

Error Code	HTTP Status	Description
ErrReplicationConfigurationNotFoundError	404	No replication configuration exists
ErrRemoteDestinationNotFoundError	400	Target bucket doesn't exist
ErrReplicationDestinationMissingLock	400	Target needs Object Lock enabled
ErrRemoteTargetNotFoundError	400	Target ARN not found in configuration
ErrReplicationRemoteConnectionError	503	Cannot connect to remote target
ErrBucketRemoteIdenticalToSource	400	Target is same as source (loop prevention)
ErrRemoteTargetNotVersionedError	400	Target bucket not versioned
ErrReplicationSourceNotVersionedError	400	Source bucket not versioned

Sources: cmd/api-errors.go127-148

Logging

Replication operations log errors using specialized functions:

• replLogIf(): Logs replication errors
• replLogOnceIf(): Logs error once to avoid spam

These logging functions help track replication issues without flooding logs during persistent failures.

Sources: cmd/bucket-replication.go276 cmd/bucket-replication.go443 cmd/bucket-replication.go514

---

API Operations

Configuration APIs

Bucket replication is configured via S3 API operations handled in cmd/bucket-handlers.go72:

API Operation	Handler	Purpose
PUT ?replication	PutBucketReplicationConfigHandler	Set replication rules
GET ?replication	GetBucketReplicationConfigHandler	Retrieve current configuration
DELETE ?replication	DeleteBucketReplicationConfigHandler	Remove replication

Replication-Aware Object APIs

Object operations check and update replication metadata:

Operation	Replication Aspect
PUT Object	Evaluates rules, queues replication
GET Object	May proxy to target if object missing
HEAD Object	Returns replication status headers
DELETE Object	Replicates delete markers and purges
POST Object (Copy)	Replication metadata handling

Sources: cmd/object-handlers.go313-578 cmd/object-handlers.go746-983

---

Performance Considerations

Asynchronous Replication

By default, replication is asynchronous:

1. Client PUT request returns immediately after storing locally
2. Replication happens in background workers
3. Client doesn't wait for remote target confirmation

This provides low latency for write operations.

Synchronous Replication

Targets can be configured for synchronous replication where:

1. PUT request blocks until replication completes
2. Ensures data is on both sites before returning to client
3. Higher latency but stronger consistency guarantees

The mode is determined by tgt.replicateSync flag checked at cmd/bucket-replication.go302

Worker Pool

The globalReplicationPool manages worker goroutines that:

• Process queued replication tasks
• Limit concurrent replication operations
• Balance load across targets
• Handle retries and backoff

Batching and Optimization

MinIO optimizes replication by:

• Streaming object data directly without buffering in memory
• Using the Grid RPC system for efficient inter-node communication
• Tracking per-target status to avoid redundant operations
• Caching replication configurations to reduce lookups

Sources: cmd/bucket-replication.go302-304