This document describes MinIO's cluster communication system, which enables peer-to-peer coordination across nodes in a distributed MinIO cluster. The system uses NotificationSys for cluster-wide coordination, peerRESTClient and peerRESTServer for RPC calls, and the Grid protocol for high-performance binary communication. This page covers peer-to-peer notification patterns for state synchronization (IAM, configuration, metadata), cluster operations (restart, stop, updates), and cache invalidation.
For information about the data storage layer (erasure coding and disk operations), see 2.4 Storage Layer Architecture. For cluster healing and background operations, see 4.1 Healing System. For site-to-site replication across geographic clusters, see 3.4 Site Replication. For bucket-level replication, see 3.3 Bucket Replication.
Sources: cmd/notification.go1-50 cmd/peer-rest-client.go1-50 cmd/peer-rest-server.go1-50
MinIO's cluster communication system uses a symmetric peer-to-peer architecture where each node can communicate with every other node in the cluster. When state changes occur (e.g., IAM user created, configuration updated, cache invalidated), the coordinating node broadcasts these changes to all peers to maintain cluster-wide consistency.
The system consists of three main components:
Communication occurs over two protocols:
Sources: cmd/notification.go48-52 cmd/peer-rest-client.go43-50 cmd/peer-rest-server.go50-51
Sources: cmd/notification.go48-52 cmd/globals.go229 cmd/peer-rest-client.go43-50
NotificationSys is the central component responsible for broadcasting operations to peer nodes. It maintains two lists of peer clients:
The NotificationSys is initialized during server startup and stored as a global:
| Global Variable | Type | Purpose |
|---|---|---|
globalNotificationSys | *NotificationSys | Singleton instance used cluster-wide |
| Initialized at | Server startup | cmd/server-main.go452 |
| Peer clients | []*peerRESTClient | One client per remote peer node |
Key Methods:
| Method | Purpose | Returns |
|---|---|---|
DeleteUser() | Delete user across all peers | []NotificationPeerErr |
LoadPolicy() | Reload policy on all peers | []NotificationPeerErr |
DeleteBucket() | Delete bucket metadata cluster-wide | []NotificationPeerErr |
ReloadPoolMeta() | Reload pool metadata | []NotificationPeerErr |
SignalService() | Send service signal (restart/stop) | []NotificationPeerErr |
Sources: cmd/notification.go48-52 cmd/globals.go229 cmd/server-main.go452
Each peerRESTClient represents a connection to a single peer node. The client is initialized with both REST and Grid connection capabilities:
Initialization Process:
*xnet.Host from peer addressSources: cmd/peer-rest-client.go43-106 cmd/peer-rest-client.go52-106
MinIO uses two communication protocols for peer coordination:
High-performance binary RPC framework optimized for frequent, small operations:
| Feature | Description |
|---|---|
| Protocol | Binary RPC over TCP |
| Use Cases | Disk operations, health checks, frequent state sync |
| Handler Registration | grid.NewSingleHandler, grid.NewStreamHandler |
| Examples | deleteUserRPC, loadPolicyRPC, getBucketStatsRPC |
Grid Handlers in peer-rest-server.go:
Sources: cmd/peer-rest-server.go54-88
Standard HTTP REST API for admin operations and compatibility:
| Feature | Description |
|---|---|
| Protocol | HTTP/HTTPS with JSON payloads |
| Use Cases | Admin operations, binary updates, diagnostics |
| Path Prefix | /minio/peer/v39/ |
| Authentication | Authorization header with node auth token |
REST Methods:
| Constant | Value | Handler |
|---|---|---|
peerRESTMethodHealth | "/health" | Health check endpoint |
| Legacy methods | Various | Used for backward compatibility |
Sources: cmd/peer-rest-common.go22-83 cmd/peer-rest-client.go66-77
When IAM changes occur, the coordinator broadcasts updates to all peers:
IAM Synchronization Methods:
| Method | Operation | Grid Handler |
|---|---|---|
LoadUser() | Reload specific user | loadUserRPC |
DeleteUser() | Delete user cluster-wide | deleteUserRPC |
LoadPolicy() | Reload policy definition | loadPolicyRPC |
DeletePolicy() | Delete policy cluster-wide | deletePolicyRPC |
LoadPolicyMapping() | Reload user/group policy mappings | loadPolicyMappingRPC |
LoadServiceAccount() | Reload service account credentials | loadSvcActRPC |
DeleteServiceAccount() | Delete service account | deleteSvcActRPC |
LoadGroup() | Reload group membership | loadGroupRPC |
Sources: cmd/notification.go155-194 cmd/peer-rest-server.go73-76
Configuration changes are propagated to all peers to ensure consistent settings:
Configuration Operations:
| NotificationSys Method | RPC Handler | Purpose |
|---|---|---|
ReloadSiteReplicationConfig() | reloadSiteReplicationConfigRPC | Reload site replication settings |
| N/A (direct call) | getSysConfigRPC | Retrieve system configuration |
| N/A (direct call) | serverInfoHandler | Get server properties |
Sources: cmd/notification.go945-959 cmd/peer-rest-server.go84-88
To maintain consistency, cached metadata must be invalidated across the cluster:
Cache Invalidation Methods:
| Method | Purpose | RPC Handler |
|---|---|---|
ReloadBucketMetadata() | Invalidate bucket metadata cache | loadBucketMetadataRPC |
ReloadPoolMeta() | Reload pool-level metadata | reloadPoolMetaRPC |
ReloadFormat() | Reload disk format information | reloadFormatRPC |
DeleteBucketMetadata() | Remove bucket metadata from cache | deleteBucketMetadataRPC |
Sources: cmd/notification.go761-795 cmd/peer-rest-server.go73-88
Service control operations (restart, stop, freeze) are coordinated cluster-wide:
Service Signals:
| Signal Type | Value | Handler Behavior |
|---|---|---|
serviceRestart | Restart signal | Gracefully restart server process |
serviceStop | Stop signal | Gracefully stop server process |
serviceFreeze | Freeze signal | Pause S3 API request processing |
serviceUnFreeze | Unfreeze signal | Resume S3 API request processing |
Implementation Details:
The service signal handler processes signals from globalServiceSignalCh:
restartUpdateDelay (10 seconds) to allow response to be sentSources: cmd/admin-handlers.go461-522 cmd/admin-handlers.go549-635 cmd/peer-rest-server.go241-267
Server binary updates are coordinated across all nodes:
Update Process:
| Step | NotificationSys Method | Purpose |
|---|---|---|
| 1. Verify | VerifyBinary() | Check SHA256 and prepare binary |
| 2. Commit | CommitBinary() | Move binary to final location |
| 3. Signal | SignalService() | Trigger coordinated restart |
Grid RPC Handlers:
| Handler | Operation |
|---|---|
verifyBinaryHandler | Verify binary SHA256 and prepare for commit |
commitBinaryHandler | Commit binary to final location |
Sources: cmd/admin-handlers.go88-319 cmd/notification.go1129-1161 cmd/peer-rest-server.go225-240
The NotificationGroup provides a pattern for parallel execution with error collection:
Usage Pattern:
// Create group for N peers
ng := WithNPeers(len(peerClients)).WithRetries(3)
// Launch parallel operations
for idx, client := range peerClients {
ng.Go(ctx, func() error {
return client.SomeOperation()
}, idx, *client.host)
}
// Wait for all to complete
errs := ng.Wait()
Key Features:
| Feature | Description |
|---|---|
| Parallel execution | Uses worker pool to limit concurrency |
| Error collection | Collects errors from all peers in slice |
| Retry logic | Configurable retry count with exponential backoff |
| Context cancellation | Respects context cancellation for early termination |
| Throttling | WithNPeersThrottled() limits concurrent workers |
Sources: cmd/notification.go60-152 cmd/notification.go70-100
This example shows the complete flow when an IAM user is deleted:
Code Flow:
globalIAMSys.DeleteUser()globalNotificationSys.DeleteUser() to broadcastNotificationGroup and calls each peerdeleteUserRPC and invalidates cacheSources: cmd/notification.go196-208 cmd/peer-rest-server.go73-76
Peer-to-peer communication is authenticated using a node authentication token:
Token Generation:
The node auth token is generated at startup from root credentials:
| Global Variable | Purpose | Generated At |
|---|---|---|
globalNodeAuthToken | Authentication token for inter-node communication | cmd/test-utils_test.go86 |
globalActiveCred | Root credentials used to generate token | cmd/globals.go308 |
Token Caching:
Tokens are cached to avoid repeated JWT generation overhead:
type cachedAuthToken struct {
sync.RWMutex
token string
}
func newCachedAuthToken() *cachedAuthToken {
return &cachedAuthToken{}
}
Sources: cmd/test-utils_test.go81-86 cmd/peer-rest-client.go65
Peer communication can fail for various reasons:
| Error Type | Description | Handling |
|---|---|---|
| Network errors | Connection refused, timeout, DNS failure | Retry with backoff |
| Context canceled | Operation canceled by caller | No retry |
| Peer unreachable | errPeerNotReachable | Retry |
| Authentication errors | Invalid token | Log and fail |
The NotificationGroup implements retry logic with exponential backoff:
Retry Configuration:
| Method | Default Retries | Typical Use Case |
|---|---|---|
WithRetries(1) | 1 | Quick operations (IAM sync) |
WithRetries(3) | 3 | Standard operations (default) |
| No retries | 0 | Context-sensitive operations |
Backoff Formula:
waitTime = 100ms + random(0, 1000ms)
This adds jitter to prevent thundering herd problems when multiple nodes retry simultaneously.
Sources: cmd/notification.go114-151 cmd/notification.go95-99
MinIO intelligently selects between Grid and REST protocols:
| Operation | Protocol | Reason |
|---|---|---|
| IAM state sync | Grid | High frequency, small payloads |
| Disk operations | Grid | Low latency required |
| Health checks | REST | Backward compatibility |
| Binary updates | REST | Large payloads, less frequent |
| Service signals | Grid | Time-sensitive |
| Metrics collection | Grid | Streaming data |
The client lazily initializes Grid connection:
gridConn() returns non-nildeleteUserRPC, etc.)Sources: cmd/peer-rest-client.go78-105 cmd/peer-rest-server.go54-88
Initialization Steps:
initAllSubsystems()globalEndpointspeerRESTClient per remote nodeglobalNotificationSys for cluster-wide accessGlobal Variables Involved:
| Variable | Initialized | Purpose |
|---|---|---|
globalNotificationSys | cmd/server-main.go452 | Cluster-wide notification system |
globalEndpoints | Server startup | List of all cluster endpoints |
globalGrid | Grid initialization | Grid manager instance |
Sources: cmd/server-main.go450-504 cmd/notification.go1258-1280 cmd/globals.go229
All peer operations execute in parallel using worker pools:
| Configuration | Purpose | Performance Impact |
|---|---|---|
WithNPeers(N) | Create N-sized error slice | Memory: O(N) |
WithNPeersThrottled(N, W) | Limit to W concurrent workers | CPU: Max W goroutines |
| Worker pool | Reuse goroutines | Reduces allocation overhead |
Grid protocol optimizations:
| Optimization | Benefit |
|---|---|
| Binary encoding | Reduces payload size vs JSON |
| Connection pooling | Reuses TCP connections |
| No middleware overhead | Direct RPC dispatch |
| Stream support | Efficient for large data transfers |
NotificationSys ensures cache consistency but minimizes invalidation:
Sources: cmd/notification.go70-92 internal/grid/README.md
MinIO's distributed coordination system provides:
The system is designed for high availability and consistency in distributed deployments, ensuring that all nodes maintain a synchronized view of cluster state while allowing the cluster to continue operating even if individual peer communications fail.
Sources: cmd/notification.go1-1300 cmd/peer-rest-client.go1-500 cmd/peer-rest-server.go1-800
Refresh this wiki