This document explains the fundamental concepts that underpin MinIO's distributed object storage architecture. Understanding these concepts is essential for operating MinIO effectively, troubleshooting issues, and making informed deployment decisions.
For implementation details of specific components, see Storage Layer Architecture. For deployment configuration, see Deployment Options Overview.
MinIO uses Reed-Solomon erasure coding to protect data against disk and node failures. Instead of storing multiple full copies of an object (replication), erasure coding splits each object into N data shards and M parity shards, distributing them across multiple drives. The original object can be reconstructed from any subset of shards that meet the read quorum requirement.
By default, MinIO splits drives equally between data blocks and parity blocks:
| Configuration | Data Blocks | Parity Blocks | Failure Tolerance |
|---|---|---|---|
| 4 drives | 2 | 2 | Up to 2 drives |
| 8 drives | 4 | 4 | Up to 4 drives |
| 16 drives | 8 | 8 | Up to 8 drives |
The erasure coding parameters are determined at object write time based on the number of available drives and storage class configuration:
Sources: cmd/erasure.go85-97 cmd/erasure-object.go344-347 docs/erasure/README.md1-17
The erasure coding implementation uses three key parameters:
These are configured per storage pool and calculated based on drive count:
Sources: cmd/erasure-server-pool.go119-136 cmd/erasure.go86-92 cmd/erasure-metadata.go40-70
Each object is split into shards whose size depends on the object size and erasure parameters:
For a 10 MiB object with 8 data blocks and 1 MiB block size:
Sources: cmd/erasure-metadata.go54-70
MinIO enforces quorum requirements to ensure data consistency and availability. Quorum represents the minimum number of drives that must respond successfully for an operation to succeed.
Read quorum = N / 2 (where N is total drives in the set)
A read operation succeeds if at least N/2 drives return consistent metadata and data. This ensures that even if some drives have stale or corrupted data, the system can identify and return the correct version.
Sources: cmd/erasure.go94-97 cmd/erasure-object.go106-119
Write quorum = (N / 2) + 1 for data/parity balance
Write quorum = N / 2 if data blocks equal parity blocks
A write operation succeeds only if at least write quorum drives successfully persist the data. This ensures that subsequent reads will find enough shards to reconstruct the object.
The write quorum calculation logic:
dataBlocks == parityBlocks: writeQuorum = dataBlocks + 1Sources: cmd/erasure.go86-92 cmd/erasure-multipart.go446-452
Quorum checks occur at multiple stages:
xl.meta filesSources: cmd/erasure-object.go837-859 cmd/erasure-healing.go344-361
MinIO organizes storage using a three-tier hierarchy: Server Pools → Erasure Sets → Drives. This architecture enables horizontal scaling while maintaining strong consistency within each set.
A server pool is an independent collection of drives that can be added to expand capacity without downtime. Each pool operates autonomously with its own erasure coding configuration.
Key characteristics:
Sources: cmd/erasure-server-pool.go52-75 cmd/erasure-server-pool.go392-411
Within each pool, drives are organized into erasure sets of fixed size (typically 4-16 drives). Objects are assigned to sets using consistent hashing, ensuring even distribution.
Set Size Calculation:
MinIO chooses the largest set size (from 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16) that evenly divides the total drive count. For 64 drives: 64 / 16 = 4 sets.
Sources: cmd/erasure-sets.go352-376 cmd/erasure-sets.go631-658
Objects are deterministically placed using the sipHashMod function:
The distribution algorithm (formatErasureVersionV3DistributionAlgoV3 by default) ensures balanced placement across drives within the selected set.
Sources: cmd/erasure-sets.go631-679 cmd/erasure-server-pool.go88-90
MinIO supports two distribution algorithms:
| Algorithm | ID | Description |
|---|---|---|
CRCMOD | V1 | Legacy CRC-based (deprecated) |
SIPMOD | V3 | SipHash-based (current default) |
The algorithm is set at format time and remains fixed for the deployment.
Sources: cmd/erasure-sets.go189-191 cmd/format-erasure.go286-329
MinIO protects against silent data corruption (bit rot) using per-shard checksums. Every data and parity shard written to disk includes a checksum that is verified on read.
MinIO uses HighwayHash by default, a fast and cryptographically strong hash function optimized for modern processors.
Checksum information is stored in the xl.meta file:
ErasureInfo {
Checksums: [
{PartNumber: 1, Algorithm: HighwayHash256S, Hash: [32]byte}
]
}
Sources: cmd/erasure-object.go381-384 cmd/erasure-metadata.go42-51 cmd/bitrot.go
Checksums are verified at multiple stages:
If a checksum mismatch is detected during read, MinIO automatically:
Sources: cmd/erasure-object.go381-423 cmd/erasure-healing.go178-205
For inline data (small objects stored directly in xl.meta), checksums are computed on the entire inline data block rather than per-shard.
Sources: cmd/erasure-metadata.go157-178
MinIO provides strong consistency guarantees for all operations in both standalone and distributed modes.
A PutObject operation is guaranteed to be immediately visible to subsequent GetObject or HeadObject requests:
This is enforced through:
O_DIRECT and fdatasync() for durable writesSources: docs/distributed/README.md23-29 cmd/xl-storage.go87-121
Objects written via PutObject are immediately visible in ListObjects responses:
This guarantee holds because:
Sources: docs/distributed/README.md23-29
Strong consistency requires filesystems that honor POSIX O_DIRECT and fdatasync() semantics:
| Filesystem | Recommended | Notes |
|---|---|---|
| XFS | ✓ Yes | Fully compliant |
| ZFS | ✓ Yes | Fully compliant |
| Btrfs | ✓ Yes | Fully compliant |
| ext4 | ✗ No | Trades consistency for performance |
| NFS | ⚠ Caution | NFSv4 preferred, consistency not guaranteed |
Sources: docs/distributed/README.md23-29
Storage classes allow customization of the data-to-parity ratio, trading storage efficiency for fault tolerance.
The default storage class uses N/2 data blocks and N/2 parity blocks, providing maximum fault tolerance:
MINIO_STORAGE_CLASS_STANDARD="EC:8"
This means:
Sources: docs/erasure/storage-class/README.md
For non-critical data, reduced redundancy uses fewer parity blocks:
MINIO_STORAGE_CLASS_RRS="EC:10"
Objects inherit the storage class from:
x-amz-storage-class header in PutObject requestSources: cmd/erasure-server-pool.go119-136 cmd/erasure-object.go1111-1117
Understanding how objects are distributed across the storage hierarchy is crucial for capacity planning and performance optimization.
Sources: cmd/erasure-server-pool.go618-656 cmd/erasure-sets.go631-679 cmd/erasure-object.go1143-1145
For new objects, MinIO selects a pool based on:
Sources: cmd/erasure-server-pool.go392-411 cmd/erasure-server-pool.go417-479
Within a pool, the set is deterministically chosen using SipHash:
setIndex = sipHashMod(deploymentID, objectPath, setCount)
This ensures:
Sources: cmd/erasure-sets.go631-679
MinIO optimizes small objects by storing them directly in the metadata (xl.meta) file rather than as separate data files on disk. This reduces I/O operations and improves performance for small objects.
Objects smaller than 128 KB are eligible for inline storage. The decision is made at write time based on the actual object size.
Sources: cmd/erasure-object.go1135-1178 cmd/xl-storage-format-v2.go97-222
Inline data is erasure-coded and stored in the Data field of the FileInfo struct within xl.meta:
xl.meta:
FileInfo {
Data: [erasure-coded-shards] // Inline data
Size: 65536 // Original size
Erasure {
DataBlocks: 8
ParityBlocks: 8
BlockSize: 10485760
}
}
On read:
xl.meta from quorum disksThis eliminates the need to open separate part files.
Sources: cmd/erasure-metadata.go157-178 cmd/erasure-object.go288-290
| Operation | Inline (< 128 KB) | Regular (> 128 KB) |
|---|---|---|
| Read I/O | 1 read (xl.meta) | 2 reads (xl.meta + part.1) |
| Write I/O | 1 write (xl.meta) | 2 writes (xl.meta + part.1) |
| Healing | Single file update | Multiple file updates |
| Listing | Metadata contains data | Metadata only |
Sources: cmd/erasure-object.go262-272
| Concept | Default Value | Configurable | Purpose |
|---|---|---|---|
| Data Blocks | N/2 | Via storage class | Store actual object data |
| Parity Blocks | N/2 | Via storage class | Enable reconstruction after failures |
| Read Quorum | N/2 | No | Minimum disks for consistent reads |
| Write Quorum | N/2 + 1 | No | Minimum disks for durable writes |
| Block Size | 1 MiB | No | Unit of erasure coding |
| Set Size | 4-16 drives | At format time | Fixed grouping of drives |
| Inline Threshold | 128 KB | No | Size cutoff for inline storage |
| Distribution Algorithm | SIPMOD (V3) | At format time | Object-to-set mapping |
Sources: cmd/erasure.go cmd/erasure-object.go cmd/erasure-server-pool.go cmd/erasure-sets.go cmd/erasure-metadata.go docs/erasure/README.md docs/distributed/README.md docs/distributed/DESIGN.md
Refresh this wiki