Storage Layer Architecture

Relevant source files

• .github/workflows/run-mint.sh
• cmd/erasure-healing-common.go
• cmd/erasure-healing-common_test.go
• cmd/erasure-healing.go
• cmd/erasure-healing_test.go
• cmd/erasure-metadata-utils.go
• cmd/erasure-metadata.go
• cmd/erasure-metadata_test.go
• cmd/erasure-multipart.go
• cmd/erasure-object.go
• cmd/erasure-object_test.go
• cmd/erasure-server-pool.go
• cmd/erasure-sets.go
• cmd/erasure.go
• cmd/naughty-disk_test.go
• cmd/object-api-interface.go
• cmd/object-multipart-handlers.go
• cmd/storage-datatypes.go
• cmd/storage-datatypes_gen.go
• cmd/storage-datatypes_gen_test.go
• cmd/storage-errors.go
• cmd/storage-interface.go
• cmd/storage-rest-client.go
• cmd/storage-rest-common.go
• cmd/storage-rest-server.go
• cmd/storage-rest_test.go
• cmd/xl-storage-disk-id-check.go
• cmd/xl-storage.go
• cmd/xl-storage_test.go
• internal/hash/checksum.go
• internal/hash/checksum_test.go
• internal/hash/reader.go

Purpose and Scope

This document describes MinIO's storage layer implementation, which handles low-level disk I/O operations and metadata persistence. The storage layer provides an abstraction over physical disks, manages the xl.meta metadata file format, and handles both local and remote disk access.

Related Pages:

• For erasure coding operations built on top of this storage layer, see Erasure Coding Engine
• For metadata caching and optimization, see Metadata Management
• For higher-level erasure set organization, see Server Pools and Erasure Sets

---

Storage API Overview

MinIO defines a StorageAPI interface that abstracts disk operations. This interface has two primary implementations:

Sources: cmd/storage-interface.go28-133 cmd/xl-storage.go97-130 cmd/xl-storage-disk-id-check.go84-101 cmd/storage-rest-client.go162-169 cmd/storage-rest-server.go55-57

StorageAPI Interface

The StorageAPI interface defines fundamental disk operations:

Operation Category	Key Methods	Purpose
Volume Management	MakeVol, MakeVolBulk, ListVols, StatVol, DeleteVol	Bucket-level operations
File Operations	WriteAll, ReadVersion, ReadXL, AppendFile	Read/write file data
Metadata Operations	WriteMetadata, UpdateMetadata, ReadVersion	Manage xl.meta files
Rename Operations	RenameData, RenameFile, RenamePart	Atomic file movement
Deletion	DeleteVersion, DeleteVersions, Delete	Remove files/versions
Information	DiskInfo, NSScanner	Disk stats and scanning

Sources: cmd/storage-interface.go28-133

---

XL Storage Implementation

The xlStorage struct implements local disk operations. It manages a single disk's file system operations, tracks disk health, and enforces consistency checks.

xlStorage Structure

Sources: cmd/xl-storage.go97-130

Key Implementation Details

Disk Initialization:

• Each disk has a unique format.json file at .minio.sys/format.json
• Contains deployment ID, disk ID, and erasure set configuration
• Validated on every startup and periodically during operation

Direct I/O Support:

• Uses O_DIRECT flag when supported by filesystem (oDirect field)
• Falls back to buffered I/O if not supported
• Checked via checkODirectDiskSupport() method

Disk ID Validation:

• The xlStorageDiskIDCheck wrapper validates disk ID on every operation
• Prevents operations on replaced/stale disks
• Returns errDiskNotFound if disk ID mismatch detected

Sources: cmd/xl-storage.go217-364 cmd/xl-storage.go455-487 cmd/xl-storage-disk-id-check.go84-101

---

Disk I/O Operations

Write Operations

MinIO implements several write strategies based on object size and operation type:

Write Operation Flow:

Sources: cmd/xl-storage.go56-73 cmd/xl-storage.go1686-1838

Small File Optimization

For objects smaller than smallFileThreshold (128 KiB), data is stored inline within the xl.meta file:

smallFileThreshold = 128 * humanize.KiByte

This optimization reduces disk I/O by eliminating separate data file reads for small objects.

Sources: cmd/xl-storage.go56-73

WriteAll Method

The WriteAll() method handles complete file writes:

1. Temporary File Creation: Files are written to .minio.sys/tmp/<uuid> first
2. Data Writing: Uses O_DIRECT for large writes when available
3. Metadata Creation: Generates xl.meta with checksums and versioning info
4. Atomic Rename: Moves from temp location to final path atomically

Sources: cmd/xl-storage.go1686-1838

RenameData Method

The RenameData() method performs atomic data movement with metadata updates:

1. Validates source files exist
2. Creates destination directory structure
3. Writes new xl.meta with updated information
4. Renames data files atomically
5. Cleans up old data on success

Sources: cmd/xl-storage.go1840-2028

Read Operations

Read Operation Types:

Sources: cmd/xl-storage.go1455-1577 cmd/xl-storage.go1579-1625 cmd/xl-storage.go1288-1354

ReadVersion Method

Reads a specific object version's metadata and optionally inline data:

1. Reads xl.meta file from disk
2. Parses metadata using getFileInfo()
3. Returns FileInfo with version-specific metadata
4. Includes inline data if opts.ReadData is true

Sources: cmd/xl-storage.go1455-1577

ReadXL Method

Optimized method for reading the latest version metadata:

• Returns RawFileInfo (raw bytes) instead of parsed FileInfo
• Allows deferred parsing for performance
• Used extensively in distributed operations

Sources: cmd/xl-storage.go1579-1625

Delete Operations

Delete Operation Strategies:

Method	Use Case	Behavior
DeleteVersion()	Delete specific version	Removes version from xl.meta, deletes data if last version
DeleteVersions()	Batch delete	Processes multiple versions efficiently
Delete()	Recursive delete	Moves to trash, optionally immediate purge

Trash Mechanism:

Files are moved to .minio.sys/tmp/.trash/<uuid> before deletion:

• Enables recovery from accidental deletes
• Background cleanup via cleanupTrashImmediateCallers()
• Immediate purge when disk is nearly full

Sources: cmd/xl-storage.go1186-1201 cmd/xl-storage.go1203-1286 cmd/xl-storage.go1232-1286

---

xl.meta Metadata Format

The xl.meta file is MinIO's core metadata format. Each object version has an xl.meta file containing all metadata, checksums, and optionally inline data.

File Location

<bucket>/<object>/xl.meta

For versioned objects:

<bucket>/<object>/xl.meta  (contains all versions)

Sources: cmd/xl-storage.go68-72

xl.meta Structure

xl.meta Format V2:

Sources: cmd/storage-datatypes.go57-154 cmd/xl-storage.go490-527

FileInfo Structure

The FileInfo struct represents parsed xl.meta data:

Key Fields:

• Erasure: Contains DataBlocks, ParityBlocks, BlockSize, Distribution
• Parts: Multipart upload part information with checksums
• Data: Inline data for objects < 128 KiB
• Metadata: User-defined and system metadata (Content-Type, encryption, etc.)

Sources: cmd/storage-datatypes.go94-154

Metadata Operations

Writing Metadata:

1. Serializes FileInfo to msgpack format
2. Writes to <path>/xl.meta
3. Uses atomic write + rename
4. Updates version list in existing xl.meta

Updating Metadata:

• Reads existing xl.meta
• Updates specific version's metadata
• Preserves other versions
• Atomic replacement

Sources: cmd/xl-storage.go2030-2104 cmd/xl-storage.go2140-2227

---

Remote Storage Access

For distributed deployments, MinIO uses a storage REST protocol to access disks on remote nodes.

Storage REST Architecture

Sources: cmd/storage-rest-client.go162-169 cmd/storage-rest-server.go55-57

Grid RPC Handlers

MinIO uses the Grid framework for high-performance RPC:

Handler	Type	Purpose
storageReadVersionRPC	Single	Read version metadata
storageReadXLRPC	Single	Read raw xl.meta
storageWriteAllRPC	Single	Write complete file
storageRenameDataRPC	Single	Rename with metadata
storageDeleteVersionRPC	Single	Delete version
storageListDirRPC	Stream	List directory contents
storageNSScannerRPC	Stream	Scan namespace for usage

Sources: cmd/storage-rest-server.go60-77

Remote Operation Flow

Example: Remote ReadVersion

Sources: cmd/storage-rest-client.go333-351 cmd/storage-rest-server.go236-245

Disk ID Validation

Every remote operation includes disk ID validation:

1. Client sends diskID in RPC parameters
2. Server calls checkID(diskID)
3. Compares against stored disk ID from format.json
4. Returns errDiskNotFound if mismatch
5. Prevents operations on replaced/stale disks

Sources: cmd/storage-rest-server.go211-228

---

Disk Health and Monitoring

Health Tracking

The xlStorageDiskIDCheck wrapper tracks disk health and metrics:

Sources: cmd/xl-storage-disk-id-check.go84-101 cmd/xl-storage-disk-id-check.go103-127

DiskInfo Method

Provides comprehensive disk information:

Sources: cmd/storage-datatypes.go64-84

Metrics Collection

Tracks performance per operation type:

Operation Types Tracked:

• storageMetricReadVersion
• storageMetricWriteAll
• storageMetricDeleteVersion
• storageMetricRenameData
• ~30 other operation types

Sources: cmd/xl-storage-disk-id-check.go45-81 cmd/storage-datatypes.go85-92

Disk Healing Tracker

Each disk maintains a healing tracker file at .minio.sys/buckets/.healing.bin:

Sources: cmd/xl-storage.go431-450

---

Summary

The storage layer provides a clean abstraction over disk operations with the following key characteristics:

Design Principles:

• Abstraction: StorageAPI interface supports both local and remote disks
• Consistency: Disk ID validation prevents stale disk operations
• Performance: Direct I/O, inline data for small objects, caching
• Reliability: Atomic operations, trash mechanism, health monitoring
• Observability: Comprehensive metrics per operation type

Key Components:

• xlStorage: Local disk implementation
• xlStorageDiskIDCheck: Validation and metrics wrapper
• storageRESTClient/Server: Remote disk access via Grid RPC
• xl.meta: Versioned metadata format with inline data support

Integration Points:

• Erasure coding layer (see Erasure Coding Engine) uses StorageAPI for data/parity shard operations
• Metadata caching (see Metadata Management) builds on top of ReadXL() and ReadVersion()
• Healing system uses disk health metrics and the healing tracker