Metadata Management

Relevant source files

• cmd/metacache-bucket.go
• cmd/metacache-bucket_test.go
• cmd/metacache-entries.go
• cmd/metacache-manager.go
• cmd/metacache-server-pool.go
• cmd/metacache-set.go
• cmd/metacache-stream.go
• cmd/metacache-stream_test.go
• cmd/metacache-walk.go
• cmd/metacache.go
• cmd/metacache_gen.go
• cmd/metacache_test.go
• cmd/naughty-disk_test.go
• cmd/object-api-listobjects_test.go
• cmd/storage-datatypes.go
• cmd/storage-datatypes_gen.go
• cmd/storage-datatypes_gen_test.go
• cmd/storage-errors.go
• cmd/storage-interface.go
• cmd/storage-rest-client.go
• cmd/storage-rest-common.go
• cmd/storage-rest-server.go
• cmd/storage-rest_test.go
• cmd/testdata/xl-meta-consist.zip
• cmd/xl-storage-disk-id-check.go
• cmd/xl-storage-format-utils.go
• cmd/xl-storage-format-utils_test.go
• cmd/xl-storage-format-v2-legacy.go
• cmd/xl-storage-format-v2.go
• cmd/xl-storage-format-v2_test.go
• cmd/xl-storage-format_test.go
• cmd/xl-storage-free-version_test.go
• cmd/xl-storage.go
• cmd/xl-storage_test.go

Overview

This document describes MinIO's metadata management system, which ensures consistent and efficient storage and retrieval of object metadata across a distributed erasure-coded storage cluster. It covers the xl.meta file format (xlMetaV2), version management, inline data storage, the metacache system for list optimization, and metadata quorum resolution. For information about the storage layer that persists these structures, see Storage Layer Architecture. For details about the erasure coding that distributes metadata across disks, see Erasure Coding Engine.

Sources: cmd/xl-storage.go1-2358 cmd/xl-storage-format-v2.go1-2522 cmd/metacache-set.go1-1319

xl.meta File Format (xlMetaV2)

MinIO stores all object metadata in files named xl.meta located alongside object data. The xlMetaV2 format is a versioned, binary format that supports multiple object versions, delete markers, and inline data storage.

File Structure

Sources: cmd/xl-storage-format-v2.go42-103 cmd/xl-storage-format-v2.go148-187

The file begins with an 8-byte header containing the magic bytes 'X', 'L', '2', ' ' followed by a 4-byte version number encoded as little-endian [major:2, minor:2]. The current version is 1.3. After the header, the payload is encoded using MessagePack (msgpack) format.

Component	Description	Location
xlHeader	Magic bytes identifying XL format	First 4 bytes
xlVersionCurrent	Major and minor version (1.3)	Bytes 4-8
Payload	Msgpack-encoded metadata	Bytes 8+

Sources: cmd/xl-storage-format-v2.go42-71 cmd/xl-storage-format-v2.go219-241

Version Journal Structure

The core of xlMetaV2 is the version journal - an array that stores all versions of an object. Each entry represents either an object version, a delete marker, or a legacy object from the xlV1 format.

Sources: cmd/xl-storage-format-v2.go105-187 cmd/xl-storage-format-v2.go149-175

The xlMetaV2Version struct is a discriminated union where the Type field determines which of the three optional fields is populated:

• ObjectType: Contains xlMetaV2Object with full object metadata including erasure coding parameters, part information, size, modification time, and user/system metadata.
• DeleteType: Contains xlMetaV2DeleteMarker representing a delete marker with version ID and modification time.
• LegacyType: Contains xlMetaV1Object for backward compatibility with the previous format.

Sources: cmd/xl-storage-format-v2.go181-187

Version Header and Flags

Each version can be efficiently represented by a compact header for operations that don't require full metadata:

Sources: cmd/xl-storage-format-v2.go248-267 cmd/xl-storage-format-v2.go189-217

The flags field provides quick information about version characteristics:

• xlFlagFreeVersion: Marks a version as a "free version" - a special type used to track tiered content of deleted/overwritten versions visible only to the scanner
• xlFlagUsesDataDir: Indicates the version stores data in a separate directory (normal for most objects)
• xlFlagInlineData: Indicates data is stored directly in the xl.meta file (optimization for small files)

Sources: cmd/xl-storage-format-v2.go189-217

Inline Data Storage

For small files (typically under 128 KiB), MinIO can store the actual object data directly within the xl.meta file instead of creating separate data files. This reduces the number of disk operations and improves performance for small objects.

Sources: cmd/xl-storage.go59-73 cmd/xl-storage-format-v2.go384-401

The smallFileThreshold constant defines the size boundary (128 KiB optimized for NVMe/SSDs). When data is inlined, it's stored in the data map of the xlMetaV2 structure, keyed by version ID. The flag xlFlagInlineData is set in the version header to indicate this optimization.

Sources: cmd/xl-storage.go59-73

Version Management

Version Types and Lifecycle

MinIO maintains a complete version history for objects in versioned buckets. The version journal in xl.meta stores versions in most-recent-first order.

Sources: cmd/xl-storage-format-v2.go73-103 cmd/xl-storage-format-v2.go384-387

The version array maintains three types of entries:

1. Object versions: Created by PUT, CopyObject, or multipart uploads
2. Delete markers: Created by DELETE operations in versioned buckets
3. Free versions: Special delete markers with MetaSys entries that track transitioned content requiring async cleanup

Sources: cmd/xl-storage-format-v2.go85-88

Version Operations

The xlMetaV2 structure provides methods for managing versions:

Operation	Method	Description
Add Version	AddVersion()	Appends a new version to the journal
Delete Version	DeleteVersion()	Removes or marks a version as deleted
Get Latest	ToFileInfo()	Retrieves the most recent version
List Versions	ListVersions()	Returns all non-free versions
Free Versions	FreeVersions	Separate array for cleanup tracking

Sources: cmd/xl-storage-format-v2.go445-461

Version Signature and Matching

Each version has a signature used to verify consistency across disks in the erasure set:

Sources: cmd/xl-storage-format-v2.go405-416 cmd/xl-storage-format-v2.go269-285

The signature is a 4-byte hash that uniquely identifies a version's content and metadata. During read operations, signatures are compared across disks to detect inconsistencies. The matchesNotStrict() method allows some flexibility in matching (e.g., ignoring signature differences but requiring version ID and type to match) which is useful during healing operations.

Sources: cmd/xl-storage-format-v2.go269-278

Metadata Quorum Resolution

In an erasure-coded environment, xl.meta files exist on multiple disks within an erasure set. MinIO must reconcile potentially divergent metadata copies to determine the authoritative version.

Quorum Read Process

Diagram: Metadata Quorum Resolution Flow

Sources: cmd/erasure-object.go1-66 cmd/xl-storage-format-v2.go248-285 cmd/xl-storage-format-v2.go357-382 cmd/xl-storage-format-v2.go445-461 cmd/metacache-entries.go69-157

The quorum resolution algorithm:

1. Parallel read: Read xl.meta from all available disks in the erasure set (typically 8-16 disks)
2. Header extraction: Extract xlMetaV2VersionHeader from each xl.meta containing version ID, signature, modification time, and flags
3. Signature grouping: Group versions by signature to identify matching copies
4. Quorum check: Require at least N/2 disks to return the same version (read quorum)
5. Version selection: Choose the version with the most matching signatures
6. Healing: If mismatches are detected, trigger background healing to repair inconsistent copies

Sources: cmd/xl-storage-format-v2.go258-267 cmd/xl-storage-format-v2.go269-285

Version Header Matching

The xlMetaV2VersionHeader provides a compact representation for comparison:

Sources: cmd/xl-storage-format-v2.go269-321

Matching modes:

• Strict matching: Requires identical version ID, type, signature, and erasure coding parameters (EcN, EcM)
• Non-strict matching: Used during healing operations, allows signature differences if version ID, type, and modtime match
• Tiebreaking: When multiple valid versions exist, sortsBefore() selects based on newest modtime, then consistent ordering by signature and version ID

Sources: cmd/xl-storage-format-v2.go269-321

Read Operations with Quorum

Diagram: Quorum-Based Read Methods

Sources: cmd/xl-storage.go489-527 cmd/xl-storage.go1288-1361

Key reading methods:

• readMetadata(): Reads xl.meta with a context deadline, returns raw bytes from a single disk
• readMetadataWithDMTime(): Additionally returns the file's modification time for cache validation
• ReadVersion(): Decodes xl.meta and returns a specific version as FileInfo from a single disk
• ReadXL(): Returns raw xl.meta content with minimal processing from a single disk
• getObjectFileInfo(): Higher-level erasure layer method that reads from multiple disks and applies quorum resolution

Sources: cmd/xl-storage.go522-527 cmd/xl-storage.go489-520

Writing Metadata with Write Quorum

Metadata writes must achieve write quorum (N/2 + 1 disks) to be considered successful:

Diagram: Metadata Write with Quorum

Sources: cmd/xl-storage.go1086-1184 cmd/storage-datatypes.go32-41

Write operations follow these steps:

1. Load existing xl.meta (if it exists) using LoadOrConvert()
2. Modify the version journal (add, update, or delete versions)
3. Encode to MessagePack format
4. Write atomically to each disk (via temporary file and rename)
5. Wait for write quorum (N/2 + 1 disks) before returning success
6. Background healing will eventually replicate to any failed disks

For DeleteVersion(), if the version's DataDir is specified, it also removes the associated data directory after updating xl.meta. The erasure layer ensures that version deletions are coordinated across all disks with appropriate quorum checks.

Sources: cmd/xl-storage.go1086-1184 cmd/storage-rest-server.go435-458

Quorum Requirements

Operation	Read Quorum	Write Quorum	Notes
GetObject	N/2	-	Must read consistent metadata from majority
PutObject	-	N/2 + 1	Must write to majority before success
DeleteObject	N/2	N/2 + 1	Read to find version, write to mark deleted
ListObjects	1 (per disk)	-	Uses metacache, eventual consistency
HeadObject	N/2	-	Same as GetObject for metadata

Sources: cmd/erasure.go1-66 cmd/xl-storage.go1086-1184

Storage-Level Caching

Diagram: Storage Layer Caching Strategy

Sources: cmd/xl-storage.go115-130 cmd/xl-storage.go771-781

The storage layer intentionally avoids caching xl.meta content because:

• Each object's metadata is read infrequently (only during GET/HEAD operations)
• The OS page cache effectively caches frequently accessed files
• Metadata changes must be immediately visible across all reads
• Quorum resolution provides implicit validation across multiple disks
• The metacache system (described below) handles caching for the expensive listing operations

The only cached values at the storage layer are:

• diskInfoCache: Caches disk information for 1 second to avoid repeated syscalls in DiskInfo() method
• Disk ID: Cached from format.json for up to 1 second in GetDiskID() to validate requests

Sources: cmd/xl-storage.go115-117 cmd/xl-storage.go820-897 cmd/xl-storage-disk-id-check.go289-341

Metacache System

The metacache system is a specialized caching layer designed to optimize expensive directory listing operations (ListObjects, ListObjectsV2). It creates compressed, indexed snapshots of directory structures that can be efficiently streamed to clients.

Purpose and Architecture

Sources: cmd/metacache-manager.go27-72 cmd/metacache.go30-75

The metacache system consists of:

• localMetacacheMgr: Global singleton that manages all caches
• metacacheBucket: Per-bucket cache storage, managing multiple cache entries
• metacache: Individual cache entry with lifecycle state (started, success, error)
• Compressed blocks: The actual cached directory listing data stored as objects in .minio.sys/buckets/<bucket>/.metacache/<id>/

Sources: cmd/metacache-manager.go32-55 cmd/metacache-bucket.go26-60

Cache Entry Structure

Each metacache entry stores metadata about a listing operation and the associated cached data:

Sources: cmd/metacache.go30-75 cmd/metacache-set.go336-345

The cache stores data in S2-compressed blocks (Snappy variant) at paths like .minio.sys/buckets/<bucket>/.metacache/<id>/block-N.s2. Each block contains a sequential portion of the directory listing, allowing streaming reads without loading the entire listing into memory.

Sources: cmd/metacache-set.go343-345

Cache Lifecycle

Sources: cmd/metacache.go77-98 cmd/metacache-set.go148-164

Cache lifecycle states:

• scanStateNone: No cache exists or cache has been invalidated
• scanStateStarted: Background scan is actively walking the directory tree and writing blocks
• scanStateSuccess: Scan completed successfully, cache is ready for use
• scanStateError: Scan failed, cache is unusable

A cache entry in the scanStateSuccess state can serve multiple concurrent ListObjects requests by streaming from the stored blocks. The cache remains valid until:

• The bucket receives write operations that might invalidate the cached data
• The cache expires after a period of inactivity
• An administrator forces cache invalidation

Sources: cmd/metacache.go77-98

Cache Creation and Streaming

When a ListObjects request is received, the system determines whether to use an existing cache or create a new one:

Diagram: Cache Decision Flow with Function Names

Sources: cmd/metacache-server-pool.go60-159 cmd/metacache-set.go271-316 cmd/metacache-set.go369-416 cmd/metacache-set.go418-609 cmd/metacache-stream.go94-126

Key operations:

1. Cache lookup: metacacheBucket.findCache() locates a compatible cache based on bucket, prefix, and recursive flag
2. Starting block: listPathOptions.findFirstPart() locates the first cache block containing entries >= the marker
3. Streaming: erasureObjects.streamMetadataParts() reads blocks sequentially, decompressing and filtering entries
4. Filtering: metacacheReader.filter() applies prefix, delimiter, and limit constraints to cached entries
5. Background scan: erasureServerPools.listAndSave() creates new cache by calling xlStorage.WalkDir() on each disk

Sources: cmd/metacache-set.go271-316 cmd/metacache-set.go369-416 cmd/metacache-set.go418-609 cmd/metacache-walk.go73-455

Cache Invalidation and Cleanup

The metacache system must invalidate caches when underlying data changes:

Sources: cmd/metacache-bucket.go146-185 cmd/metacache-bucket.go187-215

Invalidation mechanisms:

• On write operations: Caches with matching or parent prefixes are marked for deletion
• Periodic cleanup: Background process removes old, unused cache entries
• Transient caches: For reserved buckets or during errors, caches are marked transient and not persisted to disk
• Handout timeout: Caches not accessed recently (lastHandout) are candidates for cleanup

Sources: cmd/metacache-bucket.go187-215

Metacache Entry Format

Entries in metacache blocks use a specialized compact format:

Sources: cmd/metacache-entries.go32-60

The metaCacheEntry structure represents:

• Directories: name ends with / and metadata is empty
• Objects: name is the full path and metadata contains xl.meta content
• Object directories: Objects with names ending in / (explicitly created with trailing slash)

The cached field optionally holds a decoded xlMetaV2 structure to avoid repeated decoding. The reusable flag indicates whether the metadata buffer can be reused (single reference) or must be copied.

Sources: cmd/metacache-entries.go32-45

Performance Characteristics

The metacache system provides significant performance benefits:

Operation	Without Cache	With Cache
First ListObjects	O(N) disk walk	O(N) disk walk + write cache
Subsequent ListObjects	O(N) disk walk	O(K) cache read (K = results returned)
Paginated listing	O(N) per page	O(K) per page, resume from marker
Concurrent listings	N × O(N)	N × O(K) + single cache creation

Sources: cmd/metacache-set.go703-767

The streaming design allows handling arbitrarily large directories without excessive memory consumption, as only the current block (typically 4-32 MB compressed) needs to be in memory at once.

Sources: cmd/metacache-stream.go94-126

---

Sources: cmd/xl-storage.go1-2358 cmd/xl-storage-format-v2.go1-2522 cmd/metacache-set.go1-1319 cmd/metacache-walk.go1-455 cmd/metacache-entries.go1-735 cmd/metacache-bucket.go1-277 cmd/metacache-manager.go1-178 cmd/metacache.go1-256 cmd/metacache-stream.go1-764