Healing System

Relevant source files

• cmd/admin-heal-ops.go
• cmd/background-heal-ops.go
• cmd/background-newdisks-heal-ops.go
• cmd/bucket-encryption-handlers.go
• cmd/bucket-lifecycle-handlers.go
• cmd/bucket-lifecycle.go
• cmd/data-scanner.go
• cmd/endpoint-ellipses.go
• cmd/endpoint-ellipses_test.go
• cmd/endpoint.go
• cmd/endpoint_test.go
• cmd/erasure-sets_test.go
• cmd/event-notification.go
• cmd/format-erasure.go
• cmd/format-erasure_test.go
• cmd/global-heal.go
• cmd/net.go
• cmd/net_test.go
• cmd/object-handlers-common.go
• cmd/prepare-storage.go
• internal/bucket/lifecycle/action_string.go
• internal/bucket/lifecycle/and.go
• internal/bucket/lifecycle/delmarker-expiration.go
• internal/bucket/lifecycle/delmarker-expiration_test.go
• internal/bucket/lifecycle/filter.go
• internal/bucket/lifecycle/filter_test.go
• internal/bucket/lifecycle/lifecycle.go
• internal/bucket/lifecycle/lifecycle_test.go
• internal/bucket/lifecycle/noncurrentversion.go
• internal/bucket/lifecycle/noncurrentversion_test.go
• internal/bucket/lifecycle/rule.go
• internal/bucket/lifecycle/rule_test.go
• internal/event/name.go
• internal/event/name_test.go

Purpose and Scope

The Healing System is MinIO's automated mechanism for detecting and repairing data inconsistencies across distributed erasure-coded storage. It ensures data durability by:

• Detecting and repairing bitrot (data corruption) on drives
• Reconstructing missing data shards using erasure coding
• Restoring objects on newly added or reformatted drives
• Fixing metadata inconsistencies across erasure sets

This document covers the healing infrastructure, background healing processes, and manual healing operations. For information about the data scanner that triggers healing, see Data Scanner and Usage Calculation. For storage initialization and disk formatting, see Storage Initialization and Formatting.

Sources: cmd/global-heal.go1-595 cmd/data-scanner.go1-100 cmd/admin-heal-ops.go1-100

---

Architecture Overview

The healing system consists of several interconnected components that work together to maintain data integrity:

Healing System Architecture

Sources: cmd/global-heal.go43-75 cmd/admin-heal-ops.go92-103 cmd/background-newdisks-heal-ops.go40-101 cmd/background-heal-ops.go51-56

---

Healing Components

Background Healing Sequence

The background healing sequence (bgHealingUUID) is a special, long-running heal operation that continuously monitors and repairs the cluster:

Background Healing Sequence Structure

The background heal sequence is initialized with a special UUID (0000-0000-0000-0000) and runs continuously:

• Token: bgHealingUUID constant identifies this as the background sequence
• Settings: healDeleteDangling = true removes objects without read quorum
• Reserved Bucket: Uses minioReservedBucket to avoid conflicts with user operations
• Never Ends: The sequence's hasEnded() method always returns false for background healing

Sources: cmd/global-heal.go43-75 cmd/admin-heal-ops.go589-598

Heal Workers and Task Queue

Healing operations are distributed across multiple worker goroutines for parallel execution:

Heal Worker Pool and Task Distribution

Worker count is determined by:

• Default: runtime.GOMAXPROCS(0) / 2
• Environment override: _MINIO_HEAL_WORKERS
• Minimum: 4 workers if calculated value is 0

Sources: cmd/background-heal-ops.go51-175 cmd/admin-heal-ops.go407-503

Healing Tracker for New Disks

When a disk is reformatted or newly added, the healingTracker maintains persistent state to resume interrupted healing:

Healing Tracker Persistence Structure

The tracker uses MessagePack encoding for efficient serialization and includes:

• Identity: Validates disk UUID to prevent healing wrong disk
• Progress: Tracks items/bytes healed, failed, and skipped
• Position: Stores current bucket and object for resumption
• Resume Points: Saves progress at bucket boundaries for consistent restart

Sources: cmd/background-newdisks-heal-ops.go40-371

Scan Modes

MinIO supports two healing scan modes with different bitrot detection strategies:

Scan Mode	Description	Bitrot Detection	Use Case
HealNormalScan	Fast metadata validation	Metadata checksums only	Regular background healing
HealDeepScan	Full data verification	Read and verify all data blocks	Periodic deep scans, manual healing

Scan Mode Selection Logic:

The data scanner automatically transitions between modes based on configuration and cycle count:

Scan Mode Decision Flow

Sources: cmd/data-scanner.go89-107 cmd/data-scanner.go949-973

---

Healing Workflow

Detection and Queuing

Objects are selected for healing through multiple detection mechanisms:

Object Healing Detection Flow

The data scanner uses probabilistic selection for healing:

• Heal Probability: 1 in healObjectSelectProb (1024) objects selected per cycle
• Hash-Based Selection: thisHash.modAlt(cycle/divisor, probability) ensures deterministic selection
• Abandoned Children: Objects missing from current scan but present in cache are checked

Sources: cmd/data-scanner.go395-892 cmd/data-scanner.go506-507

Erasure Set Healing Execution

When healing a complete erasure set (e.g., after adding a new disk), the process follows a structured workflow:

Erasure Set Healing Workflow

Key aspects of the healing process:

1. Worker Calculation: numHealers = min(numCores/4, NRRequests/4) with minimum of 4 and override via globalHealConfig.GetWorkers()

2. Lifecycle Filtering: Objects already expired by ILM rules are skipped to avoid unnecessary healing

3. Version Filtering: Objects uploaded after healing started are skipped (based on tracker.Started timestamp)

4. Progress Tracking: Results are collected via channel and periodically persisted to healingTracker

Sources: cmd/global-heal.go152-575 cmd/global-heal.go388-506

Bitrot Detection and Repair

During healing operations, MinIO verifies data integrity and repairs corruption:

Bitrot Detection and Repair Process

Healing modes and their verification depth:

• Normal Scan (heal.bitrot = false): Validates metadata checksums only, fast but less thorough
• Deep Scan (heal.bitrot = true): Reads all data blocks and verifies HighwayHash checksums, slower but comprehensive

The healDeleteDangling flag (set to true) removes objects that cannot achieve read quorum, preventing orphaned metadata from accumulating.

Sources: cmd/data-scanner.go952-973 cmd/global-heal.go422-463

---

Integration with Data Scanner

The data scanner is the primary driver of background healing, integrating healing checks into its regular traversal:

Data Scanner Healing Integration

The scanner maintains healing state across restarts:

• Background Heal Info: Stored at backgroundHealInfoPath in .minio.sys/config
• Bitrot Cycle Tracking: Records when deep scans started and last occurred
• Cycle Counter: Incremented each scan to rotate through different object selections

Sources: cmd/data-scanner.go152-232 cmd/data-scanner.go676-839

---

Heal Sequence Management

Heal Sequence State Machine

Each heal sequence progresses through distinct states:

Heal Sequence State Transitions

State durations and timeouts:

• Keep Duration: keepHealSeqStateDuration = 10 minutes - results retained after completion
• Idle Timeout: healUnconsumedTimeout = 24 hours - aborts if results not consumed
• Result Buffer: maxUnconsumedHealResultItems = 1000 - blocks new results when full

Sources: cmd/admin-heal-ops.go36-64 cmd/admin-heal-ops.go589-603

Heal Sequence Registry

The global heal state manages all active and completed heal sequences:

Global Heal State Structure

The heal state prevents conflicting operations through path validation:

• Overlap Detection: New sequences cannot start if their path overlaps with existing sequences
• Force Start: Admin can force-stop existing sequence to start new one on same path
• Background Exception: The bgHealingUUID sequence never ends and can coexist with manual sequences

Sources: cmd/admin-heal-ops.go92-355

---

Automatic Disk Healing

New Disk Detection and Healing

MinIO automatically detects and heals newly added or reformatted disks:

Automatic Disk Healing Workflow

The monitoring loop runs continuously when _MINIO_AUTO_DRIVE_HEALING=on:

1. Detection Interval: defaultMonitorNewDiskInterval = 10 seconds
2. Healing Lock: Prevents parallel healing of same erasure set using distributed lock
3. Retry Logic: Failed healings persist tracker for retry on next cycle
4. Validation: Checks that healed disk count doesn't equal total endpoints (prevents healing fresh setups)

Sources: cmd/background-newdisks-heal-ops.go373-535 cmd/background-newdisks-heal-ops.go389-409

Distributed Lock Coordination

Healing operations use distributed locks to prevent conflicts:

Distributed Lock for Healing Coordination

The dynamic timeout prevents healing conflicts:

• Base Timeout: 30 seconds
• Additional Wait: 10 seconds per attempt
• Purpose: Ensures only one node heals an erasure set at a time

Sources: cmd/background-newdisks-heal-ops.go411-442

---

Configuration and Tuning

Healing Configuration Options

MinIO provides several configuration parameters for healing behavior:

Configuration	Environment Variable	Default	Description
Heal Workers	_MINIO_HEAL_WORKERS	runtime.GOMAXPROCS(0) / 2	Number of parallel heal workers
Auto Drive Healing	_MINIO_AUTO_DRIVE_HEALING	on	Enable automatic new disk healing
Bitrot Scan Cycle	heal.bitrotscan	(varies)	Frequency of deep bitrot scans
Max IO	heal.max_io	100	Max concurrent operations during healing
Max Wait	heal.max_wait	15 seconds	Max wait for low IO before proceeding

Heal Configuration Structure:

Healing Configuration Hierarchy

Sources: cmd/data-scanner.go62-72 cmd/background-heal-ops.go97-100 cmd/background-heal-ops.go156-170

Wait for Low IO Strategy

Healing operations throttle themselves based on cluster load:

Wait for Low IO Logic

The throttling mechanism:

• Active Listeners Excluded: Bucket notifications and HTTP trace subscribers don't count toward maxIO
• Tick Interval: Checks every 100ms whether IO has dropped below threshold
• Max Wait Enforcement: After maxWait duration, healing proceeds regardless of IO level

Sources: cmd/background-heal-ops.go57-100

---

API and Monitoring

Admin Heal API Endpoints

Administrators can start, monitor, and stop healing operations via the Admin API:

Admin Heal API Structure

The heal result items contain detailed before/after drive states:

Sources: cmd/admin-heal-ops.go287-355 cmd/admin-heal-ops.go357-405

Background Heal Status

The background healing status is exposed through the storage info API:

Background Heal Status Response Structure

The status includes:

• Set-Level Information: Each erasure set shows which disks are healing
• Scanned Items Count: Total objects examined by background scanner
• Storage Class Parity: EC configuration for different storage classes
• Healing Endpoints: List of disk endpoints currently being healed

Sources: cmd/global-heal.go78-141

---

Summary

The MinIO Healing System provides comprehensive automated and manual data repair capabilities:

Key Components:

• Background healing through data scanner integration
• Manual healing via Admin API with configurable options
• Automatic new disk detection and healing
• Persistent healing state for resumable operations

Healing Strategies:

• Probabilistic object selection for continuous background healing
• Deep vs. Normal scan modes for bitrot detection
• Quorum-based reconstruction using Reed-Solomon erasure coding

Operational Controls:

• Worker pool sizing for parallel healing
• IO throttling to minimize impact on production workload
• Distributed locks to prevent conflicting healing operations
• Comprehensive metrics and status reporting

The healing system ensures data durability in MinIO's distributed erasure-coded architecture by continuously monitoring for and repairing data inconsistencies, corruption, and missing shards across the cluster.

Sources: cmd/global-heal.go1-595 cmd/data-scanner.go1-893 cmd/admin-heal-ops.go1-665 cmd/background-newdisks-heal-ops.go1-535