Monitoring and Metrics

Relevant source files

• cmd/bucket-replication-stats.go
• cmd/bucket-stats.go
• cmd/bucket-stats_gen.go
• cmd/bucket-stats_gen_test.go
• cmd/http-stats.go
• cmd/last-minute.go
• cmd/last-minute_gen.go
• cmd/last-minute_gen_test.go
• cmd/metrics-router.go
• cmd/metrics-v2.go
• cmd/metrics-v2_test.go
• cmd/metrics-v3-api.go
• cmd/metrics-v3-types.go
• cmd/metrics.go
• docs/metrics/prometheus/list.md
• internal/http/request-recorder.go
• internal/http/response-recorder.go

This document describes MinIO's comprehensive monitoring and metrics system, which exposes operational telemetry through Prometheus-compatible endpoints. The system collects and exports metrics for cluster health, storage capacity, replication, API performance, and system resource utilization.

For information about audit logging, see Audit Logging. For configuration of metrics endpoints, see Configuration System.

Overview

MinIO provides multiple generations of metrics APIs that expose operational data in Prometheus format:

• Legacy Metrics (/prometheus/metrics) - Original metrics endpoint
• Metrics v2 (/v2/metrics/*) - Categorized metrics with cluster, bucket, node, and resource scopes
• Metrics v3 (/metrics/v3/*) - Hierarchical metrics with advanced filtering and dynamic bucket selection

All metrics are exported in Prometheus text exposition format and can be scraped by Prometheus servers or compatible monitoring systems. The metrics system integrates with the request processing pipeline to capture real-time operational data.

Sources: cmd/metrics-router.go28-37 docs/metrics/prometheus/list.md1-15

Architecture Overview

The metrics system collects data from various global state objects and exposes it through multiple endpoints. Each endpoint serves a specific scope of metrics, from cluster-wide to per-bucket granularity.

Sources: cmd/metrics-router.go53-75 cmd/metrics-v2.go49-125 cmd/metrics.go74-111

Metrics Versions and Endpoints

Endpoint Routing

MinIO registers metrics endpoints under the reserved bucket path prefix:

Endpoint	Version	Scope	Handler
/minio/prometheus/metrics	Legacy	Node	metricsHandler()
/minio/v2/metrics/cluster	v2	Cluster	metricsServerHandler()
/minio/v2/metrics/bucket	v2	Bucket	metricsBucketHandler()
/minio/v2/metrics/node	v2	Node	metricsNodeHandler()
/minio/v2/metrics/resource	v2	Resource	metricsResourceHandler()
/minio/metrics/v3/*	v3	Hierarchical	metricsV3Server

Authentication: By default, metrics endpoints require JWT authentication (MINIO_PROMETHEUS_AUTH_TYPE=jwt). Set MINIO_PROMETHEUS_AUTH_TYPE=public to disable authentication for Prometheus scraping.

Sources: cmd/metrics-router.go28-75 cmd/metrics-router.go40-50

Metrics v2 Architecture

Metrics v2 organizes metrics into groups with caching and dependency management. Each group specifies dependencies (e.g., dependGlobalObjectAPI) and a cache interval.

Sources: cmd/metrics-v2.go61-125 cmd/metrics-v2.go336-424

Metrics v3 Architecture

Metrics v3 introduces a hierarchical path-based structure with advanced features:

The MetricsGroup type implements the prometheus.Collector interface and manages metric loading, caching, and transformation to Prometheus format.

Sources: cmd/metrics-v3-types.go350-423 cmd/metrics-v3-types.go108-140

Metric Namespaces and Types

Metric Namespaces

MinIO uses hierarchical namespaces to organize metrics:

Sources: cmd/metrics-v2.go128-178

Metric Types

MinIO implements three primary metric types aligned with Prometheus conventions:

Type	Description	Example
Counter	Monotonically increasing value	minio_s3_requests_total
Gauge	Value that can increase or decrease	minio_cluster_capacity_raw_free_bytes
Histogram	Distribution of values across buckets	minio_s3_requests_ttfb_seconds_distribution

Sources: cmd/metrics-v2.go307-314 cmd/metrics-v3-types.go66-105

Key Metric Categories

HTTP and API Metrics

HTTP metrics track API request counts, errors, latency (TTFB - Time To First Byte), and traffic. The ResponseRecorder captures timing information, and HTTPStats.updateStats() aggregates counters per API operation.

TTFB Histogram: The system maintains histogram metrics for TTFB using Prometheus histogram vectors with configurable buckets (e.g., 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 seconds).

Sources: cmd/http-stats.go370-453 cmd/metrics.go186-238 internal/http/response-recorder.go33-79

Storage and Capacity Metrics

Storage metrics provide visibility into raw and usable capacity, drive health, and I/O performance. Drive-level metrics include latency measurements, error counts (timeout, I/O errors), and availability status.

Sources: cmd/metrics.go369-481 docs/metrics/prometheus/list.md23-46 docs/metrics/prometheus/list.md194-209

Replication Metrics

Replication metrics track the health and performance of both bucket replication and site replication. Key metrics include:

• Completed operations: Bytes and object counts successfully replicated
• Failed operations: Tracked per time window (last minute, last hour, total)
• Queue depth: Objects/bytes waiting for replication
• Worker utilization: Current, average, and maximum active workers
• Transfer rates: Separate tracking for large (>128MiB) and small objects
• Latency histograms: Replication latency distribution by object size

Sources: cmd/bucket-replication-stats.go39-238 cmd/bucket-stats.go153-231 docs/metrics/prometheus/list.md79-135

System Resource Metrics

System metrics leverage the procfs library to read Linux process statistics and Go runtime metrics to track resource utilization.

Sources: docs/metrics/prometheus/list.md246-273 cmd/metrics-v2.go44

Metrics Collection and Caching

MetricsGroup Caching

Each MetricsGroupV2 includes a cachevalue.Cache that stores computed metrics with a configurable TTL. The cache prevents redundant computation during frequent Prometheus scrapes.

Cache Behavior:

• Cache is invalidated after the configured interval
• The RegisterRead() function wraps the metrics computation with caching
• Dependencies (e.g., dependGlobalObjectAPI) are checked before loading metrics
• If a dependency is not ready, empty metrics are returned

Sources: cmd/metrics-v2.go336-424 cmd/metrics-v2.go441-455

Dependency Management

Metrics groups declare dependencies on global system components:

This dependency system ensures metrics are only collected when the required subsystems are initialized and available.

Sources: cmd/metrics-v2.go342-424

Network and Traffic Metrics

Network metrics distinguish between:

• Internode traffic: Communication between MinIO cluster nodes
• S3 traffic: External client traffic
• Per-bucket traffic: Traffic attributed to specific buckets

Atomic operations (atomic.AddUint64) ensure thread-safe counter updates in high-concurrency scenarios.

Sources: cmd/http-stats.go30-93 cmd/http-stats.go207-300 cmd/metrics.go240-282

Healing and Data Integrity Metrics

Healing metrics track the background healing process that detects and repairs data corruption, missing shards, and metadata inconsistencies. The metrics include time since last activity and counts broken down by object type.

Sources: cmd/metrics.go133-184 docs/metrics/prometheus/list.md137-143

Scanner and Lifecycle Metrics

The data scanner metrics provide visibility into the continuous background process that traverses all objects for lifecycle management, usage calculation, and anomaly detection.

Sources: docs/metrics/prometheus/list.md220-243 docs/metrics/prometheus/list.md262-273

Authentication and Authorization

Metrics Endpoint Authentication

By default, metrics endpoints require authentication and authorization:

1. JWT Authentication: Request must include valid bearer token
2. IAM Policy: User must have PrometheusAdminAction permission
3. Public Mode: Set MINIO_PROMETHEUS_AUTH_TYPE=public to disable authentication

The metricsRequestAuthenticate() function validates JWT claims and checks IAM policies before serving metrics.

Sources: cmd/metrics-router.go52-75 cmd/metrics.go532-573

Time-Series Aggregation

Last Minute/Hour Tracking

MinIO uses circular buffer structures to track metrics over time windows. The lastMinuteLatency maintains 60 one-second buckets, while ReplicationLastHour maintains 60 one-minute buckets. The forwardTo() method automatically clears expired buckets as time advances.

Object Size Histogram: The LastMinuteHistogram categorizes operations by object size (< 1KiB, < 1MiB, < 10MiB, < 100MiB, < 1GiB, > 1GiB) for latency analysis.

Sources: cmd/last-minute.go28-209 cmd/bucket-stats.go32-58

Prometheus Integration

Collector Implementation

MinIO implements the prometheus.Collector interface for each metrics scope. The Describe() method declares metric descriptors, while Collect() populates current values.

Metric Registration:

• Legacy collector registered in init() via prometheus.MustRegister()
• v2 collectors registered per-request in handler functions
• Metrics encoded using expfmt.Encoder for Prometheus text format

Sources: cmd/metrics.go74-111 cmd/metrics.go483-525 cmd/metrics-v2.go49-55

Metrics v3 Dynamic Collection

Metrics v3 supports dynamic bucket filtering:

The v3 API allows clients to specify which buckets to include in metrics via query parameters, reducing response size for large deployments.

Sources: cmd/metrics-v3-types.go350-423 cmd/metrics-v3-types.go475-483

Configuration

Environment Variables

Variable	Default	Description
MINIO_PROMETHEUS_AUTH_TYPE	jwt	Authentication type: jwt or public
MINIO_PROMETHEUS_OPEN_METRICS	-	Enable OpenMetrics format

Sources: cmd/metrics-router.go40-43

Prometheus Scrape Configuration

Example Prometheus configuration for scraping MinIO metrics:

For public access mode:

Sources: docs/metrics/prometheus/list.md6-14