[Bug] Metrics page shows aggregated data from all clusters instead of filtering by connected cluster Summary #1021

Summary

The metrics page in Attu aggregates Prometheus data from all Milvus clusters sharing the same Prometheus instance, regardless of which cluster the current connection is pointing to. When multiple Milvus clusters report metrics to the same Prometheus, all connections display identical metrics data.

Environment

Steps to Reproduce

1. Deploy two or more Milvus clusters, all scraping into the same Prometheus instance
2. In Attu, create separate connections for each cluster with the same Prometheus URL
3. Navigate to the Metrics tab for connection A
4. Navigate to the Metrics tab for connection B
5. Observe that all metric panels show identical values across both connections

Expected Behavior

Each connection's metrics page should display data only for the connected Milvus cluster, filtered by the cluster's identity label (e.g. app_kubernetes_io_instance).

Actual Behavior

All connections display the same aggregated metrics regardless of which cluster is selected.

For example, the QPS panel shows the total QPS of all clusters combined, not just the connected one. Switching between connections does not change any metric values.

Root Cause

By inspecting the compiled server bundle inside the running Attu container at:

/app/.output/server/_ssr/metric-types-qyt8JsOR.mjs

All 16 PromQL queries contain no instance-level label filters:

sum(rate(milvus_proxy_req_count{status="success"}[5m]))
sum(rate(milvus_proxy_insert_vectors_count[5m]))
histogram_quantile(0.99, ...)
sum(rate(process_cpu_seconds_total[5m]))
sum(process_resident_memory_bytes)
sum(milvus_datacoord_stored_rows_num)
sum(milvus_querynode_entity_num)
milvus_rootcoord_collection_num
sum(milvus_datacoord_segment_num)
sum(increase(milvus_proxy_slow_query_count[1m]))
max(milvus_rootcoord_time_tick_delay)
max(milvus_datanode_consume_tt_lag_ms)
sum(rate(milvus_datanode_flushed_data_size[5m]))

Every query aggregates data from all clusters in Prometheus without any cluster-level filtering.

Milvus Prometheus metrics include the label app_kubernetes_io_instance (set to the Helm release name) which uniquely identifies each cluster:

milvus_build_info{
  app_kubernetes_io_instance="cluster-a",
  namespace="milvus",
  container="proxy",
  ...
}
milvus_build_info{
  app_kubernetes_io_instance="cluster-b",
  namespace="milvus",
  container="proxy",
  ...
}

This label exists on every Milvus metric and is the standard mechanism for distinguishing clusters in a shared Prometheus environment.

The prometheus.server module correctly scopes the Prometheus URL per connection (each connection can have its own Prometheus URL configured), but the PromQL query strings themselves carry no per-cluster filter — making per-connection URL isolation ineffective when multiple clusters share one Prometheus.

Proposed Fix

Inject the current connection's cluster identifier into every PromQL query as a label filter.

Option 1 — Automatically resolve app_kubernetes_io_instance from the cluster (Recommended)

When a connection is established, query the Milvus metrics endpoint (host:9091/metrics) and extract the app_kubernetes_io_instance label value. Store it on the connection record and inject it into all PromQL queries:

-- Before
sum(rate(milvus_proxy_req_count{status="success"}[5m]))

-- After
sum(rate(milvus_proxy_req_count{
  status="success",
  app_kubernetes_io_instance="${instanceName}"
}[5m]))

Option 2 — Add an explicit "Instance Label Value" field to the connection form

Allow users to specify the label value manually in the connection configuration form (similar to how Grafana handles data source variable filtering). This gives operators flexibility when the label name or value differs from the default:

Connection form:
  Address:              10.x.x.x:19530
  Prometheus URL:       http://prometheus:9090
  Instance Label:       my-cluster-name   ← new optional field

Workaround

Use separate Prometheus endpoints per cluster. For example, if using VictoriaMetrics with multi-tenancy, assign each cluster its own tenant ID and configure the connection Prometheus URL accordingly:

Each tenant only returns metrics for the clusters writing to it, providing natural isolation.

Additional Context

• Verified by inspecting the compiled server bundle inside the running Attu container
• All 16 metric panels are affected: QPS, Insert Rate, Search Latency P99, Search Core Latency P99, Mutation Latency P99, CPU Usage, Memory Usage, Disk Usage, Entity Count, Loaded Entities, Collection Count, Segment Count, Slow Queries, Time Tick Delay, DataNode Lag, Flush Throughput
• The prometheus.server module correctly scopes the Prometheus URL per connection, but the query strings in metric-types carry no per-cluster label filter
• app_kubernetes_io_instance is set automatically by the Milvus Helm chart and is present on all Milvus metrics, making it the most reliable filter key