Runtime hot reload leaves duplicate Prometheus metric samples (OpenCensus meter lifecycle) #10535

In what area(s)?

/area runtime

What version of Dapr?

Observed in Kubernetes with Dapr 1.18.3.

Source inspection shows the same meter lifecycle in v1.18.4 and current master. I did not run a full daprd reproduction on those two versions.

Expected Behavior

After an in-process runtime restart caused by HotReload/SIGHUP, Prometheus metrics remain valid. The old runtime releases its OpenCensus meter before the replacement runtime records the same metric names and labels.

Actual Behavior

After a Configuration/Resiliency-triggered runtime reload, affected sidecars repeatedly log errors on metric scrapes:

error gathering metrics: ... [from Gatherer #1] collected metric "dapr_grpc_io_server_server_latency" ... was collected before with the same name and label values

The error affects multiple families, including dapr_grpc_io_*, dapr_http_*, and dapr_component_pubsub_*. It persists after the reload completes. A pod restart clears it; a subsequent reload can bring it back.

We observed this across multiple applications after a namespace-level Resiliency change triggered SIGHUP reloads. We also observed fresh pods start clean and develop the same error immediately after a later Configuration change.

Suspected cause

The runtime factory creates a new view.NewMeter() on every invocation, passes it into the runtime, and returns rt.Run(ctx) without an explicit meter stop in that factory:

• v1.18.4 meter creation
• v1.18.4 runtime return

The library-level reproduction below shows that two active OpenCensus meters containing the same metric produce the same Prometheus duplicate-sample error. Stopping the old meter removes that error.

This suggests a missing meter lifecycle cleanup during in-process restart. A fix should cover normal shutdown and initialization errors, and should stop the old meter only after the old runtime stops using it. A full Dapr integration test is still needed to confirm the correct ownership and cleanup order.

Steps to Reproduce the Problem

Observed Kubernetes sequence

1. Start a Dapr 1.18.3 sidecar with HotReload enabled.
2. Generate application traffic so the sidecar records metrics.
3. Change a Configuration or Resiliency resource that triggers an in-process restart.
4. Generate traffic again and scrape the sidecar metrics endpoint.
5. Check the sidecar logs for the duplicate-name-and-label error above.
6. Restart the pod and verify the duplicate error clears.

Minimal library-level reproduction (executed)

This isolates the suspected meter lifecycle. It does not launch daprd or Kubernetes.

Create an empty directory, save this as main.go, and run:

go mod init dapr-metrics-repro
go get contrib.go.opencensus.io/exporter/prometheus@v0.4.2 github.com/prometheus/client_golang@v1.22.0 go.opencensus.io@v0.24.0
go mod tidy
go run .

package main

import (
 "context"
 "fmt"
 "strings"
 ocprom "contrib.go.opencensus.io/exporter/prometheus"
 prom "github.com/prometheus/client_golang/prometheus"
 "go.opencensus.io/stats"
 "go.opencensus.io/stats/view"
)

func newMeter() view.Meter {
 m := view.NewMeter()
 m.Start()
 measure := stats.Int64("repro/count", "repro", stats.UnitDimensionless)
 if err := m.Register(&view.View{Name:"repro_count", Measure:measure, Aggregation:view.Count()}); err != nil {panic(err)}
 if err := stats.RecordWithOptions(context.Background(), stats.WithRecorder(m), stats.WithMeasurements(measure.M(1))); err != nil {panic(err)}
 // RetrieveData serializes with the recorder and ensures that the value exists.
 if _, err := m.RetrieveData("repro_count"); err != nil {panic(err)}
 return m
}
func gather(label string) {
 reg := prom.NewRegistry()
 if _,err := ocprom.NewExporter(ocprom.Options{Namespace:"dapr", Registry:reg}); err != nil {panic(err)}
 _,err := (prom.Gatherers{reg, prom.DefaultGatherer}).Gather()
 fmt.Printf("%s: duplicate=%v error=%v\n",label,err!=nil && strings.Contains(err.Error(),"was collected before"),err)
}
func main() {
 first:=newMeter()
 gather("one runtime")
 second:=newMeter()
 gather("reload without old meter.Stop")
 first.Stop()
 gather("old meter.Stop applied")
 second.Stop()
}

Observed output:

one runtime: duplicate=false error=<nil>
reload without old meter.Stop: duplicate=true error=[from Gatherer #1] collected metric "dapr_repro_count" { counter:{value:1}} was collected before with the same name and label values
old meter.Stop applied: duplicate=false error=<nil>

Workaround

Disable HotReload in each affected application's Configuration, then restart affected pods to clear the old meters:

spec:
  features:
    - name: HotReload
      enabled: false

Disabling reload alone does not clear meters already accumulated in a running process.

Related work and duplicate search

I searched open and closed issues and PRs for collected before, error gathering metrics, duplicate metrics, meter.Stop, and metrics with reload/SIGHUP. I found no matching report.

• daprds: Adds SIGHUP handler reloader #9579 and hotreload: Configuration, HTTPEndpoint, and Resiliency #9582 introduced the relevant in-process reload paths.
• Scheduler: embedded etcd (etcd_*) metrics missing from metrics endpoint (port 9090) after 1.18 upgrade #10252 / Metrics: Restore etcd metrics on the scheduler #10254 address missing scheduler etcd metrics and exporter registry behavior. This report concerns duplicate samples after runtime reload, rather than missing etcd metrics or a collector-registration panic.

Could maintainers confirm whether this is already tracked and whether a fix can be considered for a 1.18 patch release?

Release Note

RELEASE NOTE: FIX Prevent duplicate Prometheus metric samples after an in-process runtime reload by releasing the old runtime's metrics resources.