Autoscaler is reporting stale metrics for services/revisions #16200
Description
Since the switch of the metrics backend the autoscaler is reporting stale metrics for services/revisions. We run:
metrics-protocol: prometheus
request-metrics-protocol: prometheus
What version of Knative?
v1.19.6
Expected Behavior
Metrics of deleted revisions should no longer be reported.
Actual Behavior
Metrics of deleted revisions are reported with their last value.
Steps to Reproduce the Problem
Create a revision and delete it. These services/revisions all no longer exist and yet all metrics still report values like kn_revision_pods_count and kn_revision_pods_requested. All of which are stale:
kn_revision_pods_count{k8s_namespace_name="21uj92vxm973",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1
kn_revision_pods_count{k8s_namespace_name="21ujeqd9uia7",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1
kn_revision_pods_count{k8s_namespace_name="21ujiqdz3bpr",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1
kn_revision_pods_count{k8s_namespace_name="21uklm90kaa7",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1
kn_revision_pods_requested{k8s_namespace_name="21uh7qgrjzf3",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1
kn_revision_pods_requested{k8s_namespace_name="21uhb52o4rnz",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1