networkPolicies.restrictInternalTraffic is blocking also Metrics scrape endpoint #13732

This is the chart working as written, not a scrape misconfiguration on your side. The longhorn-manager policy is an ingress allow-list that covers all ports on the manager pods, and Longhorn serves /metrics on the same port as its internal REST API, so Prometheus gets swept up with everything else.

chart/templates/network-policies/manager-network-policy.yaml (v1.12.1, lines 1-35):

{{- if .Values.networkPolicies.restrictInternalTraffic }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: longhorn-manager
  namespace: {{ include "release_namespace" . }}
spec:
  podSelector:
    matchLabels:
      app: longhorn-manager
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: longhorn-manager
    - podSelector:
        matchLabels:
          app: longhorn-ui
    - podSelector:
        matchLabels:
          app: longhorn-csi-plugin
    - podSelector:
        matchLabels:
          longhorn.io/managed-by: longhorn-manager
        matchExpressions:
          - { key: recurring-job.longhorn.io, operator: Exists }
    - podSelector:
        matchExpressions:
          - { key: longhorn.io/job-task, operator: Exists }
    - podSelector:
        matchLabels:
          app: longhorn-driver-deployer
{{- end }}

Three properties of that rule make it fatal for scraping:

1. No monitoring source is on the list. The six from entries are manager, UI, CSI plugin, recurring-job pods, job-task pods, and driver-deployer. Nothing describes a Prometheus workload, so a scraper is denied even if it runs inside longhorn-system — moving it into the namespace does not help.
2. No ports: key on the ingress rule (compare recovery-backend-network-policy.yaml lines 18-20, which does scope itself to port: 9503), so the allow-list governs every port on the pod, 9500 included.
3. Every from entry is a bare podSelector with no namespaceSelector (the string namespaceSelector does not appear in the file at all), and a bare podSelector matches only pods in the policy's own namespace — so a scraper outside the release namespace can never match any entry regardless of its labels.

One precondition: these resources only bite where your CNI actually enforces NetworkPolicy. Longhorn says so explicitly — "These policies are enforced only when the cluster has a network plugin that supports and enforces Kubernetes NetworkPolicy. Without such a provider, the resources are created but do not affect traffic" (important-notes/_index.md:255, same point at references/networking.md:17). Your screenshot says yours does.

And 9500 is unambiguously the metrics port. In longhorn-manager, /metrics is registered on the same mux as the v1 API — api/router.go:56:

r.Methods("GET").Path("/metrics").Handler(registry.Handler())

That router is served at types.GetAPIServerAddressFromIP(currentIP) (app/daemon.go:374,378), which is net.JoinHostPort(ip, strconv.Itoa(DefaultAPIPort)) with DefaultAPIPort = 9500 (types/types.go:136,459-460). currentIP is the pod IP, not a host IP (app/daemon.go:294, util.GetRequiredEnv(types.EnvPodIP)), so the listener is squarely subject to the policy. The chart wires the scrape path to that same port end to end: containerPort: 9500 / name: manager (chart/templates/daemonset-sa.yaml:51-53), the longhorn-backend Service - name: manager / port: 9500 / targetPort: manager (daemonset-sa.yaml:203-206), and the generated ServiceMonitor scrapes - port: manager (chart/templates/servicemonitor.yaml:28). The only other listener the daemon starts is the debug server on 127.0.0.1:6060 (daemon.go:383-390) — there is no separate metrics port and no exemption anywhere in the policy.

What actually changed under you

The policy body has not changed since v1.12.0 — only its gate did. At v1.12.0 line 1 reads {{- if .Values.networkPolicies.enabled }}, and values.yaml:37 has enabled: false, so nothing was rendered unless you asked for it. At v1.12.1 the gate is restrictInternalTraffic, and values.yaml:39 has restrictInternalTraffic: true. That is commit 4f339f83 ("feat(security): network policies for internal components", master, 2026-06-11), backported to v1.12.x as 0b8fe1ed; tracked in #13438 (milestone v1.13.0) with backport #13439 (milestone v1.12.1). The manifest install path got the same treatment — deploy/longhorn.yaml carries zero NetworkPolicy objects at v1.12.0 and six at v1.12.1, with the manager one at lines 103-136 (same for deploy/longhorn-okd.yaml).

So on a policy-enforcing CNI, a plain 1.12.1 install with metrics.serviceMonitor.enabled=true breaks scraping out of the box: the ServiceMonitor is opt-in (values.yaml:680, default false) while the policy blocking it is opt-out.

You do not need to turn it off

NetworkPolicies are additive — Longhorn's own docs state it: "Network policies are additive, and when both the source and destination are isolated by NetworkPolicies, a connection must be permitted by both the source's egress policy and the destination's ingress policy" (advanced-resources/security/network-policy.md, line 8). So add a second, narrowly scoped policy and leave restrictInternalTraffic=true:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: longhorn-manager-allow-metrics
  namespace: longhorn-system
spec:
  podSelector:
    matchLabels:
      app: longhorn-manager
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: cattle-monitoring-system   # your Prometheus namespace
      podSelector:
        matchLabels:
          app.kubernetes.io/name: prometheus                      # tighten to your actual Prometheus pod labels
    ports:
    - protocol: TCP
      port: 9500

The namespaceSelector and podSelector are in the same list element on purpose — that is an AND, not two alternatives. kubernetes.io/metadata.name is the same namespace selector the chart itself uses for the UI policy (ui-frontend-network-policy.yaml:16-18 for rke1, 25-27 for rke2).

You can ship it with the release rather than managing it out of band, via extraObjects (values.yaml:713, rendered by templates/extra-objects.yaml, which is just {{- range .Values.extraObjects }} / {{- tpl (toYaml . ) $ }}):

extraObjects:
  - apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: longhorn-manager-allow-metrics
      namespace: longhorn-system
    spec:
      podSelector:
        matchLabels:
          app: longhorn-manager
      policyTypes:
      - Ingress
      ingress:
      - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: cattle-monitoring-system
          podSelector:
            matchLabels:
              app.kubernetes.io/name: prometheus
        ports:
        - protocol: TCP
          port: 9500

That way it survives helm upgrade. One naming caveat if you are on Rancher with project network isolation: do not call it np-default. Longhorn does not touch custom policies — maintainer response on #12505 (2026-01-20): "Longhorn does not natively monitor or enforce Network Policies after the initial installation. Because np-default is not a standard Longhorn resource, it is not managed by the system." It was Rancher's project isolation that kept replacing that specific name.

Note this only re-opens 9500 from the one namespace; it does not weaken any of the other five internal policies, which is what restrictInternalTraffic=false would do (backing-image-data-source, backing-image-manager, instance-manager, longhorn-recovery-backend, longhorn-webhook — the full list is at important-notes/_index.md:265-270).

One thing that looks like it should help but does not: the manager pods also carry longhorn.io/admission-webhook: longhorn-admission-webhook, which each manager stamps onto its own pod when it takes the webhook lease (app/daemon.go:207 → types.GetAdmissionWebhookLabel(), types/types.go:544-548, inside the leader-elected fnStartWebhook at daemon.go:159-216). That means webhook-network-policy.yaml selects the same pods, and with the default empty kubeAPIServerSourceCIDRs (values.yaml:43) it renders an ingress rule with no from at all — which allows any source. But that rule lists only ports 9501 and 9502 (webhook-network-policy.yaml:14-18), so it never opens 9500. The docs confirm the same reading at network-policy.md:67.

If your Prometheus runs with hostNetwork: true, that only helps for the manager pod co-located with it — the docs' note that "traffic to and from the node hosting a Pod is always allowed for NetworkPolicy purposes" (network-policy.md:109) is node-local, and longhorn-manager is a DaemonSet (daemonset-sa.yaml:2), so the other nodes' targets would still fail.

Worth filing

There is a real docs gap. The networking reference's longhorn-manager ingress matrix (references/networking.md:23-34) lists only manager/UI/CSI plugin/recurring-job/driver-deployer on 9500 plus the webhook and recovery-backend ports — no monitoring row, and the words "Prometheus", "metrics", and "monitoring" do not appear in the file at all. #13591 ("[DOC] NetworkPolicy setup guidance", milestone v1.13.0) produced advanced-resources/security/network-policy.md, which covers enabled, restrictInternalTraffic, and kubeAPIServerSourceCIDRs in detail but has no metrics-scraping section either. And the upgrade note (important-notes/_index.md:255-276) says to "leave this setting enabled unless the policies block traffic required by your environment" without calling out that metrics scraping is exactly such a case.

#12505 hit the same wall in January ("[BUG] No network policy to allow prometheus to scap metrics on RKE2") but with networkPolicies.enabled: true set explicitly, and the stale bot closed it not_planned on 2026-02-26. Since the policies now default to on, this is worth a fresh issue rather than a reopen — either a docs fix or a networkPolicies.monitoringNamespaceSelector-style chart value, given the chart already ships the ServiceMonitor that the chart's own default policy blocks. (This thread is a Q&A discussion, so it will not get milestoned on its own.)