fix(metrics): bound reason label cardinality for blocked responses by 0xERR0R · Pull Request #2114 · 0xERR0R/blocky

0xERR0R · 2026-06-12T21:08:24Z

Problem

The block reason gained the matched rule per group in #2091 (e.g. BLOCKED (ads: /snapchat/)). That string is used verbatim as the reason label of blocky_response_total (see MetricsResolver.Resolve). For exact-match deny lists the matched rule equals the blocked domain, so the label is effectively unbounded: every distinct blocked domain that clients query creates a new Prometheus series. With large deny lists (e.g. StevenBlack) on a busy network this can grow to tens of thousands of series, and it's influenced by whatever clients happen to query.

Regex/wildcard entries are bounded (the rule is the pattern), but exact-match lists are the common case and the dangerous one.

Fix

Add a separate low-cardinality model.Response.ReasonLabel that carries the matched group names only (BLOCKED (ads)). MetricsResolver prefers it for the reason label and falls back to Reason when empty, so non-blocked responses are unchanged.

The detailed Reason (with the matched rule) is untouched and still flows to:

the query log (responseReason field)
structured/debug logs
EDE (Extended DNS Errors)

This reverts the metric label to its pre-#2091 cardinality while keeping the richer rule detail everywhere it doesn't cost a Prometheus series.

Implementation

model.Response.ReasonLabel — new field; empty ⇒ metrics fall back to Reason.
formatBlockReason / formatBlockReasonLabel share a renderBlockReason(…, includeRules bool) helper; the label variant joins sorted group names only.
handleBlocked threads reasonLabel through and sets it on the response.

Tests

MetricsResolver: uses ReasonLabel for the label when set; falls back to Reason when empty.
BlockingResolver: blocked response keeps detailed Reason but exposes a group-only ReasonLabel.
formatBlockReasonLabel unit suite, incl. the cardinality guarantee: different rules in the same group produce the same label.
New HaveReasonLabel gomega matcher.

Dashboards & docs

docs/prometheus_grafana.md: note explaining the metric reason label is group-level for blocked responses, with the full rule available in the query log.
docs/blocky-grafana.json: the Top block reasons and Response reasons panels keep working unchanged (they aggregate sum by (reason) / topk, which don't depend on the label value). Their descriptions are updated to clarify the label is group-level and to point at the query-log dashboard for per-rule drill-down. No query edits — the panels now bucket by group, which is the intended bounded behaviour.
The query-log SQL dashboards (blocky-query-grafana.json / -postgres.json) read the full reason from the query-log DB and are unaffected — they still show the matched rule/domain.

Compatibility note

This changes existing reason label values for blocked responses (BLOCKED (group: rule) → BLOCKED (group)). Anyone with alerts/recording rules keyed on the full rule string would need to adjust — documented in docs/prometheus_grafana.md.

The block reason gained the matched rule per group in #2091 (e.g. "BLOCKED (ads: /snapchat/)"). That string is used verbatim as the `reason` label of blocky_response_total, so for exact-match deny lists the rule equals the blocked domain and the label becomes unbounded: every distinct blocked domain that clients query creates a new series. Add a separate low-cardinality Response.ReasonLabel that carries the matched group names only ("BLOCKED (ads)"). MetricsResolver prefers it and falls back to Reason when empty, so non-blocked responses are unchanged. The detailed Reason (with the matched rule) is untouched and still flows to the query log, structured logs and EDE. Reverts the metric label to its pre-#2091 cardinality while keeping the richer rule detail everywhere it does not cost a Prometheus series.

codecov · 2026-06-12T21:10:56Z

Codecov Report

❌ Patch coverage is 94.59459% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.03%. Comparing base (9702c8f) to head (a9e8700).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
resolver/metrics_resolver.go	90.90%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2114      +/-   ##
==========================================
+ Coverage   86.96%   87.03%   +0.07%     
==========================================
  Files         122      122              
  Lines        9358     9372      +14     
==========================================
+ Hits         8138     8157      +19     
+ Misses        979      975       -4     
+ Partials      241      240       -1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR addresses unbounded Prometheus series growth by ensuring the reason label on blocky_response_total uses a low-cardinality value for blocked responses, while preserving the detailed matched-rule reason everywhere else (query log, debug logs, EDE).

Changes:

Introduces model.Response.ReasonLabel as a low-cardinality alternative to Reason for metrics.
Updates MetricsResolver to prefer ReasonLabel for the reason metric label (fallback to Reason when empty).
Extends BlockingResolver to populate ReasonLabel with group-only blocked reasons and adds unit tests + documentation update.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
resolver/metrics_resolver.go	Uses `Response.ReasonLabel` (fallback to `Reason`) for Prometheus `reason` label to bound cardinality.
resolver/metrics_resolver_test.go	Adds coverage for `ReasonLabel` preference and fallback behavior.
resolver/blocking_resolver.go	Threads a group-only `reasonLabel` into blocked responses; shares formatter logic via `renderBlockReason`.
resolver/blocking_resolver_test.go	Verifies blocked responses keep detailed `Reason` while exposing group-only `ReasonLabel`; adds label-format unit tests.
model/models.go	Adds `Response.ReasonLabel` field with documentation on intended use.
helpertest/helper.go	Adds `HaveReasonLabel` gomega matcher for assertions in tests.
docs/prometheus_grafana.md	Documents the `reason` label behavior change for blocked responses and the rationale.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…log for rule detail

- Extract repeated "CACHED" literal into a cachedReason constant - Use an early-return guard in MetricsResolver.Resolve to reduce nesting

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/0xerr0r/blocky](https://github.com/0xERR0R/blocky) | minor | `v0.31.0` → `v0.32.0` | --- ### Release Notes <details> <summary>0xERR0R/blocky (ghcr.io/0xerr0r/blocky)</summary> ### [`v0.32.0`](https://github.com/0xERR0R/blocky/releases/tag/v0.32.0) [Compare Source](0xERR0R/blocky@v0.31.0...v0.32.0) #### Changelog ##### Features - [`c851293`](0xERR0R/blocky@c851293): feat(blocking): log the matched rule in the block reason ([#2091](0xERR0R/blocky#2091)) ([@0xERR0R](https://github.com/0xERR0R)) - [`87be127`](0xERR0R/blocky@87be127): feat(cache): Shard the result cache to remove the single-lock read ceiling ([#2097](0xERR0R/blocky#2097)) ([@0xERR0R](https://github.com/0xERR0R)) - [`7c6da15`](0xERR0R/blocky@7c6da15): feat(config): structurally merge config folder files ([#1827](0xERR0R/blocky#1827)) ([#2112](0xERR0R/blocky#2112)) ([@0xERR0R](https://github.com/0xERR0R)) - [`e7958e0`](0xERR0R/blocky@e7958e0): feat(lists): opt-in on-disk download cache with conditional revalidation ([#2087](0xERR0R/blocky#2087)) ([@0xERR0R](https://github.com/0xERR0R)) - [`b82199b`](0xERR0R/blocky@b82199b): feat(querylog): ignore domains (exact/wildcard/regex) in query log ([#2084](0xERR0R/blocky#2084)) ([@0xERR0R](https://github.com/0xERR0R)) - [`bee2d8b`](0xERR0R/blocky@bee2d8b): feat(resolver): UDP-first plain-DNS upstream and EDNS0 buffer floor ([#2100](0xERR0R/blocky#2100)) ([@0xERR0R](https://github.com/0xERR0R)) - [`0b70e5c`](0xERR0R/blocky@0b70e5c): feat(resolver): add DNS rebinding protection ([#2111](0xERR0R/blocky#2111)) ([@0xERR0R](https://github.com/0xERR0R)) - [`1b8e08a`](0xERR0R/blocky@1b8e08a): feat(resolver): pool DoT connections and enable TLS session resumption ([#2098](0xERR0R/blocky#2098)) ([@0xERR0R](https://github.com/0xERR0R)) - [`4018d9e`](0xERR0R/blocky@4018d9e): feat: in-memory statistics subsystem with /api/stats REST endpoint ([#2093](0xERR0R/blocky#2093)) ([@0xERR0R](https://github.com/0xERR0R)) - [`7abca44`](0xERR0R/blocky@7abca44): feat: support PROXY protocol on proxied DoT/DoH listeners ([#2094](0xERR0R/blocky#2094)) ([@kastakhov](https://github.com/kastakhov)) ##### Bug fixes - [`2496d12`](0xERR0R/blocky@2496d12): fix(dnssec): close DNSSEC validation bypass & cache-scope pollution (GHSA-x845-2f78-7v36) ([#2119](0xERR0R/blocky#2119)) ([@0xERR0R](https://github.com/0xERR0R)) - [`06555e0`](0xERR0R/blocky@06555e0): fix(metrics): bound reason label cardinality for blocked responses ([#2114](0xERR0R/blocky#2114)) ([@0xERR0R](https://github.com/0xERR0R)) - [`e0ea9b3`](0xERR0R/blocky@e0ea9b3): fix(resolver): eliminate recursive RLock deadlock in blocking group resolution ([#2106](0xERR0R/blocky#2106)) ([@0xERR0R](https://github.com/0xERR0R)) - [`80f742e`](0xERR0R/blocky@80f742e): fix(server): answer browser CORS preflights for custom headers and Private Network Access ([#2109](0xERR0R/blocky#2109)) ([@0xERR0R](https://github.com/0xERR0R)) - [`9702c8f`](0xERR0R/blocky@9702c8f): fix(stats): populate allow/denylist counts in /api/stats at startup ([#2113](0xERR0R/blocky#2113)) ([@0xERR0R](https://github.com/0xERR0R)) ##### Build and dependencies - [`821e378`](0xERR0R/blocky@821e378): build(deps): bump codecov/codecov-action from 6 to 7 ([#2089](0xERR0R/blocky#2089)) ([@dependabot](https://github.com/dependabot)\[bot]) - [`d63f84b`](0xERR0R/blocky@d63f84b): build(deps): bump github.com/0xERR0R/expiration-cache from 0.1.0 to 0.2.0 ([#2095](0xERR0R/blocky#2095)) ([@dependabot](https://github.com/dependabot)\[bot]) - [`81f2b2a`](0xERR0R/blocky@81f2b2a): build(deps): bump github.com/onsi/ginkgo/v2 from 2.29.0 to 2.30.0 ([#2108](0xERR0R/blocky#2108)) ([@dependabot](https://github.com/dependabot)\[bot]) - [`9c5abdf`](0xERR0R/blocky@9c5abdf): build(deps): bump github.com/onsi/ginkgo/v2 from 2.30.0 to 2.31.0 ([#2116](0xERR0R/blocky#2116)) ([@dependabot](https://github.com/dependabot)\[bot]) - [`6c0ec5f`](0xERR0R/blocky@6c0ec5f): build(deps): bump github.com/onsi/gomega from 1.41.0 to 1.42.0 ([#2115](0xERR0R/blocky#2115)) ([@dependabot](https://github.com/dependabot)\[bot]) - [`4beb9fe`](0xERR0R/blocky@4beb9fe): build(deps): bump github.com/quic-go/quic-go from 0.59.1 to 0.60.0 ([#2090](0xERR0R/blocky#2090)) ([@dependabot](https://github.com/dependabot)\[bot]) - [`8fe2202`](0xERR0R/blocky@8fe2202): build(deps): bump golang.org/x/net from 0.55.0 to 0.56.0 ([#2101](0xERR0R/blocky#2101)) ([@dependabot](https://github.com/dependabot)\[bot]) - [`52fa6a4`](0xERR0R/blocky@52fa6a4): build(deps): bump golang.org/x/sys from 0.45.0 to 0.46.0 ([#2096](0xERR0R/blocky#2096)) ([@dependabot](https://github.com/dependabot)\[bot]) ##### Misc - [`1c9f717`](0xERR0R/blocky@1c9f717): ci(e2e): speed up e2e job via Ginkgo proc oversubscription ([#2085](0xERR0R/blocky#2085)) ([@0xERR0R](https://github.com/0xERR0R)) - [`8d49922`](0xERR0R/blocky@8d49922): ci: cross-compile multi-arch docker image instead of QEMU emulation ([#2081](0xERR0R/blocky#2081)) ([@0xERR0R](https://github.com/0xERR0R)) - [`355a9a1`](0xERR0R/blocky@355a9a1): docs(grafana): redesign Grafana dashboard and update integration guide ([#2110](0xERR0R/blocky#2110)) ([@0xERR0R](https://github.com/0xERR0R)) - [`c791e80`](0xERR0R/blocky@c791e80): perf(e2e): probe blocky healthcheck at 250ms during start period ([#2086](0xERR0R/blocky#2086)) ([@0xERR0R](https://github.com/0xERR0R)) - [`a0fb375`](0xERR0R/blocky@a0fb375): perf(lists): speed up blocklist loading (cache build + parsing) ([#2083](0xERR0R/blocky#2083)) ([@0xERR0R](https://github.com/0xERR0R)) - [`316b073`](0xERR0R/blocky@316b073): perf(resolver): pre-classify client groups to cut per-query allocations ([#2103](0xERR0R/blocky#2103)) ([@0xERR0R](https://github.com/0xERR0R)) - [`d502806`](0xERR0R/blocky@d502806): perf(server): let Truncate decide response compression ([#2102](0xERR0R/blocky#2102)) ([@0xERR0R](https://github.com/0xERR0R)) - [`8b32512`](0xERR0R/blocky@8b32512): test(e2e): de-flake PROXY protocol and upstream init.strategy specs ([#2099](0xERR0R/blocky#2099)) ([@0xERR0R](https://github.com/0xERR0R)) *** #### ❤️ Support Blocky Blocky is free and open source, developed in my spare time — no telemetry, no ads, no hidden filtering. If it's useful to you, please consider supporting its continued development: [GitHub Sponsors](https://github.com/sponsors/0xERR0R) · [thanks.dev](https://thanks.dev/u/gh/0xERR0R) · [Liberapay](https://liberapay.com/spx01) · [Ko-fi](https://ko-fi.com/0xerr0r) · [PayPal](https://paypal.me/spx01) Thank you to everyone supporting Blocky! 🙏 </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).

Copilot AI review requested due to automatic review settings June 12, 2026 21:08

Copilot started reviewing on behalf of 0xERR0R June 12, 2026 21:09 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

docs(grafana): clarify reason panels are group-level, point to query …

72a2133

…log for rule detail

0xERR0R added this to the v0.32.0 milestone Jun 15, 2026

0xERR0R added the 🐞 bug Something isn't working label Jun 15, 2026

0xERR0R added 2 commits June 15, 2026 21:27

style(lint): satisfy goconst and nestif linters

79543b0

- Extract repeated "CACHED" literal into a cachedReason constant - Use an early-return guard in MetricsResolver.Resolve to reduce nesting

Merge branch 'main' into fix/metrics-reason-label-cardinality

a9e8700

0xERR0R enabled auto-merge (squash) June 15, 2026 19:27

0xERR0R merged commit 06555e0 into main Jun 15, 2026
23 checks passed

0xERR0R deleted the fix/metrics-reason-label-cardinality branch June 15, 2026 19:32

0xERR0R mentioned this pull request Aug 7, 2026

blocky_query_total can't be split by blocklist reason — no way to alert on "real" vs blocked query rate #2198

Closed

Navigation Menu

Repository navigation

fix(metrics): bound reason label cardinality for blocked responses - #2114

fix(metrics): bound reason label cardinality for blocked responses#2114
0xERR0R merged 4 commits into
mainfrom
fix/metrics-reason-label-cardinality

0xERR0R commented Jun 12, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

0xERR0R commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Implementation

Tests

Dashboards & docs

Compatibility note

Uh oh!

codecov Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

0xERR0R commented Jun 12, 2026 •

edited

Loading

codecov Bot commented Jun 12, 2026 •

edited

Loading