🧭 Add root-cause and correlation work with its dashboard support - #606

What this changes

Ten items from docs/specs/core-round-2026-11.md, in one branch. Alert text stays terse and technical; beginner help is on the dashboard only. Nothing here forecasts: every sentence says what kwatch saw now.

0. Gates headroom. The slowest notify-tier first message was 5m0s against the 5m gate. A workload short of ready replicas under an autoscaler at its ceiling waited the full 5 minute unavailable grace for a cause the ceiling already named; it now waits 3 minutes (CeilingUnavailable). Notify max 5m0s to 4m15s, p95 4m15s to 3m15s. The staging peak fell 28/h to 26/h as a side effect of item 4.
1. Dependency hosts from ConfigMaps. The informer transform extracts only host:port targets from non-credential keys (same parser, bounds and credential-name filter as env values, values over 512 bytes skipped, never Secrets) before digesting; no raw value is kept. A join gives pods that read the ConfigMap via envFrom or configMapKeyRef a Calls edge, whichever of pod and ConfigMap arrives first. Messages say (from ConfigMap api-config, DB_HOST).
2. Errors while Kubernetes says healthy. A bounded sampler reads the last 2 minutes of one Running+Ready pod's log per workload (at most 6 reads per 30 s, one pod per workload per window, 1000 lines / 256 KiB). New finding LogErrorBurst (mode ErrorBurst, Pro, notify): at least 20 error lines, at least 5 times the workload's own earlier rate per window, with at least 10 minutes of earlier windows. A refused log read pauses it for 5 minutes without a finding. Opt out with investigation.sampleLogs: false (also off when investigation.logs is false).
3. Cause check on resolve. At resolve time the cause is looked at again (node Ready, endpoint probe ok, Service endpoints ready). Still there: ... is healthy again, but <cause> still <state>. The same cause is still present.; recovered: <cause> accepts connections again.
4. Same-time correlation. Two workloads that begin failing within 30 s on a healthy node are now enough to suspect the node (was three), with a bounded co-onset proof (+0.05) and the fact They started failing 4 seconds apart. Independent failures on separate nodes, or minutes apart, stay apart (adversarial scenarios).
5. User endpoints. Users can't reach shop.example.com/checkout, shop.example.com and 2 more. from Ingress and HTTPRoute rules (path only when the rule has one), 2 named then and N more, worst first. Dashboard Traffic rows list up to 8 Users reach it at rows.
6. Cascade timeline. Timeline: 10:09 postgres out of memory → 10:10 api unavailable → 10:11 service api no ready endpoints. for chains of 3 or more steps, in onset order, observed events only. Dropped from messages past 600 characters; never in the short form or a resolve. The dashboard has it as an ordered list.
7. Missing-evidence gates. TestMissingEvidenceNeverStatesAWrongRoot, wired into make alert-quality-gate, replays every labelled scenario with logs denied, without metrics and with Secrets and ConfigMaps hidden.
8. Blast-overlap change ranking. A changed candidate that shares failing pods with another change is scored by the share of the failing pods downstream of it (+0.1 at all pods, -0.05 under 30%). Message: 5 of 5 failing pods read it.
9. Close-call check. When two causes are within the rival margin and one is an external endpoint, one DNS lookup and one TCP connect (bounded, private addresses refused, remembered 2 minutes) run during investigation: kwatch checked: DNS resolves db.example.com fine, so DNS is not the cause.

Decisions on the spec's open points

• Resolve wording is the factual form. No forecast words anywhere.
• kwatch will keep watching it is not said: a resolved incident is not kept open. While the cause is present the incident stays open (scenario resolve-cause-still-present); the resolve form only occurs when the cause is not a member of the incident.
• Error baseline: kept in memory per workload, the previous hour of 2-minute windows. The words say the previous N min when less than an hour of history exists.
• Probes that exist: DNS lookup and TCP connect. No readiness call or image reachability check is run, so none is claimed.
• Gateway routes are read (HTTPRoute addresses via its Gateway listeners).
• No docs glossary links; dashboard explanations are visible text.

Before and after

See the goldens in internal/notification/compose/testdata and internal/scenarios/testdata/messages: external_endpoint_storm, log_error_burst, log_error_burst_no_baseline, resolved_cause_still_present, resolved_cause_recovered, user_endpoints, user_endpoints_partial, close_call_checked, and the scenario goldens for co-onset-pair-shares-node, chain-timeline-line, change-blast-overlap, dependency-host-from-configmap, healthy-pods-error-burst, user-endpoints-named.

before  api in shop is crash looping because external endpoint db.example.com:5432 refuses connections.
after   api in shop is crash looping because external endpoint db.example.com:5432 (from ConfigMap api-config, DB_HOST) refuses connections.

before  (no message)
after   orders in shop is up but failing: 340 errors in 5 min. Pods are Running and Ready; the log shows "pq: connection refused" 312 times, up from 2 in the previous hour. To read the recent errors, run kubectl logs deploy/orders -n shop --since=5m | grep -i error

before  billing-a in billing is healthy again. It was down for 15 minutes because node n1 was failing.
after   billing-a in billing is healthy again. Node n1 is Ready again. It was down for 15 minutes.

Scorecard

SCORECARD.md is regenerated; 28 of 28 gates pass.

Added scenarios: 14 labelled, each with expect.json and a message golden.

Checks

• make verify run locally: build, vet, lint, race tests with coverage, line and diff checks, architecture, test layout, hygiene, catalogs, docs, manifests, negative regressions and dead code all pass. The one race-run failure is TestProviderHealthIsPublishedWithoutFailingReadiness in internal/app; it fails identically on main under -race here (a delivery health event wait), so it is not from this branch. It passes without -race.
• Dashboard browser tests (go test -tags browser -run Browser ./internal/dashboard/) pass, including a new test for the issue page block.
• make alert-quality-gate: 28 of 28 gates, plus the ablation and missing-evidence tests.

Needs CI

• The e2e label: runtime behaviour changes (new sampler, new Calls edges, new probe).
• Kind coverage for the new pods/log reads is exercised by the existing RBAC (pods/log is already granted).

Not done or done differently

• The Evidence > Logs per-minute strip and the Details > Why candidate table are not drawn. The data (the burst and the two candidates) is on the issue page in a single "What kwatch found" block, not spread across the tabs.
• The persisted incident record gained optional fields (Source on the cause); no schema bump, since old files decode and omitted fields are absent.
• Missing-evidence gate: fifteen scenarios with hidden Secrets or ConfigMaps still state the workload's rollout or own configuration as the cause. They are listed as tolerated defects in toleratedWithoutEvidence; the test fails when one stops being true.
• Messages per incident (p95 3, most 5) did not move; the five-message incident exists on the staging day only.