Pro auto-evidence: kwatch runs the read-only checks and shows the result - #584

Implements Pro auto-evidence (AUTO_EVIDENCE_PLAN.md, delivery row 2): if a command only reads state, kwatch runs it, correlates it and shows the result. Only commands that change the cluster stay as steps.

What changed

• Check registry (internal/pipeline/investigate): every applicable check runs in parallel under a 10 s plan budget, with per-check timeouts, at most 10 checks, at most 4 API reads in flight, and panic isolation. The 7 investigators are ported and still lead the bounded channel summary in their old priority order. New checks: pod state, logs (previous and current, per container, deduplicated by signature), events, memory use (kubelet stats only; no metrics-server needed), node pods, volume, Service and Ingress chains, rollout, control plane (/readyz and component pods), finalizers, object state, image and pull secrets, certificates, quota, HPA, Job, probes, and config key names (never values).
• Report: Result now carries Report []incident.Section (status ok, denied, timeout or absent, with a reason of rbac or off) and a Correlation. The report is redacted at capture and again at serve, capped at about 64 KB and 500 lines per section, and kept in memory only (no schema change).
• Correlation (correlate.go): classifies the log signature and links the host to a Service, Pod or Node. It also finds the nearest prior change, the spread across pods and nodes, and the rollout image and resources diff. Confirm rules cover an unreachable Service, OOM, a missing ConfigMap or Secret, a missing StorageClass, a missing pull secret, a kubelet waiting reason, node conditions, a selector that matches no pod, and a crash exit code.
• Steps (Pro): read-only steps whose covering check came back ok are dropped. Denied or timed-out reads keep their step. TestStepChecksCoverReadOnlySteps scans every notification.Step literal in compose and fails when a read-only command has no covering check. TestChecksCoverEveryStepCheck fails when that check isn't registered. A confirmed cause adds "… Confirmed." and sets confidence to confirmed.
• Fitting: compose pins the Pro fix (the mutating command and its sentence). RenderBlocksWithin cuts the proof and keeps the pinned fix whole. Tests cover Discord (2000), Telegram and Slack sections, plus a shared providertest.ProFix fixture.
• Dashboard: GET /api/incidents/{id} adds evidence and correlation (omitted without a report). The new Evidence tab (?tab=evidence) has collapsible sections with badges (Read / Not allowed / Timed out / Nothing to read / Turned off), a log viewer with a container select, a previous/current toggle and marked lines, a related-objects chain, the correlation card, and Fix at the bottom. Copy follows specs/auto-evidence.md, with the "Update menu" clause dropped as decided.
• Config and RBAC: investigation.logs (default true; false reads no log). Helm rbac.readyz (default true). Audit entries add evidence (check:status) and readSteps.
• e2e: ExpectGatheredEvidence (audit based) on CrashLoop, OOM, ImagePull, missing Secret, Pending PVC, a Service with no endpoints, and node pressure. It requires 0 read-only steps, confirmed, and at least one check that read something.

Community: no teaser and no dashboard link (per the decisions). Existing compose, replay and scenario goldens are unchanged.

Checks run locally

• make verify: every stage passed, including race tests and coverage. The final deadcode stage flagged one unused helper; it was removed and make deadcode-check was re-run and passes.
• make test-licensetest passes.
• helm lint and deploy/chart/test_helm.sh pass; renders checked with the new values on and off.
• I checked the Evidence tab in a browser against a mocked API: Summary/Evidence switching, denied and timeout sections, the log viewer, no console errors.

CI must still verify

• e2e on kind (label e2e), especially the new confirmed assertions, which have only been tested against fakes.
• actionlint, ShellCheck and the mod-tidy check (not run locally).

Follow-ups in this PR

• Cut order and goldens: a Pro message that overflows loses quote lines from the top first (down to 2), then proof from the bottom, then the quote, and last the headline. The fix is never cut (safetext.FitPinned, also used by Telegram). Goldens cover each cut level 1 to 5 for Discord, Telegram and Slack (testdata/cut/level-N.golden).

• Teams and Email layouts: a Pro message with evidence gets a Teams card (cause, what kwatch found, monospace quote, fix; no action). Email gets the same order as plain text plus an HTML alternative. Both have goldens. Teams Workflows is unchanged: it sends one plain text block.

• Computed fix: a workload whose container was killed at its memory limit gets kubectl set resources … --limits=memory=<N>Mi (Mutating). N is 1.5 × the larger of the 24h peak and the limit, rounded up to 64Mi. Pro only.

• Webhook evidence: the structured payload (webhook, n8n, Zapier) adds evidence {confirmed, checks, facts, quote, fix}. It is absent without a report, so existing fields and Community payloads are unchanged.

• Read times: each section carries the time it was read, from the injected clock. The API exposes it as readAt, and the dashboard shows "Read at …".

• Discord embed (deliberate format change, both editions): incidents are one embed. The title is the headline, the colour follows the status, and the description holds cause, proof, quote and fix (4096, embed 6000; proof cut first, never the fix). The footer has kwatch, the cluster and the time. There is no field per section (Message too long for Discord #194, config doesn't filter Events in Discord messages (ref #175) #416). Mentions are neutralized and thread_id works. Plain notices stay content, cut to 2000. The delivery payload bound for Discord is now 6000. Contract goldens are updated.

• e2e fixes: a check that read and found nothing (absent) now covers its step. A read that failed for another reason gets the new error status and keeps its step. The usage check reports requests, limits and the 24h peak from kubelet stats. Events include container notes. kubectl describe service is covered by the Service check.

• No add-on dependency: the usage check reads only kubelet stats (nodes/stats, already granted). The metrics.k8s.io read, its RBAC rule and rbac.metricsApi are removed.

• Cluster DNS discovery: DNS servers are found through the DNS Service (kube-system/kube-dns, openshift-dns/dns-default, or a kube-system Service named for CoreDNS or selecting k8s-app=kube-dns), then the pods it selects and their owners of any kind. The coredns and kube-dns names are now only a fallback. This replaces kube.DNSServerNames with kube.DNSServerPods, IsDNSServer, IsDNSWorkload and TopOwner, used by the DNS metrics probe and the DNS outage rules. Tests cover OpenShift, RKE2, Helm-prefix and fallback setups.

Contract changes

• API: evidence[].readAt, and the section status error.
• Webhook, n8n and Zapier: additive evidence object.
• Discord: incidents are sent as embeds; the payload policy is 6000.
• notification.Message.Evidence (json omitempty).

Deferred

• Docs (kwatch-docs) and the Pro page line (kwatch-web).