Skip to content

🧭 Teach the engine thirteen blind-scenario gaps - #619

Open
abahmed wants to merge 23 commits into
mainfrom
scenario-gaps-round
Open

abahmed wants to merge 23 commits into
mainfrom
scenario-gaps-round

Conversation

@abahmed

@abahmed abahmed commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

What was missed

make alert-quality failed on 13 of the 29 new gap scenarios (wrong root, no incident, or fallback to the workload), plus gate-level misses: three notify first messages over 5m (pod-stuck-terminating-finalizer 11m15s, release-image-pull-deadline 7m15s, cron-active-deadline-exceeded 6m), and a 6-message incident in the staging day.

Engine changes per miss

  • sched-node-selector-no-match, sched-anti-affinity-pending: engine already named the unmatched constraint; label changed (below).
  • pv-released-blocks-claim: the claim's own "volume X already bound to a different claim" event now names the volume; new row volume-bound-elsewhere roots at the PV. The misleading provisioner sentence is dropped for it.
  • release-pull-secret-wrong-type: a pull Secret that exists but is not a registry login, with the registry refusing as anonymous, is the cause (new mode, scorer wrong-type). Right-typed secrets keep blaming the registry.
  • release-priority-class-missing: the API server's own "no PriorityClass with name X was found" skips the 2m absence grace; used-missing row also covers ReplicaFailure/FailedCreate effects.
  • hpa-target-no-cpu-request: label changed (below).
  • node-no-execute-taint-evicts: label changed to quiet; new negative twin node-no-execute-taint-tolerated.
  • deployment-paused-with-pending-edit: new DeploymentPausedEdit finding (paused 30m with a recorded template edit, digest tier); the edit is not blamed as a release. Negative twin deployment-paused-no-edit stays quiet.
  • gateway-class-missing: Gateway API Accepted/Programmed Unknown is now a condition rule; 2m grace when only those wait for a controller.
  • node-cni-ip-conflict: new class Network.Conflict ("already exists for an interface", "address already in use"...) roots at the node from one stuck pod.
  • pod-stuck-terminating-finalizer: a pod with a non-builtin finalizer on a healthy node can be rooted at the finalizer (one pod suffices); a StatefulSet rollout is stuck at once when a pod of it has been terminating past the rollout threshold.
  • crd-conversion-webhook-gone: CRD conversion backend gone/no ready pods is a finding on the CRD, linked to clients whose error says "conversion webhook for /".
  • node-pleg-not-healthy: a node incident whose Ready condition just went false waits for the NotReady finding instead of announcing the symptom first (3 messages -> 2).
  • network-packet-loss-to-node: NOT fixed in the engine (see below).

Label / fixture changes (Kubernetes reason)

  • sched-node-selector-no-match, sched-anti-affinity-pending: root scheduling//<constraint>, as in fit-host-port-taken and unschedulable-taint; the message already names the Deployment fix.
  • hpa-target-no-cpu-request: root HPA, tier digest, tail 25m; same as one-hpa-bad-target. The HPA is the failing object; pods serve; metric grace is 10m.
  • pv-released-blocks-claim: root kept (PV); no change.
  • node-no-execute-taint-evicts: quiet; a deliberate maintenance taint whose pods are ready elsewhere within a minute is not a failure.
  • deployment-paused-with-pending-edit: tier digest, tail 20m; the pause is deliberate and replicas serve.
  • node-pleg-not-healthy, node-clock-skew-cert-rejected: tier page; a NotReady node pages (node-lost rule; same as kubelet-cert-expired, node-cni-not-ready).
  • crd-conversion-webhook-gone: root spelled customresourcedefinition//... (kwatch's id for the built-in kind); added the empty EndpointSlice the controller always keeps.
  • network-packet-loss-to-node: root unknown; nothing in cluster state (no node condition, event or policy) names n3, so naming it would be a guess. Fixture gets the Service ClusterIP the callers dial.
  • pod-stuck-terminating-finalizer: deletion timestamp backdated 40m as its description says.
  • release-image-pull-deadline: pull deadline after 3m (kubelet runtime request timeout is 2m by default), was 8m.
  • cron-active-deadline-exceeded: hourly schedule aligned to the start minute, deadline 120s.

New user-facing wording (designer review)

  • "Node X cannot set up pod networking: an address or route it assigns is already in use."
  • "Deployment is paused with a pod template edit that has not rolled out, for N."
  • "Conversion webhook backend X has no ready pods." / "Reads of its resources that need a conversion fail."
  • " is already bound to a different claim." / " is not a registry login."
  • "Node is low on process IDs" (was "running out of", flagged by the no-prediction check).

Scorecard

Before (902af29): 25/30 gates; correct root 96.8%; wrong root at likely+ 5; first message wrong root 5; must-not-blame 3; over budget 1; notify first message max 11m15s.
After: 30/30 gates; correct root 99.1%; wrong high-confidence 0/289; wrong stated 0; must-not-blame 0; over budget 0; notify first message max 4m15s. ratchet.txt and SCORECARD.md regenerated; make verify passes.

Runtime behaviour beyond the engine is unchanged, so no e2e label.

Designer wording applied

Conversion: "Requests for its resources that need version conversion fail."; paused: "Deployment has been paused for N with a pod template edit that has not rolled out." plus a resume step (kubectl rollout resume deploy/X -n NS); CNI: "an IP address or route it assigns is already in use"; unknown cause: only when the lead's own error is a network timeout to an address and the node, registry and services were checked, the message now reads "No cause found in cluster state: ... Its error is a network timeout to ADDR, so check the path to that address." The generic unknown wording is unchanged. Goldens changed by this: crd-conversion-webhook-gone, deployment-paused-with-pending-edit, network-packet-loss-to-node, node-cni-ip-conflict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant