Skip to content

✨ more accurate root cause, dashboard improvements - #617

Merged
abahmed merged 2 commits into
mainfrom
enhance-kwatch7
Nov 18, 2026
Merged

abahmed merged 2 commits into
mainfrom
enhance-kwatch7

Conversation

@abahmed

@abahmed abahmed commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

More accurate root cause, clearer evidence, and dashboard improvements.

Root cause and detection

  • Newer restart semantics (restartPolicyRules, RestartAllContainers, native sidecars, short crashLoopBackOff.maxContainerRestartPeriod) are no longer read as crashes; messages name a failing sidecar or init container.
  • Failed or stuck Helm releases (Pro): latest revision failed, or pending/uninstalling for 15 min, via a metadata-only watch of release Secrets (no new permission). Notify level, never paged; steps include helm rollback.
  • Memory context on OOM kills and evictions, observed sizing at failure time, and CPU at its limit as probe evidence.
  • A link that ends (Service stops routing, NetworkPolicy stops covering, a mount removed) is a change kwatch can blame.
  • StatefulSets and DaemonSets get the same rollout undo suggestion as Deployments.

Evidence and messages

  • A change links to its source: GitOps app (new app.gitOpsLinks), commit from org.opencontainers.image.source/.revision annotations, and image.
  • Issues without a sure cause list what kwatch checked and found fine, and what it could not check (dashboard Why?, Teams, email, webhook evidence.checked).
  • "What changed just before" says why each change is listed.
  • The state file gains two optional fields (read as empty from older files).

Self-health and dashboard

  • Downtime is visible: hours kwatch was not running are marked on the Overview and in timelines; heartbeat always shown in /health and /api/status.
  • kwatch sends one low-priority "degraded" message when its own parts stay degraded for 10 minutes, and "back to normal" on recovery (app.disableSelfAlerts turns it off).

Providers and chart

  • New provider: ServiceNow (alert.servicenow), one incident per kwatch incident via the Table API, found again by correlation_id.
  • Optional priorityClass in the Helm chart (priorityClass.create).

Tests and docs

  • New scenario fixtures and golden messages, provider contract tests, docs and catalogs regenerated.

@abahmed abahmed added the e2e Run the real-cluster E2E suite on this pull request label Nov 18, 2026
The scenario library has no production code and does not use -coverpkg, so covering it added no coverage but cost most of the Verify job. Coverage now runs over every other package, and the scenarios run under -race with the alert-quality gates in a separate Scenarios job beside Verify.
@abahmed
abahmed merged commit b464523 into main Nov 18, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

e2e Run the real-cluster E2E suite on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant