feat(saas-operations): add control-plane projections, reconciliation jobs, SLOs, lifecycle, and incident evidence #880

Parent epic: #868
Depends on: #872, #873, #875, #876, #877, #879

Context

A control plane is not production-ready when state exists only in transactional tables and provider dashboards. Operators need reproducible projections, freshness, reconciliation, queue health, lifecycle controls, SLOs, alerts, and incident evidence across provisioning, entitlements, usage, subscriptions, invoices, and payments.

Objective

Add module-contributed reporting projections and operational controls that make SaaS control-plane state observable, reconcilable, capacity-aware, and maintainable.

Scope

• Register reporting projection descriptors and read models for at least:
  • tenant/provisioning status and aging;
  • lifecycle distribution and scheduled transitions;
  • effective entitlement/override inventory;
  • usage freshness, quota pressure, corrections, and reconciliation;
  • active/trial/past-due/suspended subscription counts;
  • invoice aging, credits, dunning, and payment mismatch;
  • provider health, webhook backlog, retries, DLQ, and reconciliation;
  • high-risk operator actions and support-access activity.
• Build incremental/event-driven projections with idempotent rebuild, freshness/staleness status, and source reconciliation.
• Add scheduled jobs for:
  • provisioning reconciliation;
  • entitlement expiry/revocation;
  • usage aggregation/rebuild;
  • renewal/invoice/dunning;
  • payment/provider reconciliation;
  • stale workflow/manual-intervention detection;
  • lifecycle transition processing.
• Add low-cardinality metrics, structured logs, trace/correlation propagation, SLO definitions, alert thresholds, and health/readiness output.
• Extend work-class registry, pool/capacity budgets, worker concurrency limits, and production preflight for control-plane workloads.
• Register high-volume tables with data-lifecycle descriptors, retention, partitioning evidence, archive, purge, legal hold, and restore considerations.
• Add safe operator exports and evidence packages without raw secrets/PII.
• Define RTO/RPO and provider-outage/degraded-mode expectations.

Out of scope

• Financial statutory reports, general-ledger statements, tax reports, or vertical product analytics.
• Unbounded high-cardinality metrics by tenant/resource/provider reference.
• Automatic microservice extraction, sharding, or distributed cache.

Security and integrity requirements

• Projections are derived/read-only and cannot become an authorization source without an explicit contract.
• Rebuild/reconciliation is idempotent, tenant-aware, and protected from cross-tenant mixing.
• Operational exports are permission-gated, bounded, audited, masked, and retention-controlled.
• Metrics/logs never include secrets, raw webhook tokens, full payment references, or unnecessary PII.
• Alert and health endpoints reveal safe status, not sensitive configuration values.
• Maintenance jobs use least-privilege worker roles and bounded batches/lock timeouts.

Acceptance criteria

• Every critical control-plane subsystem has freshness, backlog, failure, retry, and reconciliation visibility.
• Projections rebuild idempotently and reconcile to source records.
• Provider and worker outage scenarios produce controlled degraded state and actionable alerts.
• Production preflight validates connection/work-class/worker capacity for the declared instance count.
• High-volume tables have lifecycle descriptors and tested archive/purge/legal-hold behavior.
• Operator evidence/export is bounded, masked, and audited.
• SLOs and alerts use low-cardinality dimensions and have runbook links.
• Backup/restore rehearsal includes control-plane tables, projections, jobs, and provider references.
• Reporting never introduces direct cross-module shared-table writes.

Testing

• Unit tests for projection descriptors, freshness, reconciliation, SLO thresholds, and lifecycle registration.
• PostgreSQL integration tests for incremental projection, rebuild, RLS, job lease/concurrency, and purge/legal hold.
• Large-data/query-plan and representative load/soak/burst tests.
• Failure injection for DB saturation, worker crash, provider timeout, webhook backlog, and stale projection.
• Backup/restore and RTO/RPO rehearsal in non-production.
• Security tests for log/metric/export redaction and cross-tenant projection isolation.
• Mutation tests: stale projection or lost reconciliation mismatch must fail.

Documentation

• Operations dashboard specification, projection catalog, SLO/alert registry, job/runbook catalog, data-lifecycle matrix, capacity plan, incident response, backup/restore, RTO/RPO, provider outage guides, module READMEs, and changeset.

Dependencies

Provides evidence and readiness gates required by final pilot #881.