feat(tenant-provisioning): add idempotent provisioning workflow, compensation, reconciliation, and readiness #872

Parent epic: #868
Depends on: #869, #870, #871

Context

A SaaS tenant cannot be created safely by one long HTTP transaction. Provisioning may include tenant bootstrap, owner identity, default office/profile, module preset, entitlement assignment, public domain/subdomain, storage/configuration, and optional provider setup. Partial failure must be resumable and reconcilable.

Objective

Implement a provider-neutral, idempotent tenant-provisioning orchestration that reuses existing setup, workflow, domain-event, module-management, tenant-domain, audit, and integration patterns.

Scope

• Add admitted module/extension descriptor, permissions, jobs, events, and admin navigation.
• Define provisioning request, step, attempt, result, error classification, compensation, and reconciliation records.
• Implement a versioned provisioning plan/step registry contributed at build time.
• Provide commands to request, start/resume, cancel when safe, retry, and reconcile provisioning.
• Reuse existing tenant/setup mechanisms rather than duplicating tenant, owner, module, domain, or profile creation logic.
• Support core steps such as:
  • tenant record/bootstrap;
  • owner/platform relationship setup;
  • plan/entitlement assignment;
  • module preset application;
  • default configuration/locale;
  • optional subdomain/domain request;
  • derived-application contributed steps through capability ports.
• Use outbox/domain events for asynchronous steps and provider work.
• Add bounded retries, lock/lease ownership, idempotency, checkpoints, pause/resume, and manual intervention state.
• Add reconciliation that compares desired versus actual state without destructive auto-fix by default.
• Add API/OpenAPI, AsyncAPI, admin status/timeline UI, audit, metrics, and alerts.

Out of scope

• Direct provider calls in the source transaction.
• Hardcoded Cloudflare, R2, payment, email, or application-specific provisioning logic.
• Deleting tenant data as generic compensation after tenant becomes active.
• Cross-database sharding or tenant-placement automation.

Security and integrity requirements

• Provisioning commands are platform-operator only and default-deny.
• Idempotency key/hash binds tenant/request identity and immutable provisioning inputs.
• Concurrent runs for the same target are serialized or conflict deterministically.
• Provider secrets are references only and never appear in step payloads/logs.
• Step inputs/outputs are minimized and redacted; no password or token persistence.
• Compensation is explicitly classified as reversible, manual, or forbidden.
• Cross-tenant step execution is blocked through RLS and server-derived context.
• A failed or canceled run cannot leave an active tenant with missing mandatory security controls without a visible blocked status.

Acceptance criteria

• The same provisioning request can be replayed safely without duplicate tenant/resources.
• A failure after several completed steps can resume from a durable checkpoint.
• Concurrent provisioning for the same target cannot create duplicate tenants/owners/domains.
• Optional provider failure yields retry/manual-intervention state rather than transaction rollback of committed core state.
• Desired-versus-actual reconciliation identifies drift and offers safe operator actions.
• Derived applications can contribute reviewed steps without editing base orchestration.
• Cancellation/compensation follows documented safety classification.
• Timeline, audit, events, metrics, and safe errors expose enough evidence for operations.
• LAN/offline provisioning works with all online/provider steps absent or disabled.

Testing

• Unit tests for state transitions, plan validation, compensation classification, and error handling.
• PostgreSQL integration tests for leases/locking, idempotency, resume, RLS, and event same-commit behavior.
• Failure injection at every step boundary.
• Concurrency tests for duplicate requests and worker restarts.
• Derived-step fixture contract test.
• E2E operator request → failure → resume → active tenant.
• Security test for secret/PII redaction and cross-tenant denial.

Documentation

• ADR link, provisioning state/sequence diagrams, ERD, OpenAPI/AsyncAPI, step contribution guide, operator runbook, rollback/compensation matrix, backup/restore impact, and changeset.

Dependencies

Coordinates with tenant lifecycle #873 and operational reporting #880.