feat(domain-events): add transactional event outbox, idempotent consumers, retries, ordering, and dead-letter handling #742

Parent epic: #738
Depends on: #739
Related completed foundations: #681, #685, #695, #697, #700

Context

AWCMS-Mini already publishes AsyncAPI contracts and several modules implement their own outbox/queue patterns, including sync, email, and social publishing. Those implementations are valuable precedents but do not yet provide one generic, provider-neutral domain-event runtime for cross-module collaboration, reporting projections, and future service extraction.

The base needs a reusable runtime without forcing an external broker and without replacing domain-owned business rules.

Objective

Provide a transactional, versioned domain-event runtime with an outbox, consumer registry/checkpoints, idempotent delivery, bounded retry/backoff, dead-letter handling, replay, and explicit ordering semantics.

Scope

• Define a versioned event envelope containing only bounded metadata such as:
  • event ID/type/version;
  • occurred/recorded timestamps;
  • tenant ID where applicable;
  • aggregate type/ID/version or ordering key;
  • correlation/causation IDs;
  • producer module and schema reference;
  • minimal validated payload.
• Add transactional outbox persistence written in the same database transaction as the source state change.
• Add a static consumer registry owned by reviewed source code.
• Add consumer checkpoints/delivery attempts with idempotency and bounded state.
• Implement claim/dispatch/finalize using the shared worker runner, locks, batching, timeouts, cancellation, metrics, and safe logging.
• Add retry classification, exponential backoff with jitter, maximum attempts, dead-letter state, operator-safe replay, and pause/resume.
• Define ordering guarantees per aggregate/order key without claiming global total ordering.
• Keep an in-process/PostgreSQL default implementation; define an optional broker adapter port for future use without making it required.
• Integrate AsyncAPI parity so runtime event types/versions cannot silently drift from published contracts.
• Provide at least two representative consumers:
  • one same-process cross-module consumer;
  • one reporting/read-model projection consumer or test fixture.
• Add observability for outbox lag, oldest pending age, dispatch outcome, retry rate, consumer lag/checkpoint, and DLQ count with low-cardinality labels.

Security, privacy, and transaction requirements

• Event is visible for dispatch only after the source transaction commits.
• Consumer failure never rolls back the already-committed source transaction.
• Duplicate delivery cannot duplicate side effects; consumers must use event ID/idempotency.
• No provider/broker call occurs inside a database transaction.
• Tenant context and RLS are enforced for tenant-owned events/deliveries.
• Payloads are minimized and schema-validated; secrets, credentials, raw tokens, and unnecessary PII are prohibited.
• Replay is permission-gated, reason-required, idempotent, and audited.
• Dead-letter inspection returns safe metadata and redacted payload projections only.

Out of scope

• Kafka/RabbitMQ/NATS as a mandatory dependency.
• Event sourcing of all application state.
• Replacing module-specific external-provider job models when their semantics differ.
• Distributed transactions or exactly-once delivery claims.

Acceptance criteria

• Source state and outbox record commit atomically.
• Rolled-back source transactions produce no dispatchable event.
• Duplicate dispatch and worker restart do not duplicate consumer side effects.
• Consumer failure leaves source state committed and moves delivery through retry/backoff to DLQ when exhausted.
• Ordering per declared aggregate/order key is tested; unrelated keys can progress independently.
• Replay requires permission, idempotency, reason, and audit, and cannot replay an incompatible schema silently.
• External broker is optional; offline/LAN operation works with PostgreSQL/in-process defaults.
• Runtime registry and AsyncAPI event types/versions pass bidirectional parity checks.
• Multi-tenant RLS/ABAC negative tests prove isolation.
• Metrics and health checks expose lag/failure without high-cardinality or PII labels.
• Unit, PostgreSQL integration, concurrency, retry, crash/restart, ordering, replay, DLQ, and redaction tests pass.
• OpenAPI/AsyncAPI, ERD/data dictionary, threat model, jobs/runbook, module docs, repository inventory, and changeset are updated.
• bun run check and a worker interruption/resilience scenario pass.