Fix stalled runners and unreliable scheduled execution - #405

Description

A runner whose database registration disappeared could remain online, return a successful health response, and continue enqueueing scheduled jobs while claiming no work. This change detects a missing or retired registration and exits so the host can restart the worker with a new registration. Readiness now requires a recent confirmed heartbeat.

• Serialize webhook, Realtime, and fallback queue polling; fill available capacity and claim the next job as soon as a slot is released.
• Bound REST database requests to 15 seconds, prevent overlapping heartbeat/recovery requests, and allow two minutes before marking a runner stale.
• Retry recovery for already-dead runners and preserve ownership of unfinished work during shutdown.
• Evaluate cron expressions and template times in UTC. Add TRIGGER_SCHEDULER_ENABLED so operators can select one scheduler, and report unsuccessful Edge Function wake-up requests.
• Record structured errors and completion timestamps when team execution is blocked by plan requirements.
• Add Railway/Docker readiness checks, fix the Compose build context and Render encryption variable name, enable manual CI execution, and document diagnosis and rollout in docs/runner-operations.md.

Related Issue

No linked issue; follows the runner and scheduling reliability investigation.

Type of Change

• Bug fix
• Documentation update
• Test update

How Has This Been Tested?

• Runner: 41 tests across 6 files; TypeScript build passed. Includes 16 new regressions for registration loss, recovery retries, queue draining, readiness, rejected HTTP handlers, and UTC/catch-up evaluation.
• Frontend: 3 tests passed; production TypeScript/Vite build passed.
• npm run lint: zero errors; one existing Fast Refresh warning in DateRangePicker.tsx.
• Compose configuration validated with dummy credentials; git diff --check passed.

Docker image execution and the Deno Edge Function were not tested locally. No live model execution or output delivery was performed during validation.

Rollout and limitations

Historical queued jobs were cancelled separately with private snapshots and retained audit history; unassigned pending tasks were preserved. Both execution queues were verified empty, recurring triggers are paused, and the database scheduler is disabled before this push. Private snapshots and local Supabase metadata are excluded from the PR.

Keep schedules paused until the deployed worker has a registered, advancing heartbeat and a selected execution has been verified. Use one replica and one scheduling source; with the always-on runner, leave the database scheduler disabled. Resolve invalid team plan/configuration requirements before restoring affected schedules. Review the 48-hour catch-up window when reenabling triggers so missed maintenance occurrences are handled deliberately.

This does not add an atomic shared scheduler claim or exactly-once execution. Recovery can retry interrupted work, and external writes still need idempotency. Railway's health check gates deployment readiness; it is not continuous automatic recovery. The Edge Function source change requires a separate Supabase deployment if that scheduler is used.

Checklist

• Coding conventions followed; new test files include AGPL headers.
• Lint, tests, and builds pass without new TypeScript errors.
• Regression tests and operational documentation added.

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}