Previous incidents

We saw elevated database load on our shared production environment beginning around 05:00 UTC, which caused some multi-step workflows to experience scheduling delays and increased API/dashboard latency for affected tenants.

We reduced connection pressure on the shared database and latency has returned to normal. We’re continuing to monitor closely.

Between approximately 05:00–10:00 UTC on July 24, the shared cloud shard in our us-west-2 region experienced elevated database lock contention, which caused higher API and scheduling latency and some intermittent task-dispatch delays for tenants on that shard.

This was isolated to a single shard in us-west-2 (cloud). Other regions and shards were not affected. We have no evidence of dropped workload, and all assignments were eventually completed.

We are continuing to investigate and monitor.

We're continuing to monitor the fix. We've been seeing significant recovery since about 35 minutes ago (15:15 UTC).

If you're seeing tasks that are stuck during execution, cancelling and replaying will likely get them moving again.

We have identified the source of the latency and have pushed initial configuration changes to mitigate. We will continue to monitor for degraded performance.

We’re back! Hatchet Cloud region us-west-2 should be up and running. Thanks for bearing with us.