Skip to content

/readyz asks the database on a connection of its own - #58

Merged
gedaiu merged 1 commit into
mainfrom
fix/readyz-own-connection
Oct 27, 2026
Merged

gedaiu merged 1 commit into
mainfrom
fix/readyz-own-connection

Conversation

@gedaiu

@gedaiu gedaiu commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Closes #55. Both api replicas failed /readyz about forty times a day and dropped out of the Service, so lore's reads of the floor failed with fetch failed.

/readyz was already a plain select 1, but it took its connection from the pool every request shares. A /metrics scrape (about two seconds, a dozen queries at once) or a few slow reads of a single run (#52) can hold all ten, and the wait then outlasted the one-second probe timeout the chart left at the Kubernetes default. Both replicas share that load, so they left the Service together while still working. At idle /readyz answers in under 100 ms, which is why it only fails under load.

The check now runs on a pool of its own with one connection and a two-second limit, so it never queues behind requests. The chart sets both api probes to wait three seconds, so the floor answers 503 itself before the kubelet gives up. The liveness probe gets the same, since /healthz shares an event loop with a two-second /metrics render on half a CPU.

A new test takes every connection of the request pool and still gets ready from /readyz; it hung before the change. The store and api suites, typecheck and lint pass, and helm template shows the three-second timeouts. I could not reproduce the failure under load on the cluster, so the cause comes from the code and the numbers above; the week without Unhealthy events after deploy is the confirmation.

Not changed here: /metrics holding the pool for its two seconds, the slow single-run read (#52), and the Postgres ECONNREFUSED that restarted one pod at startup.

🤖 Generated with Claude Code

Both api replicas failed their readiness probe about forty times a day.
/readyz was a cheap select 1, but it took its connection from the pool every
request uses, and a /metrics scrape or a few slow run reads held all ten for
longer than the kubelet's default one-second timeout.

The check now has a one-connection pool that gives up after two seconds, and
the chart sets both api probes to wait three.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@lore-agent

lore-agent Bot commented Oct 27, 2026

Copy link
Copy Markdown

Lore is reviewing this PR — 124a0338-58a5-4cb8-b0b0-5fbe6435857c.

Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.

@lore-agent lore-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lore review — Approved

Dedicates a 1-connection pool with 2s connect/query timeouts to /readyz and sets 3s Kubernetes probe timeouts so readiness checks no longer queue behind request pool saturation.

Must fix (0) · Consider (0) · Nits (0)

Reviewed by gemini-4-argon

Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.

Posted by floor, visit 6443771f-1b96-4292-9ad9-f36a10c752a1.

@gedaiu
gedaiu merged commit 8e7b9fb into main Oct 27, 2026
16 checks passed
@gedaiu
gedaiu deleted the fix/readyz-own-connection branch October 27, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

floor-api fails its readiness probe about 40 times a day

1 participant