Repository navigation
/readyz asks the database on a connection of its own - #58
Merged
Merged
Conversation
Both api replicas failed their readiness probe about forty times a day. /readyz was a cheap select 1, but it took its connection from the pool every request uses, and a /metrics scrape or a few slow run reads held all ten for longer than the kubelet's default one-second timeout. The check now has a one-connection pool that gives up after two seconds, and the chart sets both api probes to wait three. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Lore is reviewing this PR — 124a0338-58a5-4cb8-b0b0-5fbe6435857c. Submit a review that requests changes to have Lore push a fix for it. Comment |
There was a problem hiding this comment.
Lore review — Approved
Dedicates a 1-connection pool with 2s connect/query timeouts to /readyz and sets 3s Kubernetes probe timeouts so readiness checks no longer queue behind request pool saturation.
Must fix (0) · Consider (0) · Nits (0)
Reviewed by gemini-4-argon
Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.
Posted by floor, visit 6443771f-1b96-4292-9ad9-f36a10c752a1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #55. Both api replicas failed
/readyzabout forty times a day and dropped out of the Service, so lore's reads of the floor failed withfetch failed./readyzwas already a plainselect 1, but it took its connection from the pool every request shares. A/metricsscrape (about two seconds, a dozen queries at once) or a few slow reads of a single run (#52) can hold all ten, and the wait then outlasted the one-second probe timeout the chart left at the Kubernetes default. Both replicas share that load, so they left the Service together while still working. At idle/readyzanswers in under 100 ms, which is why it only fails under load.The check now runs on a pool of its own with one connection and a two-second limit, so it never queues behind requests. The chart sets both api probes to wait three seconds, so the floor answers 503 itself before the kubelet gives up. The liveness probe gets the same, since
/healthzshares an event loop with a two-second/metricsrender on half a CPU.A new test takes every connection of the request pool and still gets
readyfrom/readyz; it hung before the change. The store and api suites, typecheck and lint pass, andhelm templateshows the three-second timeouts. I could not reproduce the failure under load on the cluster, so the cause comes from the code and the numbers above; the week withoutUnhealthyevents after deploy is the confirmation.Not changed here:
/metricsholding the pool for its two seconds, the slow single-run read (#52), and the PostgresECONNREFUSEDthat restarted one pod at startup.🤖 Generated with Claude Code