Skip to content

A recipe watch that expires reopens quietly - #69

Merged
gedaiu merged 1 commit into
mainfrom
watch-timeout-is-not-an-error
Nov 9, 2026
Merged

gedaiu merged 1 commit into
mainfrom
watch-timeout-is-not-an-error

Conversation

@gedaiu

@gedaiu gedaiu commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

The kube client (v2) ends every watch request after 30 seconds with a TimeoutError. watchRecipes reopened it and passed the error to the claim loop's onError, which logs "the floor could not be reached; trying again". On internal-cluster that line appeared about 12,000 times in three days, one every 15 to 20 seconds from the two watches, while the claim loop reported "alive" every minute and the floor API answered in milliseconds.

An expiry now reopens without a log line, and any other watch error is logged as "the recipe watch dropped; reopening". isWatchExpiry is tested.

🤖 Generated with Claude Code

The kube client ends every watch request after 30 seconds. The recipe watcher
reopened it and logged each end as "the floor could not be reached", about
12,000 times in three days on internal-cluster, which read as the floor being
down while it answered every claim. An expiry now reopens without a line, and a
real drop says it is the recipe watch.
@lore-agent

lore-agent Bot commented Nov 9, 2026

Copy link
Copy Markdown

Lore is reviewing this PR — 46a3749d-b225-4c0a-876b-63964dd1da67.

Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.

@gedaiu
gedaiu merged commit 1f3b63f into main Nov 9, 2026
15 checks passed
@gedaiu
gedaiu deleted the watch-timeout-is-not-an-error branch November 9, 2026 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant