Tags: obot-platform/obot
Tags
fix: coalesce auth provider group refreshes and back off after failur… …es (#7928) (#7969) ensureGroups runs inside request authentication, through UserDecorator.AuthenticateRequest, so it executes on every authenticated request. The 10 minute check window was the only thing between request volume and the auth provider, and only a successful refresh advanced it. An Okta org rate limiting group lookups therefore hit two failure modes: 1. bursts of identical calls from every request in flight when the window 2. a retry loop that never let the provider recover, with the error propagating out of authentication so logins failed outright. In-process concurrent refreshes for one identity now share a single call through singleflight, keyed on the ID sent to the provider. An identity whose refresh just failed is served the groups already in the database until groupFailureCooldown passes, which is the same data the request would have been served a moment earlier. The shared refresh runs on a detached context so a leader that disconnects does not cancel it for everyone waiting behind it, bounded by its own 10 second timeout rather than the 30 second one the admin group endpoints use. Both are per-process, so separate replicas can still compete to update an identity concurrently; before this change the slower fetch used to commit last and win, restoring memberships the faster one had just removed. persistGroups now advances auth_provider_groups_last_checked with a compare-and-set against the value its refresh started from, in the same transaction that reconciles memberships. Now and a refresh that loses that race reports the stored groups rather than its own. encryptAndUpdateIdentity and UpdateProfileIfNeeded omit that column so they cannot rewind it. (cherry picked from commit 6f81dac) Signed-off-by: Nick Hale <4175918+njhale@users.noreply.github.com>
fix: coalesce auth provider group refreshes and back off after failur… …es (#7928) (#7969) ensureGroups runs inside request authentication, through UserDecorator.AuthenticateRequest, so it executes on every authenticated request. The 10 minute check window was the only thing between request volume and the auth provider, and only a successful refresh advanced it. An Okta org rate limiting group lookups therefore hit two failure modes: 1. bursts of identical calls from every request in flight when the window 2. a retry loop that never let the provider recover, with the error propagating out of authentication so logins failed outright. In-process concurrent refreshes for one identity now share a single call through singleflight, keyed on the ID sent to the provider. An identity whose refresh just failed is served the groups already in the database until groupFailureCooldown passes, which is the same data the request would have been served a moment earlier. The shared refresh runs on a detached context so a leader that disconnects does not cancel it for everyone waiting behind it, bounded by its own 10 second timeout rather than the 30 second one the admin group endpoints use. Both are per-process, so separate replicas can still compete to update an identity concurrently; before this change the slower fetch used to commit last and win, restoring memberships the faster one had just removed. persistGroups now advances auth_provider_groups_last_checked with a compare-and-set against the value its refresh started from, in the same transaction that reconciles memberships. Now and a refresh that loses that race reports the stored groups rather than its own. encryptAndUpdateIdentity and UpdateProfileIfNeeded omit that column so they cannot rewind it. (cherry picked from commit 6f81dac) Signed-off-by: Nick Hale <4175918+njhale@users.noreply.github.com>
Backport: wake kinm watches with LISTEN/NOTIFY and refresh them on pr… …omotion (#7736) (#7763) With kinm's LISTEN/NOTIFY change a standby's watches wait to be told about writes instead of polling for them. A replica running an older version announces nothing, which the first rolling upgrade guarantees for the length of the rollout, so a standby's cache can be up to a poll interval behind. That is harmless in a standby and a problem the moment it becomes the leader and starts acting on it. kinm's Factory.Refresh wakes every watch so each lists again. The post start hook runs on promotion, so that is where it is called. The Factory was already returned by storage.Start, so it is now kept on Services too. Also pins kinm to 2231aa1 and documents KINM_DB_WATCH_POLL_SECONDS, KINM_DB_NOTIFY_DEBOUNCE_MILLISECONDS and KINM_DB_DISABLE_NOTIFY in the server configuration table and chart values. The ADR from the original commit is left on main. (cherry picked from commit 92a1bbe) I also fixed a linting error due to a previous dep bump that made a function deprecated. It's fixed on main. we dont see it here bc the full ci doesnt run (maybe it should). I just didnt want it to bite me elsewhere
Hold the controller leader lock in a SQL table instead of a Lease (#7727 ) The obot-controller election now uses nah's SQL lock, one row in a leader_lock table updated in place, instead of a Lease in kinm's versioned store, where every renew appended a row version and made the Lease the most expensive object in the database. The election algorithm, TTL, and retry period are unchanged, as is the obot-local-controller election against the real cluster. WithLegacyLeaseLock keeps one leader during the rolling update that ships this change: new replicas follow the Lease that old replicas still hold and create the row only after the last old replica releases it. Remove that call in the following release. Bumps nah to 636420908be9 (obot-platform/nah#32). (cherry picked from commit d957c3a) Cherry-picked onto v0.25-hotfixes with one adaptation: the legacy lease lock gets restConfig directly, because leaderElectionRESTConfig does not exist on this branch.
PreviousNext