Repository navigation
feat(task_queue_redis): extend Redis message claim during resumption replay - #1007
Merged
Merged
Conversation
There was a problem hiding this comment.
Code Review
This pull request implements message lease extension for the scheduling resumption stream by introducing a ClaimExtender interface and integrating it into the ResumptionWorker and Redis consumer client. Feedback on these changes highlights a high-severity risk of a type assertion panic and goroutine leak in the message processing loop, as well as a potential nil pointer dereference if the claimExtender is not initialized.
andrzejfudro
force-pushed
the
feat/redis-extend-claim
branch
from
September 29, 2026 10:03
395a799 to
5d6e385
Compare
andrzejfudro
changed the base branch from
feat/ess-replay-backlog-throttling
to
main
September 29, 2026 10:03
…replay Implement ExtendClaim on the Redis consumer. It resets the message's PEL idle time via XCLAIM (MinIdle 0) and signals processMessage, through a channel carried in the task context, to reset the local execution timer. Long-running handlers are then neither reclaimed by Redis nor timed out locally. The ESS resumption worker now calls ExtendClaim after every replayed batch and before each backlog throttle pause, replacing the related TODOs. Failures are logged but not fatal, because ResumptionWorkerID fencing already prevents duplicate replay if another worker takes over. TAG=agy CONV=e5e10511-3aa3-4e3d-9c89-4cf62fcb98f5
andrzejfudro
force-pushed
the
feat/redis-extend-claim
branch
from
September 29, 2026 12:56
5d6e385 to
8012bfa
Compare
andrzejfudro
marked this pull request as ready for review
September 29, 2026 13:11
andrzejfudro
enabled auto-merge
September 29, 2026 13:21
janzak-dev
approved these changes
Sep 29, 2026
Collaborator
There was a problem hiding this comment.
Please add a corresponding test case for the extend claim to validate on a real Redis whether it works properly https://github.com/google/xmc/blob/main/internal/task_queue_redis/validator/validator.go, can be done in a follow up PR
andrzejfudro
disabled auto-merge
September 29, 2026 15:20
andrzejfudro
added a commit
that referenced
this pull request
Sep 30, 2026
Adds TestExtendClaim to the real-Redis validator. The handler runs longer than both ProcessTimeout and AbandonedMessageTimeout while calling ExtendClaim more often than ProcessTimeout. The message must be processed exactly once and ACKed: no timeout retry, no cleanup reclaim, empty DLQ. Follow-up to #1007 requested in review. TAG=agy CONV=e5e10511-3aa3-4e3d-9c89-4cf62fcb98f5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Author note
This is coming from the design doc. The resumption process can take a long time in theory so to support that I extend redis client with option to extend the claim on the message.
Summary
Implements
ExtendClaimfor the Redis task queue consumer and calls it from the ESS resumption worker, so a long batch replay keeps its trigger message leased (b/559548459, Task 17).Key Changes
redisclient.ExtendClaim(ctx)XCLAIM(MinIdle: 0).processMessageover a buffered channel carried in the task context, which resets the localProcessTimeouttimer.Consumerinterface /FakeClientExtendClaimis added to the interface.redisclient.ClientinterfacebaseManagernow embeds aClientinterface listing the 15 go-redis commands the package uses, instead of the concrete*redis.Client.*redis.Clientstill satisfies it, so no caller changes; theClientfield name is unchanged.consumer_test.gouse an in-memory fake instead of a Redis server.ClaimExtenderdependency.ExtendClaimis called after every replayed batch and before each backlog throttle pause (DD §3).TODO(pausescheduling)markers.resumption.NewRedisConsumerbuilds a single Redis consumer that serves as both the trigger-stream consumer and the worker'sClaimExtender.startResumptionConsumersoNewProdstays under the gocyclo limit.Assumptions
WithMsgID,WithWorkerID,WithExtendChan, …). This PR stores one privateclaim{msgID, workerID, extendChan}value in the context instead. Nothing outsideredisclientneeds those values, so the exported surface isn't needed.XCLAIMstill succeeds. We rely onResumptionWorkerIDfencing in Spanner to stop duplicate replay rather than checking the claim result.ExtendClaimis logged (Warn), not fatal, for the same fencing reason.Verification
bazel test //cmd/experiment_state_server/... //internal/task_queue_redis/redisclient:redisclient_test: all pass.XCLAIMsent withMinIdle: 0;./.githooks/pre-commitpasses.TAG=agy
CONV=e5e10511-3aa3-4e3d-9c89-4cf62fcb98f5