Fix concurrent ephemeral-blob loss#5659
Merged
Merged
Conversation
1083c95 to
962a78e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Example of the old bug:
Request A starts. It does:
Blob.pushScope exeStateStack:
[ scopeA ] <- top
Request B starts before A finishes. It also does:
Blob.pushScope exeStateStack:
[ scopeB ] <- top
[ scopeA ]
Request A creates an ephemeral blob.
The old code did not remember "the scope that belongs to Request A." Instead, it always attached new blobs to whatever scope was currently on top of the shared stack. But Request B had already started, so B's scope was now on top. That means A's blob was accidentally registered under B’s scope.
scopeB tracks idA // wrong
scopeA tracks nothing
Request B finishes first.
Blob.popScope exeStateThis pops scopeB and deletes every blob ID tracked by it. That deletes idA, even though Request A is still running. blobStore no longer has idARequest A later tries to read or promote its blob. But idA was deleted when B finished, so it fails with:
ephemeral blob not foundThe core issue was that concurrent requests shared one ExecutionState.blobScopes stack. A stack only works if scopes exit in exact last-in-first-out order.
Concurrent HTTP requests don't guarantee that, so one request could accidentally put its blob into another request's top scope.
After:
DBlob(Ephemeral { id; bytes })carries the bytes directly inside the value. So as long as the Dval exists, the bytes exist. There is no shared ephemeral store and no UUID lookup that can go stale.