storage: use single batch find for a fetch #446
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
06062ae to
4dcecf1
Compare
The fetch path had two batch finds, one for the DelayedFetch.tryCompleteDiskless and seconds at DelayedFetch.onComplete. Collect the batches at tryCompleteDiskless and reuse at onComplete. This reduces the database queries to batch coordinator.
4dcecf1 to
5945ae6
Compare
|
This PR is being marked as stale since it has not had any activity in 90 days. If you If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact). If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The fetch path had two batch finds, one for the DelayedFetch.tryCompleteDiskless and seconds at DelayedFetch.onComplete.
Collect the batches at tryCompleteDiskless and reuse at onComplete. This reduces the database queries to batch coordinator.
The current code in main can have more batches at the onComplete as a new find batch query is run after the initial call at tryCompleteDiskless.
Comparisons to current main: 5875f03
On graphs the main is the left most run and right is from this PR.
Benchmark consumer fetch latency:
Find batches query time and rate:
Fetch rate:
