files: use DB.locking config option to improve input performance #1004
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Please forgive my legend labeling. Looks like we get some better memory utilization: Marginal/negligible improvement of CPU utilization: And virtually no impact on throughput: |
33bafba to
e8d146b
Compare
braydonk
approved these changes
Nov 30, 2022
Fluent Bit has this option available in the `tail` input plugin. From looking at the upstream code, it looks like should help to reduce the number of syscalls on every commit to the DB but at the price of locking the access to the database file to third party programs. But for us, that trade-off seems okay? We should soak tesk/performance test this to see if we get any visible benefit. Signed-off-by: Ridwan Sharif <ridwanmsharif@google.com>
e8d146b to
6e5243b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fluent Bit has this option available in the
tailinput plugin. From looking at the upstream code, it looks like should help to reduce the number of syscalls on every commit to the DB but at the price of locking the access to the database file to third party programs. But for us, that trade-off seems okay?We should soak test/performance test this to see if we get any visible benefit.