Skip to content

Decrease fluent-bit buffer_max_size to 2M for smaller outgoing requests. #867

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Sep 28, 2022
Merged

Decrease fluent-bit buffer_max_size to 2M for smaller outgoing requests. #867

merged 5 commits into from
Sep 28, 2022

Conversation

@igorpeshansky
Copy link
Contributor

@igorpeshansky igorpeshansky commented Sep 21, 2022

Description

Some outgoing requests end up being over the API limit. This PR reduces the chance of that happening by lowering the maximum size of the accumulated buffer.

Related issue

http://b/244498953

How has this been tested?

Only automatic unit tests.

Checklist:

  • Unit tests
    • Unit tests do not apply.
    • Unit tests have been added/modified and passed for this PR.
  • Integration tests
    • Integration tests do not apply.
    • Integration tests have been added/modified and passed for this PR.
  • Documentation
    • This PR introduces no user visible changes.
    • This PR introduces user visible changes and the corresponding documentation change has been made.
  • Minor version bump
    • This PR introduces no new features.
    • This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
    • This PR bumps the version.

@igorpeshansky igorpeshansky requested review from a team and hsmatulis and removed request for a team September 21, 2022 20:37
qingling128
qingling128 approved these changes Sep 26, 2022
View reviewed changes
Copy link
Contributor

@qingling128 qingling128 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@igorpeshansky
Copy link
Contributor Author

igorpeshansky commented Sep 27, 2022

Thanks. The Windows integration test failure was a flake. Merging.

@igorpeshansky igorpeshansky merged commit f30948b into master Sep 28, 2022
@igorpeshansky igorpeshansky deleted the igorpeshansky-decrease-buffer-max-size branch September 28, 2022 19:47
@sophieyfang
Copy link
Contributor

sophieyfang commented Oct 3, 2022 •
edited
Loading

Shall we also add a test case with a log entry really big and see it didn't get the API denied response? 🤔

@jefferbrecht jefferbrecht mentioned this pull request Oct 3, 2022
Merged
9 tasks
jefferbrecht added a commit that referenced this pull request Oct 3, 2022
@jefferbrecht
Revert "Decrease fluent-bit buffer_max_size to 2M for smaller outgoin…
847b69d 
…g requests. (#867)"

This reverts commit f30948b.
jefferbrecht added a commit that referenced this pull request Oct 3, 2022
@jefferbrecht
Update newer goldens after reverting #867
ccf7e68
jefferbrecht added a commit that referenced this pull request Oct 3, 2022
@jefferbrecht
Revert #867 (#883)
37e434e 
* Revert "Decrease fluent-bit buffer_max_size to 2M for smaller outgoing requests. (#867)"

This reverts commit f30948b.

* Update newer goldens after reverting #867
@igorpeshansky
Copy link
Contributor Author

igorpeshansky commented Nov 30, 2022

@sophieyfang:

Shall we also add a test case with a log entry really big and see it didn't get the API denied response? 🤔

I think the issue here is that the accumulated size of the log entries exceeds the API request limit. A single entry will still be rejected, as it'd be over the entry size limit (256k). While we can try to simulate the "multiple entries add up to a request that's too large" scenario, it seems very fragile and hard to reproduce, so the test will likely be flaky.

@igorpeshansky igorpeshansky mentioned this pull request Dec 2, 2022
Merged
9 tasks
igorpeshansky added a commit that referenced this pull request Dec 2, 2022
@igorpeshansky
Revert "Revert #867 (#883)"
93c457c 
This reverts commit 37e434e.
igorpeshansky added a commit that referenced this pull request Dec 2, 2022
@igorpeshansky
Decrease fluent-bit buffer_max_size to 2M for smaller outgoing reques…
668a037 
…ts (roll-forward #867). (#1013)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@qingling128 qingling128 qingling128 approved these changes

@hsmatulis hsmatulis Awaiting requested review from hsmatulis hsmatulis was automatically assigned from GoogleCloudPlatform/stackdriver-reviewers

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@igorpeshansky @sophieyfang @qingling128