Skip to content

Restore connection-level limiter - #5372

Merged
michel-laterman merged 4 commits into
elastic:mainfrom
michel-laterman:fix/restore-conn-limiter
Aug 29, 2025
Merged

Restore connection-level limiter#5372
michel-laterman merged 4 commits into
elastic:mainfrom
michel-laterman:fix/restore-conn-limiter

Conversation

@michel-laterman

@michel-laterman michel-laterman commented Aug 28, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

What is the problem this PR solves?

Use of only the throttle middleware can result in OOM incidents under high load.

How does this PR solve the problem?

Restore connection level limiter to prevent OOM incidents. (removed in #4402)
This limiter is used in addition to the request-level throttle so that once
our in-flight requests reaches max_connections a 429 is returned, but if the
total connections the server uses is over max_connections*1.1 the server drops
the connection before the TLS handshake.

Design Checklist

  • I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
  • I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
  • I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool

@michel-laterman
michel-laterman requested a review from a team as a code owner August 28, 2025 19:45
@michel-laterman michel-laterman added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Aug 28, 2025
@michel-laterman michel-laterman added the backport-active-all Automated backport with mergify to all the active branches label Aug 28, 2025
@prodsecmachine

prodsecmachine commented Aug 28, 2025 •
edited
Loading

Copy link
Copy Markdown

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

@michel-laterman
michel-laterman force-pushed the fix/restore-conn-limiter branch from 59626ad to d3641d8 Compare August 28, 2025 19:46
blakerouse
blakerouse reviewed Aug 29, 2025
View reviewed changes

@blakerouse blakerouse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good, but unit tests for the listener would be great. Could you add those?

@elastic-sonarqube

elastic-sonarqube Bot commented Aug 29, 2025

Copy link
Copy Markdown

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

blakerouse
blakerouse approved these changes Aug 29, 2025
View reviewed changes

@blakerouse blakerouse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the added unit tests. Looks good.

@michel-laterman
michel-laterman merged commit 39199ef into elastic:main Aug 29, 2025
9 checks passed
@michel-laterman
michel-laterman deleted the fix/restore-conn-limiter branch August 29, 2025 21:50
@github-actions

github-actions Bot commented Aug 29, 2025

Copy link
Copy Markdown
Contributor

@Mergifyio backport 8.17 8.18 8.19 9.0 9.1

@mergify

mergify Bot commented Aug 29, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

backport 8.17 8.18 8.19 9.0 9.1

✅ Backports have been created

Details

mergify Bot pushed a commit that referenced this pull request Aug 29, 2025
@michel-laterman @mergify
Restore connection-level limiter (#5372)
3b90768 
Restore connection level limiter to prevent OOM incidents. (removed in #4402)
This limiter is used in addition to the request-level throttle so that once
our in-flight requests reaches max_connections a 429 is returned, but if the
total connections the server uses is over max_connections*1.1 the server drops
the connection before the TLS handshake.

(cherry picked from commit 39199ef)

# Conflicts:
#	NOTICE-fips.txt
#	NOTICE.txt
#	go.mod
#	internal/pkg/api/server.go
mergify Bot pushed a commit that referenced this pull request Aug 29, 2025
@michel-laterman @mergify
Restore connection-level limiter (#5372)
ba0c38a 
Restore connection level limiter to prevent OOM incidents. (removed in #4402)
This limiter is used in addition to the request-level throttle so that once
our in-flight requests reaches max_connections a 429 is returned, but if the
total connections the server uses is over max_connections*1.1 the server drops
the connection before the TLS handshake.

(cherry picked from commit 39199ef)

# Conflicts:
#	NOTICE-fips.txt
#	NOTICE.txt
#	go.mod
#	internal/pkg/api/server.go
This was referenced Aug 29, 2025
Merged
Merged
mergify Bot pushed a commit that referenced this pull request Aug 29, 2025
@michel-laterman @mergify
Restore connection-level limiter (#5372)
04176c3 
Restore connection level limiter to prevent OOM incidents. (removed in #4402)
This limiter is used in addition to the request-level throttle so that once
our in-flight requests reaches max_connections a 429 is returned, but if the
total connections the server uses is over max_connections*1.1 the server drops
the connection before the TLS handshake.

(cherry picked from commit 39199ef)

# Conflicts:
#	go.mod
#	internal/pkg/api/server.go
@mergify mergify Bot mentioned this pull request Aug 29, 2025
Merged
6 tasks done
mergify Bot pushed a commit that referenced this pull request Aug 29, 2025
@michel-laterman @mergify
Restore connection-level limiter (#5372)
5816d09 
Restore connection level limiter to prevent OOM incidents. (removed in #4402)
This limiter is used in addition to the request-level throttle so that once
our in-flight requests reaches max_connections a 429 is returned, but if the
total connections the server uses is over max_connections*1.1 the server drops
the connection before the TLS handshake.

(cherry picked from commit 39199ef)

# Conflicts:
#	NOTICE-fips.txt
#	NOTICE.txt
#	go.mod
#	internal/pkg/api/server.go
mergify Bot pushed a commit that referenced this pull request Aug 29, 2025
@michel-laterman @mergify
Restore connection-level limiter (#5372)
c6e79a8 
Restore connection level limiter to prevent OOM incidents. (removed in #4402)
This limiter is used in addition to the request-level throttle so that once
our in-flight requests reaches max_connections a 429 is returned, but if the
total connections the server uses is over max_connections*1.1 the server drops
the connection before the TLS handshake.

(cherry picked from commit 39199ef)

# Conflicts:
#	go.mod
#	internal/pkg/api/server.go
This was referenced Aug 29, 2025
Merged
Merged
michel-laterman added a commit that referenced this pull request Aug 29, 2025
@mergify @michel-laterman michel-laterman
[8.19](backport #5372) Restore connection-level limiter (#5377)
8123b43 
* Restore connection-level limiter (#5372)

Restore connection level limiter to prevent OOM incidents. (removed in #4402)
This limiter is used in addition to the request-level throttle so that once
our in-flight requests reaches max_connections a 429 is returned, but if the
total connections the server uses is over max_connections*1.1 the server drops
the connection before the TLS handshake.

(cherry picked from commit 39199ef)

# Conflicts:
#	go.mod
#	internal/pkg/api/server.go

* Fix merge

---------

Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>
Co-authored-by: michel-laterman <michel.laterman@elastic.co>
michel-laterman added a commit that referenced this pull request Aug 30, 2025
@mergify @michel-laterman michel-laterman
[9.1](backport #5372) Restore connection-level limiter (#5379)
9972f1a 
* Restore connection-level limiter (#5372)

Restore connection level limiter to prevent OOM incidents. (removed in #4402)
This limiter is used in addition to the request-level throttle so that once
our in-flight requests reaches max_connections a 429 is returned, but if the
total connections the server uses is over max_connections*1.1 the server drops
the connection before the TLS handshake.

(cherry picked from commit 39199ef)

# Conflicts:
#	go.mod
#	internal/pkg/api/server.go

* Fix merge

---------

Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>
Co-authored-by: michel-laterman <michel.laterman@elastic.co>
michel-laterman added a commit that referenced this pull request Aug 30, 2025
@mergify @michel-laterman michel-laterman
[8.17](backport #5372) Restore connection-level limiter (#5375)
18cb2e0 
* Restore connection-level limiter (#5372)

Restore connection level limiter to prevent OOM incidents. (removed in #4402)
This limiter is used in addition to the request-level throttle so that once
our in-flight requests reaches max_connections a 429 is returned, but if the
total connections the server uses is over max_connections*1.1 the server drops
the connection before the TLS handshake.

(cherry picked from commit 39199ef)

# Conflicts:
#	NOTICE-fips.txt
#	NOTICE.txt
#	go.mod
#	internal/pkg/api/server.go

* Fix merge

* Fix file

---------

Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>
Co-authored-by: michel-laterman <michel.laterman@elastic.co>
michel-laterman added a commit that referenced this pull request Sep 4, 2025
@mergify @michel-laterman michel-laterman
[8.18](backport #5372) Restore connection-level limiter (#5376)
d4bb97a 
* Restore connection-level limiter (#5372)

Restore connection level limiter to prevent OOM incidents. (removed in #4402)
This limiter is used in addition to the request-level throttle so that once
our in-flight requests reaches max_connections a 429 is returned, but if the
total connections the server uses is over max_connections*1.1 the server drops
the connection before the TLS handshake.

(cherry picked from commit 39199ef)

# Conflicts:
#	NOTICE-fips.txt
#	NOTICE.txt
#	go.mod
#	internal/pkg/api/server.go

* Fix merge

* Fix file

---------

Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>
Co-authored-by: michel-laterman <michel.laterman@elastic.co>
michel-laterman added a commit that referenced this pull request Sep 4, 2025
@mergify @michel-laterman michel-laterman
[9.0](backport #5372) Restore connection-level limiter (#5378)
5ee5686 
* Restore connection-level limiter (#5372)

Restore connection level limiter to prevent OOM incidents. (removed in #4402)
This limiter is used in addition to the request-level throttle so that once
our in-flight requests reaches max_connections a 429 is returned, but if the
total connections the server uses is over max_connections*1.1 the server drops
the connection before the TLS handshake.

(cherry picked from commit 39199ef)

# Conflicts:
#	NOTICE-fips.txt
#	NOTICE.txt
#	go.mod
#	internal/pkg/api/server.go

* Fix merge

---------

Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>
Co-authored-by: michel-laterman <michel.laterman@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@blakerouse blakerouse blakerouse approved these changes
@swiatekm swiatekm Awaiting requested review from swiatekm

Assignees

No one assigned

Labels

backport-active-all Automated backport with mergify to all the active branches bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@michel-laterman @prodsecmachine @blakerouse