[Problem RFC] A database connection cap set too low starves a service's connection pool #846

Hello! 👋🏼

I am submitting/proposing this RFC as part of the UIUC++ SRSE evaluation task for a new SREGym benchmark problem.

I noticed that the MongoDB-related problems we already have cover authentication (revoke_auth, auth_miss_mongodb) and data or version issues (update_incompatible_correlated), but none of them cover the database refusing to accept connections because its connection limit is set too low for the client that depends on it. That is a different failure from "the database is down" or "auth is wrong", while the database process is perfectly healthy, & it is simply turning away new connections.

I may be over-separating this from the existing resource problems, but I think the diagnosis path is genuinely distinct. The database pod and the service pod are both Running and Ready, the node is fine, and yet the service's queries fail. The signal is not a crash or an eviction, it is a connection being refused before it is even authenticated.

Hence, I would like to propose a problem that reproduces the classic database connection-pool-exhaustion failure.

Motivation

This is one of the most common ways a healthy database takes down the services in front of it. A fleet of application replicas, each holding a client-side connection pool, collectively opens more connections than the server is configured to accept. Once the server hits its limit it refuses new connections with an error like "too many open connections", and every request that needs a fresh connection fails, even though the database itself is idle and healthy.

The class is significant enough at scale that the industry built managed infrastructure specifically to absorb it. AWS RDS Proxy exists because serverless and containerized fleets routinely exhaust a database's connection limit, and PgBouncer occupies the same role for PostgreSQL. The MongoDB server has a direct knob for this limit (net.maxIncomingConnections, exposed on the command line as --maxConns), and setting it too low for the workload is a documented way to cause exactly this outage.

I find this scenario valuable for the following reasons:

1. Both pods stay Running and Ready. In the Hotel Reservation app neither mongodb-geo nor geo carries a health probe, so kubectl get pods shows everything green while the geo service cannot get the database connections it needs.
2. My expectation is that the first reflex is to restart the service or the database. I want a setup where that looks reasonable and fails, because the connection cap is part of the database's startup configuration and is still in effect after any restart. The fix has to change the cap, not bounce a pod.

Proposal

A new problem connection_pool_exhaustion_hotel_reservation on the Hotel Reservation app. It targets Deployment/mongodb-geo (the database) and the geo service that depends on it, in the hotel-reservation namespace. I feel it is safe to say no new fault-injector module is needed, since the problem class patches the database's startup arguments through kubectl. A new mitigation oracle (ConnectionPoolMitigationOracle) is added because the default pod-health oracle is blind here, both pods are Ready throughout.

The injection sequence:

1. Snapshot the current mongodb-geo container arguments.
2. Add --maxConns 5 (5 is the smallest value mongod accepts, and far below the driver pool the service opens) and let the database roll out.
3. Confirm the cap is active by opening a burst of connections and observing the database refuse them after a handful.

Recovery restores the original arguments from the snapshot and rolls the database back out.

NOTE: In the said context, mongodb-geo and geo are the app's own components, and --maxConns is a real mongod flag, so the only thing the agent sees is a database configured with a low connection limit, which is what the real incident looks like.

CC'ing for further visibility: @Saadmrp1038
Would appreciate some feedback/advice on this, and then I'll create a PR :)

References

• AWS RDS Proxy (managed connection pooling, built to absorb connection exhaustion from large client fleets): https://aws.amazon.com/rds/proxy/
• MongoDB manual, net.maxIncomingConnections / --maxConns: https://www.mongodb.com/docs/manual/reference/configuration-options/#mongodb-setting-net.maxIncomingConnections
• A documented production outage from connection-pool exhaustion: https://www.c-sharpcorner.com/article/postgresql-connection-pool-exhaustion-lessons-from-a-production-outage/