[Problem RFC] HPA Control Loop Fails Due to Missing CPU Requests #812

Hi! This is an RFC I’ve written for a new SREGym benchmark problem as part of my evaluation task for UIUC++ SRSE. It primarily focuses on a Kubernetes HorizontalPodAutoscaler (HPA) control-loop failure. I’ll try to explain the problem I’ve tried to model via a scenario. Consider an HPA which is correctly configured for a user-facing service, but its target pods are missing an effective CPU request. Because a CPU-based HPA calculates usage as a percentage of the pod’s request, it can't compute the metric (source). The HPA drops into a broken state, reporting <unknown>/60%, ScalingActive=False, andFailedGetResourceMetric.

The problem itself is based on a real Kubernetes autoscaling failure mode. This AWS re:Post thread introduced me to the existence of the problem. From here I was then able to find a bunch of other posts and documentation referencing variations of the same issue, some of which I have attached below:

1. HPA won't work: FailedGetResourceMetric, confused by having two different deployments sharing the same name of a container kubernetes/kubernetes#95237
2. https://docs.vertica.com/26.2.x/en/containerized/troubleshooting/vas/
3. HPA on deployments containing dapr sidecar cpu issue dapr/dapr#7813

For the SREGym implementation, I’m thinking about using the Hotel Reservation app from the Quiz, specifically targeting its frontend deployment. The fault injection here would deploy a standard HPA (e.g., frontend-capacity), and then remove both requests.cpu and limits.cpu from the frontend container spec. I’m phrasing this as “effective CPU request” because in Kubernetes, if a CPU limit remains while the CPU request is removed, the admitted pod can still end up with an effective CPU request equal to the limit, which would prevent the intended HPA failure from appearing as my testing revealed. I’ve manually validated this on the hotel-reservation/frontend service. Removing the effective CPU request reliably triggers the <unknown>/60% failure state, and putting them back immediately restores the HPA to ValidMetricFound.

Thus a valid mitigation must restore a computable HPA metric. A hacky workaround like manually scaling the deployment shouldn't pass, as it leaves the autoscaler completely broken.

Does this sound like a good problem to implement as a PR? Please let me know if anything needs to be updated or if you have any feedback! I’ll try coding it up soon if all is good.