Describe the bug
We are using knative serving in many services and did not have an issue processing millions of events. Lately we are seeing a weird behaviour where the queue proxy logs timeouts. The application container itself is not under heavy load and can handle thousands of requests. From the user-container itself it looks like the request does not reach the user-container and gets stuck.

Under normal condition the latency is below 0.01s. Suddenly the latency starts to increase to several seconds and even higher until timeouts get fired. The timeout happens at random times. The logs inside the user-container itself are not showing any issues, errors or longer processing times.

After restarting the pod everything is working fine for some minutes / hours until this behaviour starts again.

Expected behavior
There should not be any delay introduced between the queue proxy and the user-container.

To Reproduce
Can not find a pattern to reproduce.

Knative release version
v1.32.6-eks-931bdca

queue-proxy.log

Additional context
Logs from the queue proxy:
2025-08-29T09:48:57.328736105Z {"severity":"ERROR","timestamp":"2025-08-29T09:48:57.328515697Z","logger":"queueproxy","caller":"network/error_handler.go:33","message":"error reverse proxying request; sockstat: sockets: used 18\nTCP: inuse 3 orphan 0 tw 19 alloc 62 mem 12\nUDP: inuse 0 mem 1\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n","commit":"4853ead","knative.dev/key":"docs/docs-filter-cx-helm-ksvc-00010","knative.dev/pod":"docs-filter-cx-helm-ksvc-00010-deployment-88bfcb487-8rr4c","error":"EOF","stacktrace":"knative.dev/pkg/network.ErrorHandler.func1\n\tknative.dev/pkg@v0.0.0-20250415155312-ed3e2158b883/network/error_handler.go:33\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\tnet/http/httputil/reverseproxy.go:486\nknative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP\n\tknative.dev/serving/pkg/queue/request_metric.go:201\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3\n\tknative.dev/serving/pkg/queue/handler.go:76\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2294\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4\n\tknative.dev/serving/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2294\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\tknative.dev/serving/pkg/http/handler/timeout.go:118"} 2025-08-29T09:48:57.329030117Z {"httpRequest": {"requestMethod": "POST", "requestUrl": "/", "requestSize": "709", "status": 502, "responseSize": "4", "userAgent": "Vert.x-WebClient/4.5.14", "remoteIp": "10.0.63.225:53338", "serverIp": "10.0.121.80", "referer": "", "latency": "68.097221185s", "protocol": "HTTP/1.1"}, "traceId": "[]"} 2