One stalled upstream request permanently poisons its cache key, and every request that joins it leaks a goroutine.
A pod of ours sat unready for three hours because a single cache key never recovered. Every request for that key returned 504 after the backend timeout, while an identical request with any other key was served in ~7ms. I have a goroutine dump from the affected process and the chain is unambiguous, so I thought it worth writing up properly.
Versions
github.com/darkweak/souin v1.7.8, github.com/darkweak/souin/plugins/caddy v1.7.8golang.org/x/sync v0.22.0- Caddy
v2.11.4, FrankenPHP v1.12.7, Go's handleRequestWithRegularPHPThreads
- Storage: badger.
mode strict. Startup logged Set backend timeout to 10s and Set cache timeout to 10ms.
What happened
The process had been up 177 minutes. A Kubernetes readiness probe polls one URL every 10s with a 1s client timeout, so that one cache key was requested ~1060 times and every request but the first was a duplicate of an in-flight call.
The dump contains 1160 goroutines. 1129 are souin-related. Of those:
- 1 is the singleflight owner,
goroutine 71 [select, 177 minutes] — blocked since process start:
goroutine 71 [select, 177 minutes]:
github.com/dunglas/frankenphp.handleRequestWithRegularPHPThreads(...) threadregular.go:164
github.com/dunglas/frankenphp.ServeHTTP(...) frankenphp.go:427
...
github.com/darkweak/souin/plugins/caddy.(*SouinCaddyMiddleware).ServeHTTP.func1(...) httpcache.go:92
github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream.func2() middleware.go:493
golang.org/x/sync/singleflight.(*Group).doCall.func2(...) singleflight.go:198
golang.org/x/sync/singleflight.(*Group).doCall(...) singleflight.go:200
golang.org/x/sync/singleflight.(*Group).Do(...) singleflight.go:113
github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream(...) middleware.go:492
github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func16(...) middleware.go:1042
created by ...ServeHTTP in goroutine 25 middleware.go:1035
- 1127 are joiners, all identical, blocked for up to 177 minutes:
goroutine 111 [sync.WaitGroup.Wait, 177 minutes]:
sync.(*WaitGroup).Wait(...) waitgroup.go:206
golang.org/x/sync/singleflight.(*Group).Do(...) singleflight.go:99
github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream(...) middleware.go:492
github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func16(...) middleware.go:1042
1127 joiners against ~1060 expected polls over 177 minutes is the accumulation rate you would predict if none are ever released.
Two distinct consequences
The key never recovers. Do holds the entry until fn returns. The owning fn is blocked in the upstream handler and never will, so the entry stays in the group for the life of the process. Every later request for that key joins a call that cannot complete. The backend timeout does bound what the client sees — each one 504s after 10s — but it does not release the entry or let a fresh attempt take over, so the key is dead until the process restarts. In our case that meant a pod that could never pass its readiness probe, on a URL it was otherwise serving correctly.
Joiners leak. The 504 goes back to the client at the 10s mark, but the goroutine stays parked in wg.Wait() afterwards — the dump shows joiners at 3, 5, 11, 29, 47, 69, 99, 103 minutes and beyond. So a single stalled upstream call leaks one goroutine per subsequent request for that key, unbounded, along with whatever the request context retains. Anything polling a cached URL on a fixed interval turns that into steady growth.
What triggered the stall
Ours came from FrankenPHP, and the timing is specific: the container runs migrations before Caddy starts listening, so by the time it served its first request the probe's initial delay had elapsed and the kubelet was already polling. The first request arrived 0.8s after serving initial configuration and the client hung up at its 1s timeout. That request is goroutine 71, still in select inside handleRequestWithRegularPHPThreads 177 minutes later. Whether it should have returned when the client disconnected is a FrankenPHP question and I will follow it up there.
But souin decides the blast radius. One upstream request that never returns is a lost request; here it became indefinite unavailability for that key plus an unbounded goroutine leak, and nothing in the process recovers from it.
Suggestion
Give the shared call the same bound the client gets. DoChan plus a select on the channel against the backend timeout (or the request context) would let joiners give up and return, and Forgeting the key on that path would let the next request start a fresh attempt rather than inheriting a dead one. That keeps the deduplication benefit while making a stalled upstream cost one timeout rather than the key.
Happy to test a patch against the reproduction, and I can share the full dump (1.2MB) if useful.
One stalled upstream request permanently poisons its cache key, and every request that joins it leaks a goroutine.
A pod of ours sat unready for three hours because a single cache key never recovered. Every request for that key returned 504 after the backend timeout, while an identical request with any other key was served in ~7ms. I have a goroutine dump from the affected process and the chain is unambiguous, so I thought it worth writing up properly.
Versions
github.com/darkweak/souin v1.7.8,github.com/darkweak/souin/plugins/caddy v1.7.8golang.org/x/sync v0.22.0v2.11.4, FrankenPHPv1.12.7, Go'shandleRequestWithRegularPHPThreadsmode strict. Startup loggedSet backend timeout to 10sandSet cache timeout to 10ms.What happened
The process had been up 177 minutes. A Kubernetes readiness probe polls one URL every 10s with a 1s client timeout, so that one cache key was requested ~1060 times and every request but the first was a duplicate of an in-flight call.
The dump contains 1160 goroutines. 1129 are souin-related. Of those:
goroutine 71 [select, 177 minutes]— blocked since process start:1127 joiners against ~1060 expected polls over 177 minutes is the accumulation rate you would predict if none are ever released.
Two distinct consequences
The key never recovers.
Doholds the entry untilfnreturns. The owningfnis blocked in the upstream handler and never will, so the entry stays in the group for the life of the process. Every later request for that key joins a call that cannot complete. The backend timeout does bound what the client sees — each one 504s after 10s — but it does not release the entry or let a fresh attempt take over, so the key is dead until the process restarts. In our case that meant a pod that could never pass its readiness probe, on a URL it was otherwise serving correctly.Joiners leak. The 504 goes back to the client at the 10s mark, but the goroutine stays parked in
wg.Wait()afterwards — the dump shows joiners at 3, 5, 11, 29, 47, 69, 99, 103 minutes and beyond. So a single stalled upstream call leaks one goroutine per subsequent request for that key, unbounded, along with whatever the request context retains. Anything polling a cached URL on a fixed interval turns that into steady growth.What triggered the stall
Ours came from FrankenPHP, and the timing is specific: the container runs migrations before Caddy starts listening, so by the time it served its first request the probe's initial delay had elapsed and the kubelet was already polling. The first request arrived 0.8s after
serving initial configurationand the client hung up at its 1s timeout. That request is goroutine 71, still inselectinsidehandleRequestWithRegularPHPThreads177 minutes later. Whether it should have returned when the client disconnected is a FrankenPHP question and I will follow it up there.But souin decides the blast radius. One upstream request that never returns is a lost request; here it became indefinite unavailability for that key plus an unbounded goroutine leak, and nothing in the process recovers from it.
Suggestion
Give the shared call the same bound the client gets.
DoChanplus aselecton the channel against the backend timeout (or the request context) would let joiners give up and return, andForgeting the key on that path would let the next request start a fresh attempt rather than inheriting a dead one. That keeps the deduplication benefit while making a stalled upstream cost one timeout rather than the key.Happy to test a patch against the reproduction, and I can share the full dump (1.2MB) if useful.