Fix false error logs for non-worker child processes in reap_workers by joho54 · Pull Request #3566 · benoitc/gunicorn

joho54 · 2026-03-31T05:45:45Z

Summary

When gunicorn runs as PID 1 in a container (common in Docker/ECS/Kubernetes without tini or dumb-init), reap_workers() emits false ERROR logs for non-worker child processes reaped via waitpid(-1).

This PR moves the WORKERS membership check before exit status logging, so that non-worker processes are reaped silently at DEBUG level while real worker exits continue to be reported as errors.

Problem

reap_workers() calls os.waitpid(-1), which reaps any terminated child process. When gunicorn is PID 1, orphaned child processes are reparented to it by the kernel. Upon termination, these non-worker processes are reaped and incorrectly logged as:

[ERROR] Worker (pid:30740) exited with code 23

This triggers false alerts in error tracking systems (e.g. Sentry).

Current code flow

waitpid(-1) -> get exit code -> log ERROR -> check if worker -> skip if not worker

The error log fires before the worker membership check, so every non-worker child with a non-zero exit code produces a spurious error.

Evidence from production

Environment: AWS ECS Fargate, gunicorn 21.2.0, Python 3.11, Django backend

Observed:

Sentry captured Worker (pid:30740) exited with code 23 from gunicorn.error logger
CloudWatch log analysis showed only 3 active workers in the container: pid 40, 41, 42 (matching --workers 3)
Zero application logs from pid 30740 — it was never a gunicorn worker
Zero Booting worker logs for pid 30740
max_requests was not configured, ruling out worker recycling
All 3 real workers continued processing requests normally before and after the event
The error was a single occurrence over 7 days — a non-worker child process reaped by PID 1

Fix

Move the WORKERS.pop(wpid) check to the top of the else branch:

worker = self.WORKERS.pop(wpid, None)
if not worker:
    self.log.debug(
        "Reaped unknown child process (pid:%s, status:%s). "
        "Not a gunicorn worker.", wpid, status)
    continue

# Only report exit status for actual workers
exitcode = ...

What stays the same:

waitpid(-1) is preserved — PID 1 must reap all children to prevent zombies
Worker error reporting, HaltServer on boot/load errors, signal logging — all unchanged
child_exit hook only fires for real workers (same as before)

What changes:

Non-worker child processes are logged at DEBUG instead of ERROR
No false alerts in monitoring systems

Tests added

test_reap_non_worker_child_no_error_log — non-worker exit with error code produces no ERROR log
test_reap_non_worker_child_with_signal_no_error_log — non-worker killed by SIGKILL produces no OOM error
test_reap_mixed_worker_and_non_worker — mixed scenario: non-worker is silent, real worker is reported

All existing tests pass (1671 passed, 214 skipped).

Design note: why DEBUG instead of suppressing entirely

A question that may come up: is it safe to silently reap unknown child processes?

First, waitpid does not kill anything — it reaps an already terminated process from the kernel's process table. The three options are:

Approach	Result
Don't reap	Zombie accumulates in process table (worse)
Reap + ERROR log	Zombie prevented, but false alert (current behavior)
Reap + DEBUG log	Zombie prevented, no false alert, still traceable (this PR)

DEBUG was chosen over complete suppression so that operators can still trace unexpected child processes when needed (e.g. --log-level debug). This preserves observability without generating noise at the default log level.

The reaping itself is not optional — as PID 1, gunicorn must call waitpid for all children, whether it spawned them or not. This is standard POSIX behavior: the kernel reparents orphaned processes to init (PID 1), and init is responsible for reaping them.

Related issues

Workers that are not supposed to exist exit with code 255 #3220 — Workers that are not supposed to exist exit with code 255
Check all the worker pids instead of every child. #1585 — Proposed checking all worker pids instead of every child (closed, not merged)

This PR takes a simpler approach than #1585: rather than changing waitpid(-1) to per-pid waitpid, it keeps the reaping behavior intact and only fixes the log level for non-worker processes.

When gunicorn runs as PID 1 (e.g. in containers without tini/dumb-init), it inherits orphaned child processes via the standard UNIX reparenting mechanism. The current reap_workers() logs ERROR for every reaped process before checking whether it is actually a known worker, causing false alerts in monitoring systems like Sentry. Move the WORKERS membership check before exit status logging so that non-worker child processes are reaped silently (with a DEBUG log) while real worker exits continue to be reported as errors. This is a minimal, behavior-preserving fix: waitpid(-1) is kept as-is to fulfill PID 1 zombie reaping duties. Only the log level changes for processes not in the WORKERS dict. Ref benoitc#3220 Related: benoitc#1585

pajod · 2026-04-04T00:16:51Z

I don't disagree, but I believe with very slightly more complexity, a more useful logging is possible.

Non-worker child processes are logged at DEBUG instead of ERROR

Yes!

No false alerts in monitoring systems

Yes!

Unambigously explain the situation "Not a gunicorn worker"

Yes!

skip os.WIFEXITED / os.WIFSIGNALED details from the (demoted to DEBUG level) log line

No! If someone bothered to put log level to DEBUG, then they clearly were interested. Likely the oprhaned child process was reaped by Gunicorn after something went wrong. Once Gunicorn clears it, the status is gone and may not be reported at a higher level. So even after demoting the log level for the following messages and clarifying that the messages are not about a Gunicorn worker, those messages should appear. Only the special treatment of Gunicorn-worker-specific exit codes should be skipped.

Per review feedback, preserve WIFEXITED/WIFSIGNALED details in DEBUG logs for non-worker children instead of a generic status message. Only worker-specific handling (HaltServer, child_exit) is skipped.

joho54 · 2026-04-04T05:55:44Z

Thanks for the feedback! I've added WIFEXITED/WIFSIGNALED logging for non-worker children reaping, while only skipping worker specific handling. Please check 3e705a6

benoitc · 2026-08-17T17:50:00Z

Thanks for this, and for the production write up. The evidence you gathered is
what made it obvious the pid was never a worker, and moving the ownership check
ahead of the exit status logging is the right fix.

Two things came out of it beyond the log noise. The same ordering meant a
non-worker exiting with code 3 or 4 reached the HaltServer raises and shut the
whole server down. And the dirty arbiter is a third kind of child that is not in
WORKERS, so it needed its own branch rather than being treated as an unknown
child.

Merged in #3689 with those two additions and tests. Closing in favour of that one.

This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [gunicorn](https://github.com/benoitc/gunicorn) ([changelog](https://gunicorn.org/news/)) | `>=23,<24` → `>=26,<27` | ![age](https://developer.mend.io/api/mc/badges/age/pypi/gunicorn/26.2.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/gunicorn/23.0.0/26.2.0?slim=true) | --- > ⚠️ **Warning** > > Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/11) for more information. --- ### Release Notes <details> <summary>benoitc/gunicorn (gunicorn)</summary> ### [`v26.2.0`](https://github.com/benoitc/gunicorn/releases/tag/26.2.0): gunicorn 26.2.0 [Compare Source](benoitc/gunicorn@26.1.0...26.2.0) Cleartext HTTP/2 lands, and an HTTP/2 security fix. #### Cleartext HTTP/2 (h2c) `http2_cleartext` accepts `prior-knowledge`, `upgrade`, `both` or `off` (the default). Prior knowledge serves a connection that opens with the HTTP/2 preface; `upgrade` honours an HTTP/1.1 `Upgrade: h2c` request. Both work on the gthread, gevent and asgi workers. This is for deployments where TLS is terminated by a proxy that speaks HTTP/2 upstream, so the hop into gunicorn no longer drops to HTTP/1.1. Only peers in `forwarded_allow_ips` are considered; everyone else is served HTTP/1.x exactly as if the setting were off. Each mechanism is enabled separately, so turning one on does not turn the other on. Do not expose a cleartext HTTP/2 port to the internet. #### Security `HTTP2Request` built its headers straight from the stream, so nothing the HTTP/1 path enforces applied over HTTP/2: the underscore and `header_map` policy, duplicate `Host` and `Content-Type`, control characters in values, and the `forwarded_allow_ips` trust gate. An untrusted client could set `SCRIPT_NAME` and forge `HTTP_*` entries in the WSGI environ, and decide `wsgi.url_scheme` through `:scheme`. Both request classes now share one policy mixin, and the scheme comes from the transport. If you serve HTTP/2, this is the reason to upgrade. #### Other HTTP/2 fixes WSGI responses were buffered whole before anything was sent; they stream now. HEAD, 204 and 304 no longer carry a body. Events read while blocked on a flow-control window were discarded, losing requests and body data outright. `sendfile()` is refused on HTTP/2 responses rather than bypassing framing. #### Request bodies dropped on Upgrade requests On the ASGI worker with the fast parser, any request carrying an `Upgrade` header reached the application with an empty body, whatever the header's value and with HTTP/2 switched off entirely. Fixed in `gunicorn_h1c` 0.6.9, which the `fast` extra now requires. Full changelog: <https://gunicorn.org/news/> ### [`v26.1.0`](https://github.com/benoitc/gunicorn/releases/tag/26.1.0): gunicorn 26.1.0 [Compare Source](benoitc/gunicorn@26.0.0...26.1.0) ##### New Features - **Glob patterns in `reload_extra_files`**: entries containing `*`, `?` or `[` are treated as patterns, so `ui/*/config.json` watches every view's config without listing them one by one. Patterns are re-expanded on every reload check rather than once at startup, so a file created later starts being watched without restarting gunicorn, and `**` recurses. A pattern matching nothing warns instead of failing, since with live expansion it may match later ([#1643](benoitc/gunicorn#1643), [#3662](benoitc/gunicorn#3662)). ##### Security - **Dependency floors raised past known advisories**: every declared floor was checked against the advisory database. `tornado`, `h2`, `setuptools` and `pymdown-extensions` permitted vulnerable versions and now require the first clean release; `pytest` and `httpx` were unpinned and now carry floors. The `tornado` example pinned `tornado<6`, which was both the source of several advisories and older than the `>=6.5.0` the tornado worker needs, so the example could not run as pinned. ##### Bug Fixes - **SIGHUP did not reload the logger configuration**: `Arbiter.reload()` re-read the configuration file but kept using the logger built at startup, calling only `reopen_files()` on its existing handlers. Changes to `logconfig`, `logconfig_dict`, `logconfig_json` and `loglevel` were ignored until a full restart, which in containers meant replacing the pod. The existing logger now re-runs its setup on reload, so new handlers, formats and levels take effect while the process identity and its listeners are preserved, and re-running the setup no longer stacks duplicate syslog handlers. An invalid log configuration on reload is not fatal either: the error is reported on stderr, the previous working configuration is restored and the master keeps running with it ([#3353](benoitc/gunicorn#3353)). - **Truncated chunked bodies accepted**: RFC 9112 section 7.1.2 ends a chunked body with `0 CRLF CRLF`, the second CRLF being the mandatory empty trailer section. `ChunkedReader.parse_chunk_size()` swallowed the `NoMoreData` raised while scanning for it, so a body cut short right after the last chunk line was treated as complete instead of rejected. It now raises `ChunkMissingTerminator` ([#3382](benoitc/gunicorn#3382), [#3685](benoitc/gunicorn#3685)). - **`--spew` crashed on dynamically generated code**: the trace hook indexed the 2-tuple returned by `inspect.getsourcelines()` by line number rather than indexing the list of lines, so a frame with no `__file__` raised `AttributeError: 'int' object has no attribute 'rstrip'` on line 1 and `IndexError` beyond it. The tuple is now unpacked and offset by the source's starting line ([#3344](benoitc/gunicorn#3344), [#3495](benoitc/gunicorn#3495)). - **Duplicate `Host` and `Content-Type` headers accepted**: RFC 9110 section 5.3 allows only one of each, and a repeat cannot be merged into a list, so the message means different things to gunicorn and to anything downstream. Both are now rejected with `InvalidHeader`. The check lives in the policy hook shared by both parsers, so the pure-Python and fast parsers agree. Duplicate `Content-Length` was already rejected and is unchanged ([#3366](benoitc/gunicorn#3366), [#3548](benoitc/gunicorn#3548)). - **Non-worker children reported as failed workers**: `reap_workers()` reaps every child through `waitpid(-1)`, including processes the kernel reparented onto gunicorn when it runs as PID 1 in a container, but it logged the exit status before checking whether the pid was ever a worker. An unrelated process produced `Worker (pid:N) exited with code M` and triggered alerts. More seriously, such a process exiting with code 3 or 4 raised `HaltServer` and shut the server down. Ownership is now established first: the dirty arbiter is reported as itself, unknown children are reaped silently at debug level, and only real workers can halt the server ([#3220](benoitc/gunicorn#3220), [#3566](benoitc/gunicorn#3566)). - **Dirty arbiter exits were invisible on SIGCHLD**: `handle_chld()` called `reap_workers()` first, whose `waitpid(-1)` claimed the dirty arbiter before `reap_dirty_arbiter()` could identify it, so the latter always hit `ECHILD` and its reporting never ran. The dirty arbiter is now reaped first, and `reap_workers()` recognises it if it exits mid-loop. - **Dirty arbiter returned stale responses after a worker timeout**: when a request reached `dirty_timeout` the arbiter answered the client with a timeout error but kept the worker connection open. The worker's late response was then the first message waiting on that socket, so the next request routed to the same worker received the previous request's result, and every request after it stayed one response behind. The connection is now closed on timeout, so the late answer is discarded with it ([#3626](benoitc/gunicorn#3626)). - **ASGI connection count leaked on server-initiated close**: `nr_conns` was only decremented in `connection_lost()`, behind a guard keyed on the same flag `_close_transport()` sets first. Every close the server started (a `Connection: close` response, a keepalive timeout, an error abort) leaked one count, so `ASGIWorker._shutdown()` ran the full `graceful_timeout` and warned about connections that were already gone. The guard now uses its own flag, so the decrement and the rest of the cleanup run exactly once whichever side closes first ([#3661](benoitc/gunicorn#3661)). - **Inotify reloader on cwd-relative extra files**: `reload_extra_files` entries with no directory part (for example `.env`) produced an empty dirname, and watching it raised `InotifyError` with `ENOENT`. The current directory is now watched as `.` ([#3377](benoitc/gunicorn#3377), [#3667](benoitc/gunicorn#3667)). - **StatsD zero-valued metrics**: gauges, counters, histograms and timers reporting `0` were silently dropped because the value was tested for truthiness. Only `None` is skipped now ([#3676](benoitc/gunicorn#3676)). - **Spurious no-body warning from `sendfile()`**: a HEAD, 204 or 304 response served through `sendfile()` warned about dropped body bytes even when the file was empty and nothing was dropped. It now warns only when there are bytes to drop, matching `write()` ([#3684](benoitc/gunicorn#3684)). - **Bare `except` in the gevent websocket example**: narrowed to `except Exception` ([#3683](benoitc/gunicorn#3683)). - **ASGI `receive()` cancellation**: Let `asyncio.CancelledError` propagate from `BodyReceiver` instead of swallowing it and returning `http.disconnect`. Frameworks that cancel their disconnect listener after the response completes (Django) no longer see the cancel masked, so `request_finished` fires and `close_old_connections()` runs. Fixes idle database connections leaking since 25.1.0 ([#3627](benoitc/gunicorn#3627), [#3654](benoitc/gunicorn#3654)). - **Control socket leak on SIGHUP reload**: The control thread is now marked ready once its loop and server are live, and the stop paths wait on that readiness before scheduling shutdown. Reloads no longer leak one thread and its selector fd plus unix socket per worker, which eventually raised "too many open files" ([#3648](benoitc/gunicorn#3648)). - **WSGI body framing on HEAD/1xx/204/304**: Mirror the ASGI strip-and-warn behavior on the WSGI path. `Content-Length` is stripped on 1xx/204 per RFC 9110 section 6.4.2, body bytes are dropped for no-body responses in both `write()` and `sendfile()`, and a single warning is logged per request ([#3413](benoitc/gunicorn#3413)). ##### Refactoring - Pass log arguments to the logger instead of pre-formatting the worker termination message in `Arbiter.reap_workers()` ([#3678](benoitc/gunicorn#3678)). ##### Changes - **`packaging` is no longer a runtime dependency**: it was only ever imported by the gevent worker, to compare gevent's version. It moved to the `gevent` and `testing` extras, so a plain `pip install gunicorn` pulls in nothing ([#3643](benoitc/gunicorn#3643)). - **Fast HTTP Parser**: Require `gunicorn_h1c >= 0.6.6`, which rejects duplicate `Host` and `Content-Type` headers in the C parser itself. Gunicorn already refuses them on both the WSGI and ASGI paths, so this changes nothing that is reachable; it moves the rejection to where the bytes are read and lets the ASGI corpus exercise those cases against the fast parser directly. Full changelog: <https://gunicorn.org/2026-news/> ### [`v26.0.0`](https://github.com/benoitc/gunicorn/releases/tag/26.0.0) [Compare Source](benoitc/gunicorn@25.3.0...26.0.0) #### Breaking Changes - **Eventlet worker removed**: The `eventlet` worker class has been dropped. Migrate to `gevent`, `gthread`, or `tornado`. #### New Features - **ASGI Framework Compatibility Suite**: New end-to-end compatibility test harness covering Starlette, FastAPI, Litestar, Quart, Sanic, and BlackSheep. Current grid passes 438/444 tests (98%). - **ASGI Test Suite Expansion**: 134 additional ASGI unit tests covering protocol semantics, lifespan, websockets, and chunked framing. #### Security - **HTTP/1.1 Request-Target Validation** (RFC 9112 sections 3.2.3, 3.2.4): - Reject `authority-form` request-target outside `CONNECT` - Reject `asterisk-form` request-target outside `OPTIONS` - Reject `relative-reference` request-targets - **Header Field Hardening** (RFC 9110): - Reject control characters in header field-value (section 5.5) - Reject forbidden trailer field-names (section 6.5.1) - Reject `Content-Length` list form (RFC 9112 section 6.3) - **Request Smuggling Hardening**: - Tighten keepalive gate and scope `finish_body` byte cap - Keep `_body_receiver` alive across the keepalive smuggling gate so pipelined requests cannot re-enter a closed body - Address parser/protocol findings from a six-point WSGI/ASGI audit - **PROXY Protocol (ASGI)**: Enforce `proxy_allow_ips` and tighten v1/v2 parsing in the ASGI callback parser. - **Connection Draining**: Drain the connection on close per RFC 9112 section 9.6 to prevent reset-on-close truncation. #### Bug Fixes - **Body Framing on HEAD/204/304**: - Keep `Content-Length` on HEAD and 304 responses ([#3621](benoitc/gunicorn#3621)) - Drop body framing on HEAD/204/304 even when the framework set it - Warn once when an ASGI app emits a body for a no-body response - **HTTP/2 ASGI**: - Fix `_handle_stream_ended` to set `_body_complete` in the async HTTP/2 handler so request bodies finalize correctly on stream end - Add `InvalidChunkExtension` mapping and fast-parser support in ASGI tests ([#3565](benoitc/gunicorn#3565)) - **HTTP/1.1 100-Continue**: Stop adding `Transfer-Encoding: chunked` to 100-Continue interim responses. - **WebSocket Close Handshake** (RFC 6455): - Comply with the close handshake state machine - Close the transport after the close handshake completes - Fix binary send when the `text` key is `None` - **Early Hints**: Validate headers in the `early_hints` callback to match `process_headers`; pass only the header name to `InvalidHeader` ([#3588](benoitc/gunicorn#3588)). - **ASGI Framework Fixes**: - Fix ASGI disconnect handling for Django-style apps - Fix Litestar request handling (use raw ASGI receive for body/headers) - Fix Litestar HTTP endpoints for compatibility tests - Fix Quart headers endpoint to normalize keys to lowercase - Fix Quart WebSocket close test app (missing `accept()`) - Fix duplicate `Transfer-Encoding` header for BlackSheep streaming #### Refactoring - Split `BodyReceiver._closed` into separate transport and body-wait flags for clearer keepalive/EOF semantics. #### Changes - **Fast HTTP Parser**: Require `gunicorn_h1c >= 0.6.5`. Drop the last `python_only` test markers; the C extension is now used wherever available (CPython only; PyPy continues to use the Python parser). - **Test Dependencies**: Add `h2` and `uvloop` to the `testing` extra; remove `eventlet`. - **Docker Build**: Bump GitHub Actions `docker/setup-qemu-action`, `docker/setup-buildx-action`, `docker/login-action`, `docker/build-push-action`, and `docker/metadata-action` to current major versions. **Full changelog**: <benoitc/gunicorn@25.3.0...26.0.0> ### [`v25.3.0`](https://github.com/benoitc/gunicorn/releases/tag/25.3.0): Gunicorn 25.3.0 [Compare Source](benoitc/gunicorn@25.2.0...25.3.0) #### Bug Fixes - **HTTP/2 ASGI Body Duplication**: Fix request body being received twice in HTTP/2 ASGI requests, causing JSON parsing errors with "Extra data" messages ([#3558](benoitc/gunicorn#3558)) - **ASGI Chunked EOF Handling**: Add `finish()` method to callback parser to handle chunked encoding edge case where connection closes before final CRLF after zero-chunk - **HTTP/2 Documentation**: Fix `http_protocols` examples to use comma-separated string instead of list syntax ([#3561](benoitc/gunicorn#3561)) - **Chunked Encoding**: Reject chunk extensions containing bare CR bytes per RFC 9112 ([#3556](benoitc/gunicorn#3556)) - **Request Line Limit**: Fix `--limit-request-line 0` to mean unlimited as documented, instead of using default maximum. Works with both Python and fast C parser. ([#3563](benoitc/gunicorn#3563)) #### Security - **ASGI Parser Header Validation**: Add security checks per RFC 9110/9112: - Reject duplicate Content-Length headers - Reject requests with both Content-Length and Transfer-Encoding - Reject chunked transfer encoding in HTTP/1.0 - Reject stacked chunked encoding - Validate Transfer-Encoding values - Strict chunk size validation #### Changes - **Fast HTTP Parser**: Update to gunicorn\_h1c >= 0.6.3 for `asgi_headers` property and `InvalidChunkExtension` validation for bare CR rejection - **ASGI PROXY Protocol**: Add PROXY protocol v1/v2 support to callback parser - **Docker Images**: Update to Python 3.14 ### [`v25.2.0`](https://github.com/benoitc/gunicorn/releases/tag/25.2.0): Gunicorn 25.2.0 [Compare Source](benoitc/gunicorn@25.1.0...25.2.0) ##### New Features - **Fast HTTP Parser (gunicorn\_h1c 0.4.1)**: Integrate new exception types and limit parameters from gunicorn\_h1c 0.4.1 for both WSGI and ASGI workers - Requires gunicorn\_h1c >= 0.4.1 for `http_parser='fast'` - Falls back to Python parser in `auto` mode if version not met - Proper HTTP status codes for limit errors (414, 431) ##### Bug Fixes - **uWSGI Async Workers**: Fix `InvalidUWSGIHeader: incomplete header` error when using gevent or gthread workers with uwsgi protocol behind nginx. ([#3552](benoitc/gunicorn#3552), [PR #3554](benoitc/gunicorn#3554)) - **FileWrapper Iterator Protocol**: Add `__iter__` and `__next__` methods to `FileWrapper` for full PEP 3333 compliance. ([#3396](benoitc/gunicorn#3396), [PR #3550](benoitc/gunicorn#3550)) ##### Performance - **ASGI HTTP Parser Optimizations**: Improve ASGI worker HTTP parsing performance - Callback-based parsing with direct `bytearray` buffer operations - Use `bytearray.find()` directly instead of converting to bytes first - Use index-based iteration for header parsing instead of `list.pop(0)` (O(1) vs O(n)) ### [`v25.1.0`](https://github.com/benoitc/gunicorn/releases/tag/25.1.0): Gunicorn 25.1.0 [Compare Source](benoitc/gunicorn@25.0.3...25.1.0) ##### New Features - **Control Interface (gunicornc)**: Add interactive control interface for managing running Gunicorn instances, similar to birdc for BIRD routing daemon ([PR #3505](benoitc/gunicorn#3505)) - Unix socket-based communication with JSON protocol - Interactive mode with readline support and command history - Commands: `show all/workers/dirty/config/stats/listeners` - Worker management: `worker add/remove/kill`, `dirty add/remove` - Server control: `reload`, `reopen`, `shutdown` - New settings: `--control-socket`, `--control-socket-mode`, `--no-control-socket` - New CLI tool: `gunicornc` for connecting to control socket - See [Control Interface Guide](https://gunicorn.org/guides/gunicornc/) for details - **Dirty Stash**: Add global shared state between workers via `dirty.stash` ([PR #3503](benoitc/gunicorn#3503)) - In-memory key-value store accessible by all workers - Supports get, set, delete, clear, keys, and has operations - Useful for sharing state like feature flags, rate limits, or cached data - **Dirty Binary Protocol**: Implement efficient binary protocol for dirty arbiter IPC using TLV (Type-Length-Value) encoding ([PR #3500](benoitc/gunicorn#3500)) - More efficient than JSON for binary data - Supports all Python types: str, bytes, int, float, bool, None, list, dict - Better performance for large payloads - **Dirty TTIN/TTOU Signals**: Add dynamic worker scaling for dirty arbiters ([PR #3504](benoitc/gunicorn#3504)) - Send SIGTTIN to increase dirty workers - Send SIGTTOU to decrease dirty workers - Respects minimum worker constraints from app configurations ##### Changes - **ASGI Worker**: Promoted from beta to stable - **Dirty Arbiters**: Now marked as beta feature ##### Documentation - Fix Markdown formatting in /configure documentation ### [`v25.0.3`](https://github.com/benoitc/gunicorn/releases/tag/25.0.3) [Compare Source](benoitc/gunicorn@25.0.2...25.0.3) #### What's Changed ##### Bug Fixes - Fix RuntimeError when StopIteration raised in ASGI coroutine ([#3484](benoitc/gunicorn#3484)) - Fix passing maxsplit in re.split() as positional argument (deprecated in Python 3.13) ##### Documentation - Updated sponsorship section and homepage **Full Changelog**: <benoitc/gunicorn@25.0.2...25.0.3> ### [`v25.0.2`](https://github.com/benoitc/gunicorn/releases/tag/25.0.2) [Compare Source](benoitc/gunicorn@25.0.1...25.0.2) #### What's Changed ##### Bug Fixes - Fix ASGI concurrent request failures through nginx proxy - Graceful disconnect handling for ASGI worker - Lazy import dirty module for gevent compatibility ##### Other - Increase CI timeout for signal tests on PyPy - Remove trailing blank line in instrument/**init**.py **Full Changelog**: <benoitc/gunicorn@25.0.1...25.0.2> ### [`v25.0.1`](https://github.com/benoitc/gunicorn/releases/tag/25.0.1) [Compare Source](benoitc/gunicorn@25.0.0...25.0.1) #### Bug Fixes - Fix ASGI streaming responses (SSE) hanging: add chunked transfer encoding for HTTP/1.1 responses without Content-Length header. Without chunked encoding, clients wait for connection close to determine end-of-response. #### Changes - Update celery\_alternative example to use FastAPI with native ASGI worker and uvloop for async task execution #### Testing - Add ASGI compliance test suite with Docker-based integration tests covering HTTP, WebSocket, streaming, lifespan, framework integration (Starlette, FastAPI), HTTP/2, and concurrency scenarios ### [`v25.0.0`](https://github.com/benoitc/gunicorn/releases/tag/25.0.0): Gunicorn 25.0.0 [Compare Source](benoitc/gunicorn@24.1.1...25.0.0) #### New Features - **Dirty Arbiters**: Separate process pool for executing long-running, blocking operations (AI model loading, heavy computation) without blocking HTTP workers ([PR #3460](benoitc/gunicorn#3460)) - Inspired by Erlang's dirty schedulers - Asyncio-based with Unix socket IPC - Stateful workers that persist loaded resources - New settings: `--dirty-app`, `--dirty-workers`, `--dirty-timeout`, `--dirty-threads`, `--dirty-graceful-timeout` - Lifecycle hooks: `on_dirty_starting`, `dirty_post_fork`, `dirty_worker_init`, `dirty_worker_exit` - **Per-App Worker Allocation for Dirty Arbiters**: Control how many dirty workers load each app for memory optimization with heavy models ([PR #3473](benoitc/gunicorn#3473)) - Set `workers` class attribute on DirtyApp (e.g., `workers = 2`) - Or use config format `module:class:N` (e.g., `myapp:HeavyModel:2`) - Requests automatically routed to workers with the target app - New exception `DirtyNoWorkersAvailableError` for graceful error handling - Example: 8 workers × 10GB model = 80GB → with `workers=2`: 20GB (75% savings) - **HTTP/2 Support (Beta)**: Native HTTP/2 (RFC 7540) support for improved performance with modern clients ([PR #3468](benoitc/gunicorn#3468)) - Multiplexed streams over a single connection - Header compression (HPACK) - Flow control and stream prioritization - Works with gthread, gevent, and ASGI workers - New settings: `--http-protocols`, `--http2-max-concurrent-streams`, `--http2-initial-window-size`, `--http2-max-frame-size`, `--http2-max-header-list-size` - Requires SSL/TLS and h2 library: `pip install gunicorn[http2]` - New example: `examples/http2_gevent/` with Docker and tests - **HTTP 103 Early Hints**: Support for RFC 8297 Early Hints to enable browsers to preload resources before the final response ([PR #3468](benoitc/gunicorn#3468)) - WSGI: `environ['wsgi.early_hints'](headers)` callback - ASGI: `http.response.informational` message type - Works with both HTTP/1.1 and HTTP/2 - **uWSGI Protocol for ASGI Worker**: The ASGI worker now supports receiving requests via the uWSGI binary protocol from nginx ([PR #3467](benoitc/gunicorn#3467)) #### Bug Fixes - Fix HTTP/2 ALPN negotiation for gevent and eventlet workers when `do_handshake_on_connect` is False (the default). The TLS handshake is now explicitly performed before checking `selected_alpn_protocol()`. - Fix setproctitle initialization with systemd socket activation ([#3465](benoitc/gunicorn#3465)) - Fix `Expect: 100-continue` handling: ignore the header for HTTP/1.0 requests since 100-continue is only valid for HTTP/1.1+ ([PR #3463](benoitc/gunicorn#3463)) - Fix missing `_expected_100_continue` attribute in UWSGIRequest - Disable setproctitle on macOS to prevent segfaults during process title updates - Publish full exception traceback when the application fails to load ([#3462](benoitc/gunicorn#3462)) - Fix ASGI: quick shutdown on SIGINT/SIGQUIT, graceful on SIGTERM #### Deprecations - **Eventlet Worker**: The `eventlet` worker is deprecated and will be removed in Gunicorn 26.0. Eventlet itself is no longer actively maintained. Please migrate to `gevent`, `gthread`, or another supported worker type. #### Changes - Remove obsolete Makefile targets ([PR #3471](benoitc/gunicorn#3471)) - Replace RST with markdown documentation format ### [`v24.1.1`](https://github.com/benoitc/gunicorn/releases/tag/24.1.1) [Compare Source](benoitc/gunicorn@24.1.0...24.1.1) #### Bug Fixes - Fix `forwarded_allow_ips` and `proxy_allow_ips` to remain as strings for backward compatibility with external tools like uvicorn. Network validation now uses strict mode to detect invalid CIDR notation (e.g., `192.168.1.1/24` where host bits are set) ([#3458](benoitc/gunicorn#3458), [PR #3459](benoitc/gunicorn#3459)) *** **Full Changelog**: <benoitc/gunicorn@24.1.0...24.1.1> ### [`v24.1.0`](https://github.com/benoitc/gunicorn/releases/tag/24.1.0): Gunicorn 24.1.0 [Compare Source](benoitc/gunicorn@24.0.0...24.1.0) #### New Features - **Official Docker Image**: Gunicorn now publishes official Docker images to GitHub Container Registry ([PR #3454](benoitc/gunicorn#3454)) - Available at `ghcr.io/benoitc/gunicorn` - Based on Python 3.12 slim image - Uses recommended worker formula (2 × CPU + 1) - Configurable via environment variables - **PROXY Protocol v2 Support**: Extended PROXY protocol implementation to support the binary v2 format in addition to the existing text-based v1 format ([PR #3451](benoitc/gunicorn#3451)) - New `--proxy-protocol` modes: `off`, `v1`, `v2`, `auto` - `auto` mode (default when enabled) detects v1 or v2 automatically - v2 binary format is more efficient and supports additional metadata - Works with HAProxy, AWS NLB/ALB, and other PROXY protocol v2 sources - **CIDR Network Support**: `--forwarded-allow-ips` and `--proxy-allow-from` now accept CIDR notation (e.g., `192.168.0.0/16`) for specifying trusted networks ([PR #3449](benoitc/gunicorn#3449)) - **Socket Backlog Metric**: New `gunicorn.socket.backlog` gauge metric reports the current socket backlog size on Linux systems ([PR #3450](benoitc/gunicorn#3450)) - **InotifyReloader Enhancement**: The inotify-based reloader now watches newly imported modules, not just those loaded at startup ([PR #3447](benoitc/gunicorn#3447)) #### Bug Fixes - Fix signal handling regression where SIGCLD alias caused "Unhandled signal: cld" errors on Linux when workers fail during boot ([#3453](benoitc/gunicorn#3453)) - Fix socket blocking mode on keepalive connections preventing SSL handshake failures with async workers ([PR #3452](benoitc/gunicorn#3452)) - Use smaller buffer size in `finish_body()` for faster timeout detection on slow or abandoned connections ([PR #3453](benoitc/gunicorn#3453)) - Handle `SSLWantReadError` in `finish_body()` to prevent worker hangs during SSL renegotiation ([PR #3448](benoitc/gunicorn#3448)) - Log SIGTERM as info level instead of warning to reduce noise in orchestrated environments ([PR #3446](benoitc/gunicorn#3446)) - Print exception details to stderr when worker fails to boot ([PR #3443](benoitc/gunicorn#3443)) - Fix `unreader.unread()` to prepend data to buffer instead of appending ([PR #3442](benoitc/gunicorn#3442)) - Prevent `RecursionError` when pickling Config objects ([PR #3441](benoitc/gunicorn#3441)) - Use proper exception chaining with `raise from` in glogging.py ([PR #3440](benoitc/gunicorn#3440)) #### Installation ```bash pip install gunicorn==24.1.0 ``` Or use the official Docker image: ```bash docker pull ghcr.io/benoitc/gunicorn:24.1.0 ``` ### [`v24.0.0`](https://github.com/benoitc/gunicorn/releases/tag/24.0.0) [Compare Source](benoitc/gunicorn@23.0.0...24.0.0) #### New Features - **ASGI Worker (Beta)**: Native asyncio-based ASGI support for running async Python frameworks like FastAPI, Starlette, and Quart without external dependencies - HTTP/1.1 with keepalive connections - WebSocket support - Lifespan protocol for startup/shutdown hooks - Optional uvloop for improved performance - **uWSGI Binary Protocol**: Support for receiving requests from nginx via `uwsgi_pass` directive - **Documentation Migration**: Migrated to MkDocs with Material theme #### Security - **eventlet**: Require eventlet >= 0.40.3 (CVE-2021-21419, CVE-2025-58068) - **gevent**: Require gevent >= 24.10.1 (CVE-2023-41419, CVE-2024-3219) - **tornado**: Require tornado >= 6.5.0 (CVE-2025-47287) #### Install ``` pip install gunicorn==24.0.0 ``` </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.tainton.uk/repos/rsu/pulls/16 Co-authored-by: renovate[bot] <renovate-bot@git.tainton.uk>

Log exit status details at DEBUG level for non-worker child processes

3e705a6

Per review feedback, preserve WIFEXITED/WIFSIGNALED details in DEBUG logs for non-worker children instead of a generic status message. Only worker-specific handling (HaltServer, child_exit) is skipped.

benoitc mentioned this pull request Aug 17, 2026

arbiter: establish child ownership before reporting exit status #3689

Merged

pull Bot pushed a commit to Mu-L/gunicorn that referenced this pull request Aug 17, 2026

docs: changelog entries for benoitc#3566

f985a27

benoitc closed this Aug 17, 2026

Navigation Menu

Uh oh!

Fix false error logs for non-worker child processes in reap_workers - #3566

Fix false error logs for non-worker child processes in reap_workers#3566
joho54 wants to merge 2 commits into
benoitc:masterfrom
joho54:fix/reap-workers-non-worker-child-error-log

joho54 commented Mar 31, 2026

Uh oh!

pajod commented Apr 4, 2026

Uh oh!

joho54 commented Apr 4, 2026

Uh oh!

benoitc commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

joho54 commented Mar 31, 2026

Summary

Problem

Current code flow

Evidence from production

Fix

Tests added

Design note: why DEBUG instead of suppressing entirely

Related issues

Uh oh!

pajod commented Apr 4, 2026

Uh oh!

joho54 commented Apr 4, 2026

Uh oh!

benoitc commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants