Fault-tolerant background threads for Python
POV: you want a framework-agnostic way to reliably run a plain or async function or method in the background, continuously or with pauses between work units.
bgt comes to the rescue with:
-
A service that runs your code in a loop and optionally waits between work units. It wakes up on a fixed time interval, or as soon as something wakes it.
-
A supervisor that runs that loop in a background thread. If your code crashes, the supervisor restarts the loop after an exponential backoff. Write crash-only code, bgt takes care of the rest.
-
Thorough instrumentation via structlog and Prometheus.
-
Framework and platform independence.
bgt is not a job queue like Celery or RQ. Common use cases include:
- Periodic cleanup duties for expired caches or sessions.
- Flushing buffered metrics or events.
- Processing a continuous stream of data in bounded batches.
Here's a service that runs a work unit every two seconds and survives its own crashes:
import bgt
def do_work() -> bool:
... # one bounded work unit; if it crashes, it doesn't crash the loop
return False # False = wait for next wakeup; True = run again immediately
with bgt.SupervisedService.start(
bgt.as_work_factory(do_work),
name="example",
wakeup=bgt.IntervalOnlyWakeup(),
interval=2,
):
... # do_work runs in the background until we leave this blockReturn True from your work unit to be run again immediately.
This keeps work units short, which makes shutdowns prompt.
For async work units such as async def do_work() use bgt.as_async_work_factory().
Check out our step-by-step tutorial to get an instant feel for the features!
The package is available on PyPI under the bgt name:
$ uv pip install bgtFull documentation lives at https://bgt.hynek.me/.
There are currently two packages built on top of bgt:
-
pgbg adds PostgreSQL
LISTEN/NOTIFY-driven wakeups and leader election with automatic failover on top. If your services should wake up on database events, or only one process at a time should do the work, check it out! -
django-bgt adapts bgt for Django. It provides a
servicedecorator for your work functions, using the Django ORM to orchestrate leader election and automatic failover, while remaining database agnostic.
bgt is written by Hynek Schlawack and distributed under the terms of the MIT license.
The development is kindly supported by my employer Variomedia AG and all my fabulous GitHub Sponsors.