Skip to content

ENH Prevent Python worker oversaturation/oversubscription#1825

Open
itamarst wants to merge 12 commits into
joblib:mainfrom
itamarst:1824-python-thread-oversaturation
Open

ENH Prevent Python worker oversaturation/oversubscription#1825
itamarst wants to merge 12 commits into
joblib:mainfrom
itamarst:1824-python-thread-oversaturation

Conversation

@itamarst

Copy link
Copy Markdown
Contributor

Fixes #1824

This also passes the third-party (OpenMP, BLAS, etc) limiting env variables to multiprocessing worker processes, as a side-effect of the implementation. That part is not tested, but could be.

A demonstration of the performance effects of this PR

Consider the following script:

import os

import numpy as np
from joblib import Parallel, delayed, parallel_config


def work():
    arr = np.ones((1_000_000,))
    for _ in range(1_000):
        arr.sum()
    return arr.sum()


def start_thread_pool():
    with parallel_config(backend="threading"):
        return Parallel(n_jobs=-1)([delayed(work)() for _ in range(10)])


with parallel_config(backend="threading", n_jobs=os.cpu_count() // 2):
    expected = work()
    for _ in Parallel()(
        [delayed(start_thread_pool)() for _ in range(os.cpu_count() // 2)]
    ):
        pass

When run under main branch, with 8 cores (taskset -c 0-7):

$ taskset -c 0-7 time python nesting-saturation.py
95.04user 0.17system 0:12.32elapsed 772%CPU (0avgtext+0avgdata 550424maxresident)k
0inputs+0outputs (0major+66503minor)pagefaults 0swaps

When run under this PR:

$ taskset -c 0-7 time python nesting-saturation.py
49.33user 0.04system 0:08.50elapsed 580%CPU (0avgtext+0avgdata 112160maxresident)k
0inputs+224outputs (0major+16949minor)pagefaults 0swaps

@itamarst itamarst changed the title ENH Prevent Python thread oversaturation/oversubscription ENH Prevent Python worker oversaturation/oversubscription Jul 22, 2026
@itamarst
itamarst marked this pull request as ready for review July 22, 2026 20:51
@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.16%. Comparing base (53cea7b) to head (ff3644e).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1825      +/-   ##
==========================================
+ Coverage   94.13%   94.16%   +0.02%     
==========================================
  Files          46       46              
  Lines        7979     8033      +54     
==========================================
+ Hits         7511     7564      +53     
- Misses        468      469       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@itamarst

Copy link
Copy Markdown
Contributor Author

This is now ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prevent oversaturation of Python threads/workers

1 participant