English version | 中文版

lite-taskman is an ultra-lightweight (~100 lines of code) thread pool management tool for Python, dedicated to providing a delightful development experience. It is specifically engineered for concurrent refactoring of sequential code, dynamic task incrementation, and real-time progress feedback.

Far from the complexity of the native ThreadPoolExecutor, it supports injecting tasks on the fly and offers an intuitive way to retrieve results, helping you achieve a smooth shift from single-threaded to multi-threaded execution.

In concurrent programming, the mental burden of complexity is often the enemy of Python newcomers. lite-taskman condenses powerful functionality into about 100 lines of source code to bring you:

• A Smooth Refactoring Experience: No need to reconstruct core logic. With simple API replacement, you can elevate legacy sequential code to parallel execution instantly, reaping the performance dividends of concurrency.
• Intuitive Control: Break free from complex Future object management. Whether you need results in their original order or want to consume them via streaming, the logic is designed to align perfectly with a developer's natural thinking.
• Delightful Dynamics: As natural as "adding more rice to your bowl while eating." It supports injecting new tasks at any time during execution, making incremental scenarios like web crawling and recursive scanning more elegant than ever.
• Ultra-lightweight Feedback: Zero dependencies and millisecond-level responsiveness. Combined with real-time progress callbacks, it ensures the status of every task is always under your control.

pip install lite-taskman

Leverage multi-core performance by simply changing function calls to add().

# Original sequential logic.
# Total execution time = time(make_soymilk) + time(buy_buns) + time(make_dumplings)
# soymilk = make_soymilk(soy, milk)
# buns = buy_buns(2)
# dumplings = make_dumplings(8, type='pork')

# Transform sequential code into parallel execution with just three lines—ultra-smooth!
from lite_taskman import TaskMan

tman = TaskMan() # Initialize thread pool
tman.add(make_soymilk, soy, milk) # Add task: Simply pass the function as the first argument...
tman.add(buy_buns, 2) # and keep other arguments exactly as they were
tman.add(make_dumplings, 8, type='pork') # Use proprietary parameters for task name, weight, and extra data (see docs)

# exec() returns a result list strictly matching the add() sequence
# Total execution time = max(time(make_soymilk), time(buy_buns), time(make_dumplings))
soymilk, buns, dumplings = [r.result for r in tman.exec()]

Perfect for crawlers: If you discover a new URL while fetching the first page, just add() it immediately.

with TaskMan() as tman:
    tman.add(fetch_url, "http://example.com/page1")
    for r in tman.process():
        # process() is a generator; the loop continues as long as new tasks are added
        new_urls = parse_links(r.result)
        for url in new_urls:
            tman.add(fetch_url, url) # Dynamic addition

If you have multiple batches of tasks but want to avoid the overhead of repeatedly creating and destroying thread pools, use all().

with TaskMan(max_workers=4) as tman:
    # Batch 1
    tman.add(make_soymilk, soy, milk)
    tman.add(buy_bun, money)
    soymilk, bun = (t.result for t in tman.all()) # Blocks to get results without closing the pool
    
    # Batch 2 (Reuses the 4 existing worker threads)
    for i in range(8): tman.add(make_dumpling, i)
    dumplings = (t.result for t in tman.all()) 

To avoid naming conflicts, tool-specific parameters are prefixed with _tm_:

• _tm_name: Task name (defaults to function name) for identification and callbacks.
• _tm_batch_size: Task weight (i.e., the number of items within a task). E.g., if one page contains 20 records, set this to 20.
• _tm_extra: Metadata pass-through. Returned as-is in Result.extra.

def my_cb(name, task_done, task_all, batch_done, batch_all, elapsed_sec):
    # name: Current task name
    # task_done/task_all: Progress based on task count
    # batch_done/batch_all: Progress based on workload (batch_size)
    # elapsed_sec(float): Cumulative elapsed time in seconds

This project is licensed under the MIT License.

Author: Rocks Wang (rockswang@foxmail.com)