HydraMPP (Hydra Massive Parallel Processing) is a high-performance Python library for distributed parallel processing. It scales the same code seamlessly from:

• 💻 A single laptop
• 🖥️ A multi-core workstation
• 🌍 A multi-node HPC cluster

Built for simplicity, portability, and near-zero dependencies, HydraMPP gives you a familiar @remote / .remote() programming model without dragging in a heavyweight runtime — psutil and the Python standard library are all it needs.

flowchart LR
    A["Your Python Script"] -->|"remote tasks"| B["HydraMPP Driver"]
    B --> C["Shared Job Queue"]
    C --> D["Scheduler (main_loop)"]
    D -->|"local"| E["Worker Processes"]
    D -->|"TCP + pickle"| F["Client Node 1"]
    D -->|"TCP + pickle"| G["Client Node N"]
    H["hydra-status.py"] -. "UDP" .-> D

pip install hydraMPP

No admin rights? Install into your home folder:

pip install --user hydraMPP

conda install -c conda-forge hydraMPP

git clone https://github.com/raw-lab/HydraMPP
cd HydraMPP
pip install .

import time
import hydraMPP


# 1️⃣ Tag any function you want to run in parallel
@hydraMPP.remote
def slow_square(x, seconds=1):
    time.sleep(seconds)
    return x * x


def main():
    # 2️⃣ Initialize (local mode, auto-detect CPUs)
    hydraMPP.init()

    # 3️⃣ Fire off jobs — submission is non-blocking, returns a job id
    jobs = [slow_square.remote(i) for i in range(20)]

    # 4️⃣ Request more resources for a heavier task
    jobs.append(slow_square.options(num_cpus=4).remote(99))

    # 5️⃣ Drain results as they finish
    ready, pending = hydraMPP.wait(jobs)
    while pending:
        for job in ready:
            result = hydraMPP.get(job)
            print(f"{result[1]} -> {result[2]}")   # func name -> return value
        ready, pending = hydraMPP.wait(pending)

    hydraMPP.shutdown()


if __name__ == "__main__":
    main()

HydraMPP runs in three modes, selected by the address argument to init():

# Coordinator (host) — waits for clients to connect
hydraMPP.init(address="host", port=24515, timeout=10)

# Worker (client) — connects to the host and offers its CPUs
hydraMPP.init(address="10.0.0.5", port=24515, num_cpus=36)

hydraMPP.get(id) returns a record describing the job. Fields are accessible by index:

A helper script is included to inspect a running HydraMPP cluster over UDP.

usage: hydra-status.py [-h] [address] [port]

positional arguments:
  address     Address of the HydraMPP server [127.0.0.1]
  port        Port to connect to [24515]

options:
  -h, --help  show this help message and exit

It prints connected clients, available CPUs, and queued jobs, then exits. For continuous monitoring, wrap it with watch:

watch -n1 hydra-status.py localhost

HydraMPP can auto-configure host and client nodes inside a SLURM allocation. Just add --hydraMPP-slurm $SLURM_JOB_NODELIST to your program's invocation and HydraMPP wires up the cluster for you.

💡 Call hydraMPP.init() after all functions have been tagged with @hydraMPP.remote. Use --hydraMPP-cpus to set CPUs per node; 0 or omitted lets HydraMPP guess.

#!/bin/bash
#SBATCH --job-name=My_Slurm_Job
#SBATCH --nodes=3
#SBATCH --tasks-per-node=1
#SBATCH --cpus-per-task=36
#SBATCH --mem=100G
#SBATCH --time=1-0
#SBATCH -o slurm-%x-%j.out

echo "====================================================="
echo "Start Time  : $(date)"
echo "Submit Dir  : $SLURM_SUBMIT_DIR"
echo "Job ID/Name : $SLURM_JOBID / $SLURM_JOB_NAME"
echo "Node List   : $SLURM_JOB_NODELIST"
echo "Num Tasks   : $SLURM_NTASKS total [$SLURM_NNODES nodes @ $SLURM_CPUS_ON_NODE CPUs/node]"
echo "====================================================="

path/to/program.py --custom-args \
    --hydraMPP-slurm-auto --hydraMPP-cpus  $SLURM_CPUS_ON_NODE

Creative Commons Attribution-NonCommercial (CC BY-NC 4.0)

See the LICENSE file for details.

If you use HydraMPP in published work, please cite:

Figueroa III JL, White III RA. 2026
HydraMPP: A lightweight library for distributed massive parallel processing in Python - threading at scale. BioRxiv

We welcome:

• 🧵 Scheduling and dispatch improvements
• 🌐 Networking and fault-tolerance enhancements
• 🔐 Authentication / transport security
• 📡 Status-monitor features
• 🧪 Tests and benchmarks

Pull requests and issues are encouraged.

• 🐛 Issues: HydraMPP Issues

• 📧 Contact:

  • Dr. Richard Allen White III
  • Jose Luis Figueroa III

  If you have any questions or feedback, please feel free to get in touch by email.