biahub

Bio-image analysis hub supporting high-throughput data reconstruction on HPC clusters with Slurm workload management.

biahub was originally developed to reconstruct data acquired on the mantis microscope using the shrimPy acquisition engine, and has since been extended to process diverse multimodal datasets. biahub reconstruction workflows rely on OME-ZARR datasets (for example, as created with iohub) which enable efficient parallelization across compute nodes.

biahub uses uv for environment and dependency management. Install uv first:

curl -LsSf https://astral.sh/uv/install.sh | sh

Then clone the repository and create the environment. uv reads pyproject.toml and uv.lock, downloads a suitable Python (>=3.12) if needed, and creates a .venv in the repository:

git clone https://github.com/czbiohub-sf/biahub.git
cd biahub
uv sync

Run commands with uv run, which always resolves against the locked environment:

uv run biahub --help

Alternatively, activate the environment once and call biahub directly:

source .venv/bin/activate  # Windows: .venv\Scripts\activate
biahub --help

Heavier or task-specific dependencies live in extras. Add them with --extra:

uv sync --extra all
# or, per command:
uv run --extra stain biahub virtual-stain --help

uv sync --group dev   # same as `make setup-develop`
pre-commit install

See the contributing guide for the full development workflow.

Reconstruction is driven by the biahub CLI. The commands share a small set of conventions:

Execution is controlled by a few more flags: -l, --local runs the work on the current machine instead of submitting Slurm jobs, --cluster {slurm,local,debug} selects the backend explicitly (debug runs in-process in the foreground), --init only creates the output store and exits, and -m, --monitor follows the submitted jobs until they finish.

Some steps include an estimate-* command that writes a YAML of parameters you can inspect and edit, followed by an apply command that consumes that YAML. Run biahub <command> --help for the full option list, or browse the CLI reference.

# CONVERT TO ZARR
iohub convert -i ./acq_name/acq_name_labelfree_1 -o ./labelfree.zarr
iohub convert -i ./acq_name/acq_name_lightsheet_1 -o ./lightsheet.zarr

# DECONVOLVE FLUORESCENCE
biahub characterize-psf -i ./beads.zarr -c ./characterize.yml -o ./report/  # optional
biahub estimate-psf     -i ./beads.zarr -c ./psf.yml -o ./psf.zarr
biahub deconvolve       -i ./lightsheet.zarr -c ./deconvolve.yml \
                        --psf-dirpath ./psf.zarr -o ./lightsheet_deconvolved.zarr

# DESKEW FLUORESCENCE
biahub estimate-deskew -i ./lightsheet.zarr/0/0/0 -o ./deskew.yml
biahub deskew          -i ./lightsheet.zarr/*/*/* -c ./deskew.yml -o ./lightsheet_deskewed.zarr

# RECONSTRUCT PHASE/BIREFRINGENCE
biahub reconstruct -i ./labelfree.zarr/*/*/* -c ./recon.yml -o ./labelfree_reconstructed.zarr

# STABILIZE
biahub estimate-stabilization -i ./labelfree.zarr/*/*/* -o ./stabilization.yml \
                              --stabilize-xy --stabilize-z
biahub stabilize              -i ./labelfree.zarr/*/*/* -c ./stabilization.yml \
                              -o ./labelfree_stabilized.zarr

# REGISTER
biahub estimate-registration -s ./labelfree_reconstructed.zarr/0/0/0 \
                             -t ./lightsheet_deskewed.zarr/0/0/0 -o ./register.yml
biahub optimize-registration -s ./labelfree_reconstructed.zarr/0/0/0 \
                             -t ./lightsheet_deskewed.zarr/0/0/0 \
                             -c ./register.yml -o ./register_optimized.yml
biahub register              -s ./labelfree_reconstructed.zarr/*/*/* \
                             -t ./lightsheet_deskewed.zarr/*/*/* \
                             -c ./register_optimized.yml -o ./registered.zarr

# CONCATENATE CHANNELS
biahub concatenate -c ./concatenate.yml -o ./concatenated.zarr

# STITCH
biahub estimate-stitch -i ./acq_name.zarr/*/*/* -o ./stitch.yml
biahub stitch          -i ./acq_name.zarr/*/*/* -c ./stitch.yml -o ./stitched.zarr

The nextflow/ directory contains Nextflow pipelines that run a whole reconstruction end to end for established workflows, fanning each step out over positions and submitting the work to Slurm.

nextflow/
├── nextflow.config   # executor profiles, resources, retry policy, reports
├── mantis-v2.nf      # mantis-v2 pipeline: the step order and directory layout
└── modules/          # one path-agnostic subworkflow per step

Each step module runs the same CLI you would run by hand, in two phases: biahub <step> --init creates the output store and reports the CPU/memory/time a single position needs, then Nextflow fans out one biahub <step> --cluster debug task per position with those resources. --cluster debug makes the CLI do the work in-process, so Nextflow — not submitit — owns job submission.

Install Nextflow (requires Java 17+), then launch from the nextflow/ directory so that nextflow.config is picked up automatically:

cd nextflow
nextflow run mantis-v2.nf \
    -profile slurm \
    --input /path/to/raw.zarr \
    --output /path/to/output \
    --flat_field_config    ./configs/flat_field.yml \
    --deskew_config        ./configs/deskew.yml \
    --reconstruct_config   ./configs/reconstruct.yml \
    --virtual_stain_config ./configs/virtual_stain.yml \
    --concatenate_config   ./configs/concatenate.yml \
    --track_config         ./configs/track.yml \
    --biahub_project /path/to/biahub

With --biahub_project, each task runs as uv run --project <path> biahub ...; omit it to use whatever biahub is on PATH on the compute node. --max_positions is useful for smoke tests on a handful of positions.

Profiles select where work runs:

• -profile local — everything on the current machine. Good for debugging and single-node runs.
• -profile slurm — per-position work goes to the preempted partition, virtual-stain prediction to the gpu partition, and lightweight init steps stay local. Preempted, timed-out, and OOM-killed tasks are retried automatically (up to 5 times, with escalating time/memory); genuine errors fail fast instead of burning retries.

Nextflow's own flags still apply — most usefully -resume to reuse completed tasks after a failure or a config tweak, and -w /fast/scratch to move the work directory off the output filesystem.

Each step writes <dataset>.zarr into its own numbered subdirectory of --output, so intermediates stay inspectable:

output/
├── 0-flatfield/     1-deskew/     2-reconstruct/
├── 3-virtual-stain/ 4-track/      5-assemble/
└── nextflow/
    ├── report.html  timeline.html  dag.html  trace.txt
    ├── slurm_output/<step>/        # per-task Slurm logs
    └── work/                       # Nextflow work directory

The pipeline runs flat-field → deskew → reconstruct → virtual-stain → assemble → track; 5-assemble concatenates the deskew, reconstruct, and virtual-stain channels into one plate, which tracking then consumes.

The step modules are path-agnostic: each subworkflow is handed an input zarr, an output zarr, and a config, and knows nothing about where it sits in the pipeline. The pipeline file owns the directory layout (the directory_layout() map) and the order of steps, so reordering steps, dropping one, or pointing a step at a different upstream store is an edit in mantis-v2.nf alone. Copy it as a starting point for a new instrument or dataset.

We would appreciate bug reports and code contributions if you use this package. If you would like to contribute to this package, please read the contributing guide.