Tags: NL-BioImaging/biomero
Tags
Add Configurable Slurm Compatibility Features To BIOMERO (#30) Final PR Summary This PR upstreams a complete set of cluster-specific compatibility patches into core BIOMERO as backward-compatible, opt-in configuration options. All default behavior is unchanged unless explicitly enabled. What moved into core BIOMERO Feature Config key / env var Default Shared config/env precedence helper — (internal) — SACCT history window (absolute or relative) sacct_start_time / BIOMERO_SACCT_START_TIME, sacct_days_ago / BIOMERO_SACCT_START_DAYS_AGO unchanged ZIP/UNZIP command (7z or 7za, or explicit override) slurm_zip_cmd / BIOMERO_SLURM_ZIP_CMD auto-detect Env-file-based sbatch submission env_file_submission / BIOMERO_ENV_FILE_SUBMISSION off Generated-script env-file sourcing $OPTIONAL_ENV in job_template.sh no-op Generated-script GPU flag normalization (--nv) inject_gpu_flag / BIOMERO_INJECT_GPU_FLAG off Fallback GPU Slurm resources (gpu_partition, gpu_gres) GPU_PARTITION / BIOMERO_GPU_PARTITION, GPU_GRES / BIOMERO_GPU_GRES off (per-workflow settings still take precedence) Optional APPTAINER_BINDPATH export slurm_data_bind_path unset (not required) Optional conversion job partition slurm_conversion_partition unset Apptainer tmp/cache dir overrides slurm_apptainer_tmpdir, slurm_apptainer_cachedir unset sbatch-based image pull/build slurm_pull_command / BIOMERO_PULL_COMMAND off Script hardening (set -eo pipefail) and output verification job_template.sh shipped [MODELS] section renamed to [WORKFLOWS] config file backward-compatible alias kept slurm_default_partition fallback routing slurm_default_partition / BIOMERO_DEFAULT_PARTITION unset Global sbatch parameters for all job types sbatch_* keys in [SLURM] section unset What remains opt-in env-file submission, GPU flag injection, shared GPU fallback defaults, sbatch-based image pulling, apptainer tmp/cache overrides, slurm_data_bind_path, slurm_conversion_partition, slurm_default_partition What remains external-repo responsibility Custom slurm_script_repo contents and any cluster-specific job scripts not provided by BIOMERO Applied upstream to OMERO.biomero surf branch (not here) Metabase localhost/127.0.0.1 link rewrite (BiomeroApp.js) Tests and docs 37 commits, ~8 500 lines added across slurm_client.py, constants.py, job_template.sh, convert_job_array.sh, slurm-config.ini, docs/slurm-configuration.rst, docs/configuration-reference.rst, and tests/unit/test_slurm_client.py All existing tests remain green; new tests cover default behavior, opt-in paths, env override precedence, and the shared config helper Closes Temporary folder for cache building singularity containers at HPC #17
PreviousNext