RCC Internal Workshop

Parmanand Sinha July 17, 2023

Things go in standard directories: binaries in /bin, libraries in /lib, configuration in /etc, etc. But not research programs on Midway. Why?

● Lots of software, much of it is non-standard.

● We want different versions of the same software package available at the same time, but without conflicts.

● Dependency management: different codes may require different versions of libraries.

Download the source and install it yourself

configure // Compilation environment setting result: Makefile creation Default install location: /usr/local/***

make // compile. Run Makefile make install // copy

Command sequence

1. Unzip
2. run configure
3. make
4. make install
5. Update $PATH

$PATH is a list of paths separated by colons containing all the location to look for an executable.

Download and compile SQLite:

wget https://www.sqlite.org/2022/sqlite-autoconf-3360000.tar.gz
tar xvf sqlite-autoconf-3360000.tar.gz
cd sqlite-autoconf-3360000

./configure --prefix=$HOME/local
make -jN
make install

Download and compile GDAL:

wget http://download.osgeo.org/gdal/CURRENT/gdal-X.Y.Z.tar.gz
tar xvf gdal-X.Y.Z.tar.gz
cd gdal-X.Y.Z

./configure --prefix=$HOME/local --with-sqlite3=$HOME/local --with-static-proj4=/usr --with-geos=/usr/bin/geos-config
make -jN
make install

Software is installed at /software. ● Directories are named $SOFTNAME-$SOFTVERS-$OS-$ARCH

● For example, gedit lives at /software/gedit-2.28-el6-x86_64

● bin, lib, src, doc under that directory

‘Environment Modules’ are the mechanism by which much of the software is made available to the users.

What happens when you type “module load openmpi/1.6”? ● module looks for a file called

/software/modulefiles/openmpi/1.6 (or in ~/privatemodules …)

● Applies changes to the environment based on that file: ○ load module dependencies

○ adds $appdir/bin to PATH

○ adds $appdir/lib to LD_LIBRARY_PATH

○ adds man directory to MANPATH, include directory to CPATH, etc...

LMOD (Lmod) and Environment modules (TCLmodule) are both environment module systems for managing software packages and their dependencies on high-performance computing (HPC) systems.

TCLmodule are written in (Tcl) syntax.

Modules are not the only way of managing software on clusters: increasingly common approaches include:

• Conda package manager (Python-centric but can manage software written in any language);
• Apptainer/Singularity, a means for deploying software in containers.

A minimum example of TCL module file for loading Miniconda:

#%Module1.0
##
## Miniconda modulefile
##

## Specify the version of Miniconda
set version x.x.x

## Set the path for Miniconda
set home <path-to-miniconda>  # Replace with the actual path

## Add Miniconda binary paths to the shell's PATH variable
prepend-path PATH $home/bin

## Add Miniconda library paths to the LD_LIBRARY_PATH environment variable
prepend-path LD_LIBRARY_PATH $home/lib

## Set environmental variables specific to Miniconda
setenv CONDA_HOME $home

Midway3 Repo: https://git.rcc.uchicago.edu/rcc-staff/midway3-software.git

● Build scripts live in

Midway2: the SVN repo at pubsw/software/build Midway3: the git repo at midway3-software/build

● A subdirectory for every software package

● Build steps:

○ Create subdirectory and populate with configuration files ○ Run the build scripts from top-level directory ○ Test modulefile, then commit to repository.

buildopts.sh defines important variables for the build, notably BUILDTYPE. moduleconf.sh (new) defines the module dependancies and package dependancies of the software.

source.sh contains the command(s) required to download the source. test.bats contains tests to ensure the program was installed correctly. build.sh is a script that builds and installs the software package. This is only required if BUILDTYPE is custom. build.sh.pre and build.sh.post are optional scripts that are run before and after a build, respectively.

• configure -- ./configure; make; make install
• rsync -- use rsync to copy $SRCDIR to $PREFIX
• noop -- Only create the modulefile. The software is installed in other ways.
• custom -- use build.sh in the package directory

Content of source.sh

gitsource https://github.com/tmux/tmux.git

Content of buildopts.sh

SRCDIR=${SRCDIR}-${SOFTVERS}
BUILDTYPE=configure
CORES=8


export LIBEVENT_CFLAGS="-I${LIBEVENT_HOME}/include"
export LIBEVENT_LIBS="-L${LIBEVENT_HOME}/lib -levent_core" 

Content of moduleconf.sh

MODULEDESCRIPTION="tmux is a terminal multiplexer, it enables a number of terminals (or windows)
to be accessed and controlled from a single terminal."
MODULEDOCURL="https://github.com/tmux/tmux"
MODULELICENSE="opensource"
MODULETAGS="terminal"
BUILDDEPEND="libevent/2.1.12"
MODULEDEPEND="libevent/2.1.12"

Run build scripts

From the top-level folder: eg: for Midway3: midway3-software/build

SOFTNAME=tmux
SOFTVERS=3.2
. /get-source $SOFTNAME $SOFTVERS \# Get source for software

. /build $SOFTNAME $SOFTVERS \# compile software to /software/staging

. /build -i $SOFTNAME $SOFTVERS \# rsync from /software/staging to /software

● build puts modulefile in ~/privatemodules

● Test with “module load use.own; module load $SOFTNAME”

● If that works:

○ move ~/privatemodules/$SOFTNAME/$SOFTVER to midway3-software/modulefiles/$SOFTNAME

● Commit changes to repository

Example: R

Content of source.sh

curl -O https://cran.rstudio.com/src/base/R-4/R-${$SOFTVER}.tar.gz

Content of buildopts.sh

SRCDIR=${SRCDIR}-${SOFTVERS}
BUILDTYPE=custom

Content of build.sh

./configure --enable-R-shlib --enable-optimisations --enable-openmp \
  --enable-BLAS-shlib --with-lapack --with-blas="-lopenblas" \
  --disable-R-profiling --with-pcre1 --prefix=$PREFIX
make
make install

Content of moduleconf.sh

MODULEDOCNAME="R"
MODULEDESCRIPTION="The R language and environment for statistical computing \
and graphics."
MODULELICENSE="GNU General Public License v2"
MODULEDOCURL="https://www.r-project.org"
MODULETAGS="development, general programming, graphics, R, statistics"
SOFTWARESUFFIX=DISTARCH
MODULEDEPEND="openblas/0.3.13 java/15.0.2"
BUILDDEPEND=""
MODULECONFLICT="R"
MODULEUSAGE=''
MODULEEXTRA='
prepend-path LD_LIBRARY_PATH $appdir/lib64/R/lib
prepend-path LIBRARY_PATH $appdir/lib64/R/lib
setenv OMP_NUM_THREADS 1
'

Spack is an open source package manager that simplifies building, installing, customizing, and sharing HPC software stacks. It is written in pure Python including software build requirements and dependencies.

It follows Nix, GUIX based store model where all the software are installed inside its own filesystem

In midway3, two common repo is available at /software/spack-0.17.0-el8-x86_64/ and /software/spack-dev/ (Preferred for RCC CS) Step1: source /software/spack-0.17.0-el8-x86_64/share/spack/setup-env.sh

spack list <software-name> # list a software
spack info <software-name> # show information and variants of a software
spack spec -I <software-name> # show specifications (dependencies)

Option -I or --install-status shows status of the software dependencies i.e. installed (+) or will be installed during the installation (-).

Now let’s install a new software’s from Spack:

spack install <software_name> or <software_name@version> or <software_name@version %compiler@version> 

In general, @version for both software and compiler could be removed. Spack installs the most stable version by default (see spack versions -s <software-name>). You may find complete list of software that you can install by using spack list or in Spack online package list. Also. we can use --verbose option to see more details during the installation, --no-cache to install a package directly from the source, and --overwrite to overwrite an installed package.

After installation, we can find the software by:

spack find # to see all installed software
spack find <software-name> # to find a software (use -lfvp to see hashes, flags, variants and pathes)
spack location -i <software-name> # to find location of a software 

And we can load the software:

spack load <software_name>
spack find --loaded # see what is loaded

We can select compiler version and settings. To find and list compilers, use:

spack compiler list
spack compiler find

Compilers can be added to the Spack compilers list or removed from the list by:

spack compiler add <compiler-name@ver> # for example $(spack location -i gcc@10.1.0) add gcc 10 compiler that already is installed by Spack
spack compiler remove <compiler-name@ver>

Also, we can directly modify compilers.yaml file by:

spack config edit compilers

// Setting up inside scratch directory

git clone --depth=100 --branch=releases/v0.20 https://github.com/spack/spack.git $SCRATCH/midway3/$USER/spack

cd $SCRATCH/midway3/$USER/spack . share/spack/setup-env.sh