EasyBuild Setup for RCC Clusters (Midway3/Beagle3)

Parmanand Sinha

This document outlines the configuration and usage of the EasyBuild environment setup primarily for the RCC Midway3 and Beagle3 clusters.

EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way.

This specific setup utilizes a central installation prefix and Tcl-based Environment Modules.

To use EasyBuild with this configuration, source the setup script and load the desired EasyBuild module version:

source /software/easybuild/ebtcl.sh
ml EasyBuild/5.0.0 # Or another available version

The ebtcl.sh script configures necessary environment variables and module paths.

This EasyBuild environment uses the following configuration (set via ebtcl.sh and /home/$USER/.config/easybuild/config.cfg):

• Installation Prefix: /software/easybuild
  • Software Path: /software/easybuild/software
  • Module Path: /software/easybuild/modules (Primarily uses /software/easybuild/modules/all/Core initially)
  • Source Path: /software/easybuild/sources
  • Build Path: /tmp/easybuild/build (Temporary build directory)
  • Package Path: /software/easybuild/packages
  • Container Path: /software/src/easybuild/containers
• Module System: EnvironmentModules
• Module Syntax: Tcl
• Module Naming Scheme: HierarchicalMNS
• Recursive Module Unload: Enabled
• Job Submission Backend: Slurm
• Parallel Builds: Configured for up to 8 cores (parallel=8).
• RPATH: Enabled (EASYBUILD_RPATH=1). Installation directories are added to the RPATH of executables/libraries.
• Experimental Features: Enabled (EASYBUILD_EXPERIMENTAL=1). Necessary for certain advanced features or configurations like easystack files.
• OS Dependencies: Ignored (EASYBUILD_IGNORE_OSDEPS=1). Assumes necessary OS-level packages are present.
• Filtered Dependencies: The following system dependencies are filtered and assumed to be provided by the OS: Autoconf, bzip2, DBus, libreadline, M4, ncurses, util-linux, XZ, zlib. EasyBuild will not build these.
• Filtered Environment Variables: LD_LIBRARY_PATH is filtered from generated module files.
• Permissions: Group ID (GID) bit is set on newly created directories (set-gid-bit=True).

While not explicitly configured in ebtcl.sh, the workspace contains custom EasyBuild repositories:

• custom-config/easybuild-easyconfigs: Custom easyconfig files.
• custom-config/easybuild-easyblocks-2023: Custom easyblocks.
• custom-config/easybuild-framework: Potentially a custom framework version.

These might be picked up via the EASYBUILD_ROBOT_PATHS or included in the main configuration file if configured globally.

• Ignoring OS Dependencies: Be aware that EASYBUILD_IGNORE_OSDEPS=1 is set. Ensure all required OS-level libraries and tools are installed on the system before building software.
• Filtered Dependencies: Builds might fail if they strictly require a version of a filtered dependency different from the one provided by the OS.
• Experimental Flag: Using experimental features might lead to unexpected behavior as they can change between EasyBuild versions.

• EasyBuild Documentation: https://docs.easybuild.io/
• EasyBuild Tutorial: https://tutorial.easybuild.io/ (Rendered at https://easybuilders.github.io/easybuild-tutorial)

The current installation uses EasyBuild 5.0, which introduces several significant changes from previous 4.x versions:

• Python 3.6+: EasyBuild 5.0 requires Python 3.6 or higher (Python 3.9+ recommended)
• Modules Tool: If using Lmod, version 8.0+ is required; for Environment Modules, version 4.3.0+ is required

• RPATH Linking: Enabled by default (was already enabled in our setup)
• Trace Output: Enabled by default (was already enabled in our setup)
• Module File Improvements: Module files now use depends_on for dependencies by default and include extensions statements
• Checksums: SHA256 is now the default checksum type
• Sanity Checks: Commands now run from an empty temporary directory rather than the installation directory
• Build Parallelism: Default maximum build parallelism is increased to 16 (our configuration uses 8)

• Interactive Debugging: Failed shell commands can be debugged via auto-generated env.sh and cmd.sh scripts
• Shell Command Function: New run_shell_cmd function replaces the older run_cmd and run_cmd_qa functions
• Easystack Support: Easystack files (software stack definitions in YAML format) are now stable and no longer experimental
• Granular Exit Codes: More specific exit codes for different error types
• Header File Control: New options for controlling how header file paths are handled

• Bootstrap Script: The EasyBuild bootstrap script has been removed
• .yeb Format: Support for YAML-based easyconfigs (.yeb) has been removed
• 32-bit Targets: Support for 32-bit targets has been removed
• Dummy Toolchain: Replaced with 'system' toolchain

• Several features from 4.x have been deprecated and will be removed in 6.0:
  • parallel easyconfig parameter is deprecated
  • run_cmd and run_cmd_qa functions are deprecated (use run_shell_cmd instead)
  • The 'source' step has been renamed to 'extract'
  • Several methods in the EasyBlock class have been renamed or deprecated

To update to the latest EasyBuild version:

source /software/easybuild/ebtcl.sh
ml EasyBuild/4.9.4  # Load an existing version first
eb --install-latest-eb-release  # Install the latest version (currently 5.0.0)

For more details on EasyBuild 5.0 changes, see the official documentation.

While the default setup uses a hierarchical structure within /software/easybuild, you might need to install software into a specific custom directory with a flat module layout (no subdirectories based on toolchains). This can be achieved by overriding specific EasyBuild environment variables and using the --installpath command-line option.

1. Set Environment Variables: Before running eb, set the following environment variables in your shell:

   • export EASYBUILD_MODULE_NAMING_SCHEME=EasyBuildMNS: Switches to a flat module naming scheme. Modules will be named like Software/Version-Toolchain-Version instead of being nested under toolchain paths.
   • export EASYBUILD_SUFFIX_MODULES_PATH="": Prevents adding the /all suffix to the module path. Modules will be placed directly in <installpath>/modules.
   • export EASYBUILD_SUBDIR_SOFTWARE="": Prevents creating the software/ subdirectory within the installation path. Software will be installed directly under <installpath>.

2. Unset Potential Conflicting Variables: To prevent dependency issues:

   unset LD_LIBRARY_PATH
   unset CMAKE_LIBRARY_PATH

3. Install with Custom Path: Use the --installpath option to specify where software should be installed:

   eb Python-bundle-PyPI-2023.10-GCCcore-13.2.0.eb --robot --installpath /tmp/pnsinha/python-3.11.5+gcc-13.2.0 -f -D --rebuild

4. Optional Architecture Flags: If needed, specify architecture optimization flags:

   eb Python-bundle-PyPI-2023.10-GCCcore-13.2.0.eb --robot --installpath /tmp/pnsinha/python-3.11.5+gcc-13.2.0 -f -D \
       --optarch="-march=x86-64-v3 -mtune=generic" --rebuild

This approach allows you to maintain isolated installations for specific purposes or testing environments.

This setup includes several custom easyconfigs and easystack files to address specific build issues encountered on our clusters. These custom configurations are especially important for complex packages like RStudio-Server that depend on many components and might fail due to system incompatibilities.

The following custom easyconfigs have been created to address specific issues:

1. Xvfb-21.1.14-GCCcore-13.3.0.eb

   • Creates a wrapper module that points to the system Xvfb installation
   • Bypasses the compilation failures related to missing SHA1 headers
   • Located in custom-easyconfigs/x/Xvfb/

2. cairo-1.18.0-GCCcore-13.3.0.eb

   • Disables problematic features (dwrite, spectre, tests) that cause build failures
   • Uses proper Meson build options to avoid linking issues
   • Located in custom-easyconfigs/c/cairo/

3. binutils-2.42.eb

   • Adds additional source URLs to ensure the package can be downloaded
   • Fixed download failures by adding sourceware.org mirror
   • Located in custom-easyconfigs/b/binutils/

4. OpenSSL-3.eb

   • Creates a wrapper for system OpenSSL 3.x
   • Falls back to building from source if system version is incompatible
   • Located in custom-easyconfigs/o/OpenSSL/

5. RStudio-Server-2024.12.0+467-foss-2024a-Java-17-R-4.4.2.eb

   • Uses all the custom dependencies mentioned above
   • Disables problematic components like QtWebEngine
   • Adds additional configuration options to improve build success
   • Located in custom-easyconfigs/r/RStudio-Server/

EasyStack files provide a way to define complete software stacks in YAML format. The repository includes several easystack files, with rstudio_server_2024a.yaml specifically designed for building RStudio-Server with all required dependencies.

To build the complete RStudio-Server stack:

1. Source the environment setup script:

   source /software/easybuild/ebtcl.sh
   ml EasyBuild/5.0.0

2. Build using the easystack file:

   eb --easystack=easystack/rstudio_server_2024a.yaml

3. For selective rebuilds of components that failed, you can target specific easyconfigs:

   # Example: Rebuild just RStudio-Server using the custom easyconfig
   eb custom-easyconfigs/r/RStudio-Server/RStudio-Server-2024.12.0+467-foss-2024a-Java-17-R-4.4.2.eb --robot

If you encounter build failures:

1. Check build logs:

   # Find the most recent error
   cat /tmp/*/eb-*/easybuild-*.log | grep -A 20 "ERROR:"

2. Try debugging mode:

   # Enable more verbose output
   eb custom-easyconfigs/r/RStudio-Server/RStudio-Server-2024.12.0+467-foss-2024a-Java-17-R-4.4.2.eb --robot --debug

3. Skip tests if they're causing issues:

   eb custom-easyconfigs/r/RStudio-Server/RStudio-Server-2024.12.0+467-foss-2024a-Java-17-R-4.4.2.eb --robot --skip-test-step

4. Force rebuilds when needed:

   eb custom-easyconfigs/r/RStudio-Server/RStudio-Server-2024.12.0+467-foss-2024a-Java-17-R-4.4.2.eb --robot --rebuild

5. Use trace mode for more detailed execution information:

   eb custom-easyconfigs/r/RStudio-Server/RStudio-Server-2024.12.0+467-foss-2024a-Java-17-R-4.4.2.eb --robot --trace

If you encounter issues downloading software sources, you can:

1. Manually Download and Place in Sources Directory:

   # Create proper subdirectory in sources/
   mkdir -p /software/easybuild/sources/x/PACKAGENAME
   
   # Download using curl or wget to the correct location
   cd /software/easybuild/sources/x/PACKAGENAME
   curl -L "URL_TO_SOURCE" -o source_filename.tar.gz

2. Modify URLs in Easyconfig Files: Create custom easyconfig files with updated URLs by copying the originals:

   mkdir -p /software/easybuild/custom-easyconfigs/x/PACKAGENAME
   cp <original>.eb /software/easybuild/custom-easyconfigs/x/PACKAGENAME/
   # Edit the source_urls list to include additional mirrors

For build failures, these approaches can help:

1. Check the Detailed Build Log:

   # Look for the most recent error
   cat /tmp/*/eb-*/easybuild-*.log | grep -A 10 "ERROR:"

2. Modify Build Options: Create custom easyconfigs to change build parameters, such as:

   • Disable problematic features: Add flags like -Dfeature=disabled for Meson builds
   • Adjust compiler flags: Modify configopts for specific needs
   • Update dependencies: Change version requirements for problematic dependencies

3. Ignore Checksums During Testing:

   eb my_package.eb --ignore-checksums

When encountering library incompatibilities:

1. Check Environment:

   # Ensure clean environment
   unset LD_LIBRARY_PATH
   unset CMAKE_LIBRARY_PATH

2. Force Specific Toolchain:

   # Specify architecture optimizations
   eb package.eb --optarch="-march=x86-64-v3 -mtune=generic"

Remember to unset or reset these environment variables if you want to revert to the default hierarchical installation behavior for subsequent builds.