ATP Abnormal Termination Processing

Abnormal Termination Processing (ATP) is a system that monitors user applications, and should an application take a system trap, ATP performs analysis on the dying application.

• All of the stack backtraces of the application processes are gathered into a merged stack backtrace tree and written to disk as the file "atpMergedBT.dot".

• The stack backtrace for the first process to die is sent to stderr as is the number of the signal that caused the death.

• If the core file size limit (RLIMIT_CORE) is non-zero, a heuristically selected set of processes dump their core.

• atpMergedBT.dot can be viewed with 'statview', a component of the STAT package (module load cray-stat). The merged stack backtrace tree provides a concise, yet comprehensive, view of what the application was doing at the time of its death.

Other HPE packages via either open-source or CPE rpms:

• cti-devel
• zmqnet

Other tools needed to build ATP

• meson
• autotconf
• autotools
• python3

This version of atp supports building either stand-alone or within the HPE build pipelines. This is mosly confinded to the 'hpe' directory, but there are some ".sh" and ".yaml" files that are required to live in the top level directory. Also, the HPE build relies on submodules. For the open source build, you do not need to git submodule update which would fail because they refer to internal repositories.

Given a shared prefix $prefix, you can build cti, zmqnet and atp in turn with:

export PKG_CONFIG_PATH=$prefix/lib/cti/pkgconfig:$prefix/lib/zmqnet/pkgconfg:$PKG_CONFIG_PATH

for i in cti zmqnet atp; do
   pushd $i
   meson setup -Dprefix=$prefix -Dlibdir=lib/$i -Dlibexecdir=libexec/$i ./build
   popd
done

Assuming you've installed the prequisites for all three packages.

One could potentially create cti.wrap and zmqnet.wrap files to build the dependencies as meson subprojects, but the build definitions are not expected to be stable enough yet for that approach to be reliable.

Building is currently only set up within the HPE environment which is automated via GitHub Actions in .github/workflows/build_pipeline.yaml

The build scripts can be used directly assuming you have access to the internal HPE artifactory and GitHub repo to get the dependencies.

After checking out a version of the repository:

• git submodule update --init --recursive
• manually update the build_number variable in the "release_versioning" file

The ATP build is set up to work on a VM based on one of the supported OS's and that you have root access to add dependencies

• sudo ./hpe/runBuildPrep.sh

This installs the packages needed for the build. This is rpms from the OS, like compilers and "devel" packages, plus the CDST rpms from the artifactory:

• cray-cti
• zmqnet
• cray-dyninst

The build is set up to build the artifacts into the expected CPE install location: /opt/cray/pe/atp/[version]

You can either make sure the directory "/opt/cray/pe" is writable by you, or continue the build as root.

To build:

./hpe/runBuild.sh builds the ATP binaries.

The actual build is set up via autotools, so inside the script it's just doing the ./configure; make; make install cycle. So during a development cycle, you can often just use make or make install unless you've added a source file or changed build flags.

./hpe/runBuildPackage.sh creates the atp and atp-test rpms.

The functional tests are built into atp-test.rpm to simplify setting up the test on a real HPC machine. See tests/function/README.md for further instructions.

See doc/sphinx.

See src/README.md for a basic overview of the code structure.