Building Spheral on LC Systems

This guide explains the process for setting up and running Spheral on LC (Livermore Computing) systems.

Warning

Do not run the provided terminal commands on login nodes. Instead, select the proper scheduler and do one of the two options below:

  • Grab an allocation using salloc -N <# of nodes> --exclusive and run terminal commands normally.

  • Prepend terminal commands with srun -N <# of nodes> --exclusive.

  • Grab an allocation using flux alloc -xN <# of nodes> and run terminal commands normally.

  • Prepend terminal commands with flux run -xN <# of nodes>.

Cloning/Updating

Spheral uses multiple submodules, including BLT and PYB11Generator. Because of this, be sure to use the --recursive option when cloning Spheral:

git clone --recursive https://github.com/LLNL/Spheral

If you forget to use the --recursive argument or if you checkout from a different branch, you must run:

git submodule update --init --recursive

Similarly, to update an existing Spheral repo, use

git pull
git submodule sync --recursive
git submodule update --init --recursive

Third Party Libraries (TPLs)

Spheral uses Spack to build and install TPLs used by Spheral. Spheral includes Spack configurations and environments pre-configured for LC systems in scripts/spack/configs/$SYS_TYPE and scripts/spack/environments/$SYS_TYPE. This greatly simplifies building TPLs for Spheral.

Running TPL Manager

Spheral provides a tool (tpl-manager.py) in an attempt to simplify the Spack process for the user, which does the following:

  • Clone and setup a local Spack instance for Spheral.

  • Activate (and create, if necessary) a Spack environment for the current system architecture.

  • Generate a dependency tree of third party libraries (relative to the provided configuration).

  • Build and install all dependent libraries in the local Spack instance.

  • Generate a CMake host-config file for configuring Spheral builds. This file will be named <sys_type>-<spec>.cmake.

tpl-manager.py is located in scripts/devtools and must be run from the Spheral root directory. The most common usage of tpl-manager.py will look like:

./scripts/devtools/tpl-manager.py --spec spheral+mpi%gcc

The command above tells the TPL manager to build and install TPLs for Spheral with MPI enabled and using the GCC compiler. Overall, the following options can be provided to tpl-manager.py:

--spec <SPEC>

Spheral Spack spec to build. Optional, default behavior is to build all spec permutations in the current environment. Required if running on non-LC systems. See the Spack documentation on specs-dependencies for more information.

--show-specs

Display specs found in current environment, if environment exists.

--show-info

Display Spheral Spack info, including variants and dependencies.

--spack-dir <SPACK_DIR>

Where to create the Spack instance to use. Optional, default ./../spheral-spack-tpls.

--clean

Cleans Spack repo instance and forces Spack to reconcretize environment. Use if running into odd behavior with the script.

Configuring

After running tpl-manager, you will see a file in your Spheral root directory following the format <sys_type>-<spec>.cmake. This is the host config file that will be used to configure a new build of Spheral. We provide a tool at scripts/devtools/host-config-build.py to simplify Spheral’s CMake setup. Use it with the host config file like so:

./scripts/devtools/host-config-build.py --host-config <sys_type>-<spec>.cmake

Running host-config-build.py

Overall, the following options can be provided to host-config-build.py

--host-config <HOST_CONFIG_FILE>

Name of the host config file created by tpl-manager.py. Typically named <sys_type>-<spec>.cmake. Required.

-s, --source-dir <SOURCE_DIR>

Location of a Spheral source repo. Optional, default is current directory.

--build-dir <BUILD_DIR>

Spheral build directory. Optional, default <SOURCE_DIR>/build_<sys-type>-<spec>. Makes the Spheral build directory in <BUILD_DIR>/build.

-i, --install-dir <INSTALL_DIR>

Spheral installation directory. Optional, default <SOURCE_DIR>/<BUILD_DIR>/install.

--no-clean

Do not delete build and install locations. Optional.

--build

Run make -j <N> install after configuring. Optional.

-D<CMAKE_ARGUMENTS>

Any additional CMake arguments. See Spheral CMake Configurations for possible options.

Example:

./scripts/devtools/host-config-build.py --host-config linux-ubuntu20-gcc@10.3.1.cmake -DENABLE_DEV_BUILD=ON

Build and Install

In order to build Spheral, navigate to the <BUILD_DIR> from the Configure step and run make -j <N> install, where <N> should be replaced with the number of processes to use (typicall the number of cores on your CPU).

Although Spheral is simply a set of Python modules, Spheral sets up a Python virtual environment during the install stage. The install will also download and install all required runtime python libraries needed to execute the full Spheral test suite.

Users will find a spheral script in the install directory, which can be run as ./spheral. This will drop users into the active Spheral virtual environment, providing access to all Spheral libraries and dependencies. To ensure Spheral uses multiple ranks, run ./spheral with the appropriate run command, i.e. srun -n <number of procs> ./spheral for Slurm machines, flux run -xN <number of nodes> ./spheral, etc.

Running Tests

Basic Smoke Test

After a build and install, we recommend you perform a smoke test with Spheral to ensure the Spheral environment is installed and linked correctly.

From your install directory run:

./spheral -c "import Spheral"

This will run the Spheral python process and import the Spheral modules. If successful, you should see a banner printed with information about the version of Spheral, after which Python exits.

ATS Testing Suite

Spheral uses ATS (Automated Testing System) to execute a suite of parallel tests. During install, a script spheral-ats is created. This script handles launching allocations and setting appropriate ATS and hardware flags for running the testing suite.

From the install directory run:

./spheral-ats tests/integration.ats

For more information about using spheral-ats, run the following from the install directory:

./spheral-ats --help