Release Notes #

SUSE Linux Enterprise High-Performance Computing 15 SP2 is available for the Intel 64/AMD64 (x86-64) and AArch64 platforms.

If you are upgrading from a previous SUSE Linux Enterprise High-Performance Computing release, you should review at least the following sections:

SUSE Linux Enterprise High-Performance Computing is backed by award-winning support from SUSE, an established technology leader with a proven history of delivering enterprise-quality support services.

SUSE Linux Enterprise High-Performance Computing 15 has a 13-year life cycle, with 10 years of General Support and 3 years of Extended Support. The current version (SP2) will be fully maintained and supported until 6 months after the release of SUSE Linux Enterprise High-Performance Computing 15 SP3.

Any release package is fully maintained and supported until the availability of the next release.

Extended Service Pack Overlay Support (ESPOS) and Long Term Service Pack Support (LTSS) are also available for this product. If you need additional time to design, validate and test your upgrade plans, Long Term Service Pack Support (LTSS) can extend the support you get by an additional 12 to 36 months in 12-month increments, providing a total of 3 to 5 years of support on any given Service Pack.

For more information, see:

To receive support, you need an appropriate subscription with SUSE. For more information, see https://www.suse.com/support/programs/subscriptions/.

The following definitions apply:

For contracted customers and partners, SUSE Linux Enterprise High-Performance Computing 15 SP2 is delivered with L3 support for all packages, except for the following:

SUSE will only support the usage of original packages. That is, packages that are unchanged and not recompiled.

The SLE 15 SP2 product line is made up of modules that contain software packages. Each module has a clearly defined scope. Modules differ in their life cycles and update timelines.

The modules available within the product line based on SUSE Linux Enterprise 15 SP2 at the release of SUSE Linux Enterprise High-Performance Computing 15 SP2 are listed in the Modules and Extensions Quick Start at https://documentation.suse.com/sles/15-SP2/html/SLES-all/art-modules.html.

Not all SLE modules are available with a subscription for SUSE Linux Enterprise High-Performance Computing 15 SP2 itself (see the column Available for).

For information about the availability of individual packages within modules, see https://scc.suse.com/packages.

3.1.2 NVIDIA Compute Module #

The repositories for NVIDIA CUDA are available as the NVIDIA Compute module for x86-64 and AArch64. These repositories are provided by NVIDIA and the software in them is not supported by SUSE. All software in these repositories is licensed under the third-party NVIDIA CUDA EULA.

The NVIDIA Compute module is not enabled by default when installing SUSE Linux Enterprise Server. During installation, the module can be selected manually from the Extension and Modules screen in YaST. Within an installed system, you can add it as follows: Run yast registration from a shell as root, select Select Extensions, search for NVIDIA Compute Module and continue with Next. Verify and accept the NVIDIA repository GPG key.

Important: Do Not Use the SUSEConnect Tool to Add this Repository

Do not try to add this module with the SUSEConnect CLI tool. SUSEConnect is not yet capable of handling third-party repositories.

Important: Combining Workstation Extension and NVIDIA Compute Module Is Unsupported

The Workstation Extension module includes some of the same drivers for NVIDIA graphics cards as the NVIDIA Compute module. However, their package versions may differ. As SUSE package management installs the latest package versions by default, enabling both modules at the same time can lead to a system with a mixture of packages from both modules.

Such a setup can result in drivers not working as expected and is not supported by SUSE.

The following extension is not covered by SUSE support agreements, available at no additional cost and without an extra registration key: SUSE Package Hub, see https://packagehub.suse.com/.

This sections lists related products. Usually, these products have their own release notes documents that are available from https://www.suse.com/releasenotes.

With SUSE Linux Enterprise High-Performance Computing 15 SP2, it is possible to choose specific roles for the system based on modules selected during the installation process. When the HPC Module is enabled, these three roles are available:

The scratch partition /var/tmp/ will only be created if there is sufficient space available on the installation medium (minimum 32 GB).

The Environment Module Lmod will be installed for all roles. It is required at build time and run time of the system. For more information, see Section 7.7, “Lmod — Lua-based Environment Modules”.

All libraries specifically build for HPC will be installed under /usr/lib/hpc. They are not part of the standard search path, thus the Lmod environment module system is required.

munge authentication is installed for all roles. This requires copying the same generated munge keys to all nodes of a cluster. For more information, see Section 7.15, “mrsh/mrlogin — Remote Login Using MUNGE Authentication” and Section 7.14, “MUNGE Authentication”.

From the Ganglia monitoring system, the data collector ganglia-gmod is installed for every role, while the data aggregator ganglia-gmetad needs to be installed manually on the system which is expected to collect the data. For more information, see Section 7.3, “Ganglia — System Monitoring”.

The system roles are only available for new installations of SUSE Linux Enterprise High-Performance Computing.

This section includes upgrade-related information for the SUSE Linux Enterprise High-Performance Computing 15 SP2.

You can upgrade to SUSE Linux Enterprise High-Performance Computing 15 SP2 from SLES 12 SP3 or SUSE Linux Enterprise High-Performance Computing 12 SP3. When upgrading from SLES 12 SP3, the upgrade will only be performed if the SUSE Linux Enterprise High-Performance Computing module has been registered prior to upgrading. Otherwise, the system will instead be upgraded to SLES 15.

To upgrade from SLES 12 to SLES 15, make sure to deregister the SUSE Linux Enterprise High-Performance Computing module prior to upgrading. To do so, open a root shell and execute:

SUSEConnect -d -p sle-module-hpc/12/ARCH

Replace ARCH with the architecture used (x86_64, aarch64).

When migrating to SUSE Linux Enterprise High-Performance Computing 15 SP2, all modules not supported by the migration target need to be deregistered. This can be done by executing:

SUSEConnect -d -p sle-module-MODULE_NAME/12/ARCH

Replace MODULE_NAME by the name of the module and ARCH with the architecture used (x86_64, aarch64).

When migrating from SUSE Linux Enterprise High-Performance Computing 15 GA to SUSE Linux Enterprise High-Performance Computing 15 SP2, make sure to remove the Legacy Module if it is installed. Otherwise, the migration will fail with the error message No migration product found. To remove the legacy module, execute:

SUSEConnect -d -p sle-module-legacy/15/ARCH

Replace ARCH with the architecture used.

SUSE Linux Enterprise 15 uses Python 3 by default. Starting with SLE 15 SP1, the Python 2 runtime and modules have been moved to the Python 2 module. As SUSE Linux Enterprise High-Performance Computing 15 SP2 uses Python 2, you need to enable this module when upgrading from earlier versions.

cpuid executes the x86 CPUID instruction and decodes and prints the results to stdout. Its knowledge of Intel, AMD and Cyrix CPUs is fairly complete. It also supports Intel Knights Mill CPUs (x86-64).

To install cpuid, run: zypper in cpuid.

For information about runtime options for cpuid, see the man page cpuid(1).

Note that this tool is only available for x86-64.

ConMan is a serial console management program designed to support a large number of console devices and simultaneous users. It supports:

ConMan can be used for monitoring, logging and optionally timestamping console device output.

To install ConMan, run zypper in conman.

Important: conmand Sends Unencrypted Data

The daemon conmand sends unencrypted data over the network and its connections are not authenticated. Therefore, it should be used locally only: Listening to the port localhost. However, the IPMI console does offer encryption. This makes conman a good tool for monitoring a large number of such consoles.

Usage:

For more information about options, see the man page of ConMan.

Ganglia is a scalable distributed monitoring system for high-performance computing systems, such as clusters and grids. It is based on a hierarchical design targeted at federations of clusters.

To use Ganglia, make sure to install ganglia-gmetad on the management server. Then start the Ganglia meta-daemon: rcgmead start. To make sure the service is started after a reboot, run: systemctl enable gmetad. On each cluster node which you want to monitor, install ganglia-gmond, start the service rcgmond start and make sure it is enabled to be started automatically after a reboot: systemctl enable gmond. To test whether the gmond daemon has connected to the meta-daemon, run gstat -a and check that each node to be monitored is present in the output.

When using the Btrfs file system, the monitoring data will be lost after a rollback and the service gmetad will not start again. To fix this issue, either install the package ganglia-gmetad-skip-bcheck or create the file /etc/ganglia/no_btrfs_check.

Install ganglia-web on the management server. Depending on which PHP version is used (default is PHP 7), enable it in Apache2: a2enmod php7.

Then start Apache2 on this machine: rcapache2 start and make sure it is started automatically after a reboot: systemctl enable apache2. The Ganglia Web interface should be accessible from http://MANAGEMENT_SERVER/ganglia-web.

Support for Genders has been added to the HPC module.

Genders is a static cluster configuration database used for configuration management. It allows grouping and addressing sets of hosts by attributes and is used by a variety of tools. The Genders database is a text file which is usually replicated on each node in a cluster.

Perl, Python, Lua, C, and C++ bindings are supplied with Genders, the respective packages provide man pages or other documentation describing the APIs.

To create the Genders database, follow the instructions and examples in /etc/genders and check /usr/share/doc/packages/genders-base/TUTORIAL. Testing a configuration can be done with nodeattr (for more information, see man 1 nodeattr).

List of packages:

On SLE-HPC the GNU compiler collection version 7 is provided as the base compiler. The gnu-compilers-hpc provides the environment module for the base version of the GNU compiler suite. This package must be installed when using any of the HPC libraries enabled for environment modules.

zypper in gnu-compilers-hpc

module load gnu

zypper search '*-compilers-hpc'

module load gnu/7

hwloc provides CLI tools and a C API to obtain the hierarchical map of key computing elements, such as: NUMA memory nodes, shared caches, processor packages, processor cores, processing units (logical processors or “threads”) and even I/O devices. hwloc also gathers various attributes such as cache and memory information, and is portable across a variety of different operating systems and platforms. Additionally it can assemble the topologies of multiple machines into a single one, to let applications consult the topology of an entire fabric or cluster at once.

lstopo allows the user to obtain the topology of a machine or convert topology information obtained on a remote machine into one of several output formats. In graphical mode (X11), it displays the topology in a window, several other output formats are available as well , including plain text, PDF, PNG, SVG and FIG. For more information, see the man pages provided by hwloc and lstopo.

It also features full support for import and export of XML-formatted topology files via the libxml2 library.

The package hwloc-devel offers a library that can be directly included into external programs. This requires that the libxml2 development library (package libxml2-devel) is available when compiling hwloc.

Lmod is an advanced environment module system which allows the installation of multiple versions of a program or shared library, and helps configure the system environment for the use of a specific version. It supports hierarchical library dependencies and makes sure that the correct version of dependent libraries are selected. Environment Modules-enabled library packages supplied with the HPC module support parallel installation of different versions and flavors of the same library or binary and are supplied with appropriate lmod module files.

To install Lmod, run: zypper in lua-lmod.

Before Lmod can be used, an init file needs to be sourced from the initialization file of your interactive shell. The following init files are available:

/usr/share/lmod/lmod/init/bash
/usr/share/lmod/lmod/init/ksh
/usr/share/lmod/lmod/init/tcsh
/usr/share/lmod/lmod/init/zsh
/usr/share/lmod/lmod/init/sh

Pick the appropriate file for your shell. Then add the following to the init file of your shell:

source /usr/share/lmod/lmod/init/<INIT-FILE>

The init script adds the command module.

To list available modules, run: module spider. To show all modules which can be loaded with the currently loaded modules, run: module avail. A module name consists of a name and a version string separated by a / character. If more than one version is available for a certain module name, the default version (marked by *). If there is no default, the module with the highest version number is loaded. To reference a specific module version, you can use the full string NAME/VERSION.

module list shows all currently loaded modules. Refer to module help for a short help on the module command and module help MODULE-NAME for a help on the particular module. The module command is only available when you log in after installing lua-lmod.

To get information about a particular module, run: module whatis MODULE-NAME. To load a module, run: module load MODULE-NAME. This will ensure that your environment is modified (that is, the PATH and LD_LIBRARY_PATH and other environment variables are prepended) such that binaries and libraries provided by the respective modules are found. To run a program compiled against this library, the appropriate module load commands must be issued beforehand.

The module load MODULE command needs to be run in the shell from which the module is to be used. Some modules require a compiler toolchain or MPI flavor module to be loaded before they are available for loading.

If the respective development packages are installed, build-time environment variables like LIBRARY_PATH, CPATH, C_INCLUDE_PATH and CPLUS_INCLUDE_PATH will be set up to include the directories containing the appropriate header and library files. However, some compiler and linker commands may not honor these. In this case, use the appropriate options together with the environment variables -I PACKAGE_NAME_INC and -L PACKAGE_NAME_LIB to add the include and library paths to the command lines of the compiler and linker.

For more information on Lmod, see https://lmod.readthedocs.org.

pdsh is a parallel remote shell which can be used with multiple back-ends for remote connections. It can run a command on multiple machines in parallel.

To install pdsh, run zypper in pdsh.

On SLE-HPC, the back-ends ssh, mrsh, and exec are supported. The ssh back-end is the default. Non-default login methods can be used by either setting the PDSH_RCMD_TYPE environment variable or by using the -R command argument.

When using the ssh back-end, it is important that a non-interactive (that is, passwordless) login method is used.

The mrsh back-end requires the mrshd to be running on the client. The mrsh back-end does not require the use of reserved sockets. Therefore, it does not suffer from port exhaustion when executing commands on many machines in parallel. For information about setting up the system to use this back-end, see Section 7.15, “mrsh/mrlogin — Remote Login Using MUNGE Authentication”.

Remote machines can be specified on the command line. Alternatively, pdsh can use a machines file (/etc/pdsh/machines), dsh-style (Dancer's shell) groups, or netgroups. It can target also nodes based on the currently running Slurm jobs.

The different ways to select target hosts are realized by modules. Some of these modules provide identical options to pdsh. The module loaded first will win and handle the option. Therefore, we recommend limiting yourself to a single method and specifying this with the -M option.

The machines file lists all target hosts one per line. The appropriate netgroup can be selected with the -g command line option.

The following host-list plugins for pdsh are supported: machines, slurm, netgroup and dshgroup. Each host-list plugin is provided in a separate package. This avoids conflicts between command-line options for different plugins which happen to be identical and helps to keep installations small and free of unnecessary dependencies. Package dependencies have been set up to prevent the installation of plugins with conflicting command options. To install one of the plugins, run:

zypper in pdsh-PLUGIN_NAME

For more information, see the man page pdsh.

PowerMan allows manipulating remote power control devices (RPC) from a central location. It can control:

The communication to RPCs is controlled by “expect”-like scripts. For a list of currently supported devices, see the configuration file /etc/powerman/powerman.conf.

To install PowerMan, run zypper in powerman.

To configure it, include the appropriate device file for your RPC (/etc/powerman/*.dev) in /etc/powerman/powerman.conf and add devices and nodes. The device “type” needs to match the “specification” name in one of the included device files, the list of “plugs” used for nodes need to match an entry in the “plug name” list.

After configuring PowerMan, start its service by:

systemctl start powerman.service

To start PowerMan automatically after every boot, do:

systemctl enable powerman.service

Optionally, PowerMan can connect to a remote PowerMan instance. To enable this, add the option listen to /etc/powerman/powerman.conf.

Important: Unencrypted Transfer

Data is transferred unencrypted, therefore this is not recommended unless the network is appropriately secured.

rasdaemon is a RAS (Reliability, Availability and Serviceability) logging tool. It records memory errors using the EDAC tracing events. EDAC drivers in the Linux kernel handle detection of ECC errors from memory controllers.

rasdaemon can be used on large memory systems to track, record and localize memory errors and how they evolve over time to detect hardware degradation. Furthermore, it can be used to localize a faulty DIMM on the board.

To check whether the EDAC drivers are loaded, execute:

ras-mc-ctl --status

The command should return ras-mc-ctl: drivers are loaded. If it indicates that the drivers are not loaded, EDAC may not be supported on your board.

To start rasdaemon, run systemctl start rasdaemon.service. To start rasdaemon automatically at boot time, execute systemctl enable rasdaemon.service. The daemon will log information to /var/log/messages and to an internal database. A summary of the stored errors can be obtained with:

ras-mc-ctl --summary

The errors stored in the database can be viewed with:

ras-mc-ctl --errors

Optionally, you can load the DIMM labels silk-screened on the system board to more easily identify the faulty DIMM. To do so, before starting rasdaemon, run:

systemctl start ras-mc-ctl start

For this to work, you need to set up a layout description for the board. There are no descriptions supplied by default. To add a layout description, create a file with an arbitrary name in the directory /etc/ras/dimm_labels.d/. The format is:

Vendor: VENDOR-NAME
 Model: MODEL-NAME
   LABEL: MC.TOP.MID.LOW

Slurm is an open-source, fault-tolerant, and highly scalable cluster management and job scheduling system for Linux clusters containing up to 65,536 nodes. Components include machine status, partition management, job management, scheduling and accounting modules.

7.11.3 Upgrading Slurm #

7.11.3.1 Slurm Upgrade Compatibility #

New major versions of Slurm are released in regular intervals. With some restrictions (see below), interoperability is guaranteed between 3 consecutive versions. However, unlike updates to maintenance releases (that is, releases which differ in the last version number), upgrades to newer major versions may require more careful planning.

For existing products under general support, version upgrades of Slurm are provided regularly. Unlike maintenance updates, these upgrades will not be installed automatically using zypper patch but require the administrator to request their installation explicitly. This ensures that these upgrades are not installed unintentionally and gives the administrator the opportunity to plan version upgrades beforehand.

On new installations, we recommend installing the latest available version.

Slurm uses a segmented version number: The first two segments denote the major version, the final segment denotes the patch level.

Check the list below for available major versions.

Upgrade packages (that is, packages that were not a part of the module or service pack initially) have their major version encoded in the package name (with periods . replaced by underscores _). For example, for version 18.08, this would be: slurm_18_08-*.rpm.

To upgrade the package slurm to 18.08, run the command:

zypper install --force-resolution slurm_18_08

To upgrade Slurm subpackages, proceed analogously.

Important

If any additional Slurm packages are installed, make sure to upgrade those as well. These include:

• slurm-pam_slurm

• slurm-sview

• perl-slurm

• slurm-lua

• slurm-torque

• slurm-config-man

• slurm-doc

• slurm-webdoc

• slurm-auth-none

• pdsh-slurm

All Slurm packages should be upgraded at the same time to avoid conflicts between packages of different versions. This can be done by adding them to the zypper install command line described above.

In addition to the “three-major version rule” mentioned at the beginning of this section, obey the following rules regarding the order of updates:

1. The version of slurmdbd must be identical to or higher than the version of slurmctld

2. The version of slurmctld must the identical to or higher than the version of slurmd

3. The version of slurmd must be identical to or higher than the version of the slurm user applications.

Or in short:

version(slurmdbd) >= version(slurmctld) >= version(slurmd) >= version (Slurm user CLIs).

With each version, configuration options for slurmctld/slurmd or slurmdbd may be deprecated. While deprecated, they will remain valid for this version and the two subsequent versions but they may be removed later.

Therefore, it is advisable to update the configuration files after the upgrade and replace deprecated configuration options before finally restarting a service.

It should be noted that a new major version of Slurm introduces a new version of libslurm. Older versions of this library might no longer work with an upgraded Slurm. For all SLE software depending on libslurm an upgrade will be provided. Any locally developed Slurm modules or tools might require modification and/or recompilation.

7.11.3.2 Upgrade Workflow #

For this workflow it is assumed that MUNGE authentication is used and that pdsh, the pdsh Slurm plugin and mrsh can be used to access all machines of the cluster. That means, mrshd is running on all nodes in the cluster.

If this is not the case, install pdsh:

# zypper in pdsh-slurm

If mrsh is not used in the cluster, the SSH back-end for pdsh can be used as well for this: Replace the option -R mrsh with -R ssh in the pdsh commands below. This is less scalable and you may run out of usable ports.

Warning: Upgrade slurmdbd databases before other Slurm components

If slurmdbd is used, always upgrade the slurmdbd database before starting the upgrade of any other Slurm component. The same database can be connected multiple clusters and must be upgraded before all of them.

Upgrading other Slurm components before the database can lead to data loss.

Procedure 1: Upgrading Slurm #

1. Upgrade slurmdbd Database Daemon

   Upon the first start of slurmdbd after a slurmdbd upgrade, it will convert its database. If the database is large, the conversion will take several 10s of minutes. During this time, the database is not accessible.

   We strongly recommend creating a backup of the database in case an error occurs during or after the upgrade process. Without a backup, all accounting data collected in the database might be lost in such an event. A database converted to a newer version cannot be converted back to an older one and older versions of slurmdbd will not recognize the newer formats. To back up and upgrade slurmdbd, follow this procedure:

   1. Stop the slurmdbd service:

      # rcslurmdbd stop

      Make sure that slurmdbd is not running anymore:

      # rcslurmdbd status

      It should be noted that slurmctld might remain running while the database daemon is down. While it is down, requests intended for slurmdbd are queued internally. The DBD Agent Queue size is limited, however, and should therefore be monitored with sdiag.

   2. Create a backup of the slurm_acct_db database:

      # mysqldump -p slurm_acct_db > slurm_acct_db.sql

      

      If needed, this can be restored by running:

      # mysql -p slurm_acct_db < slurm_acct_db.sql

   3. In preparation of the conversion, make sure the variable innodb_buffer_size is set to a value >= 128 Mb:

      On the database server, run:

      # echo  'SELECT @@innodb_buffer_pool_size/1024/1024;' | \
       mysql --password --batch

      If the size is less than 128 Mb, it can be changed on the fly for the current session (on mariadb):

      # echo 'set GLOBAL innodb_buffer_pool_size = 134217728;' | \
       mysql --password --batch

      Alternatively, the size can be changed permanently by editing /etc/my.cnf and setting it to 128 Mb. Then restart the database:

      # rcmysql restart

   4. Install the upgrade of slurmdbd:

      zypper install --force-resolution slurm_version-slurmdbd

      

      Note: Update MariaDB Separately

      If you also need to update mariadb, it is recommended to perform this step separately, before performing Step 1.d.

      1. Upgrade MariaDB:

         # zypper update mariadb

      2. Run the conversion of the database tables to the new version of MariaDB:

         mysql_upgrade --user=root --password=root_db_password;

   5. Rebuild database

      Because a conversion can take a considerable amount of time, the systemd service can run into a timeout during the conversion. Thus we recommend to perform the migration manually by running slurmdbd from the command line in the foreground:

      # /usr/sbin/slurmdbd -D -v

      When you see the below message, slurmdbd can be shut down:

      Conversion done:
      success!

      To do so, use signal SIGTERM (that is, press Ctrl–C).

      When using a backup slurmdbd, the conversion needs to be performed on the primary. The backup will not start until the conversion has taken place.

   6. Before restarting the service, remove or replace deprecated configuration options. For a list of deprecated options, see Section 6.2.26.2, “Important Slurm Configuration Changes”.

      When this has been completed, restart slurmdbd. During the database rebuild that slurmdbd performs upon the first start, it will not daemonize.

      

      Note: Convert Primary slurmbd First

      If a backup database daemon is used, the primary one needs to be converted first. The backup will not start until this has happened.

      Only after the conversion has been completed, the backup will start.

2. Update slurmctld and slurmd

   When the Slurm database has been updated, the slurmctld and slurmd instances can be updated. We recommend updating the head and compute nodes all in a single pass. If this is not feasible, the compute nodes (slurmd) can be updated on a node-by-node basis. However, this requires that the master nodes (slurmctld) have been updated successfully.

   1. Back up the slurmctld/slurmd configuration

      It is advisable to create a backup copy of the Slurm configuration before starting the upgrade process. Since the configuration file /etc/slurm/slurm.conf should be identical across the entire cluster, it is sufficient to do so on the master controller node.

   2. Increase Timeouts

      Set SlurmdTimeout and SlurmctldTimeout in /etc/slurm/slurm.conf to sufficiently high values to avoid timeouts while slurmctld and slurmd are down. We recommend at least 60 minutes, more on larger clusters.

      1. Edit /etc/slurm/slurm.conf on the master controller node and set the values for this variable to at least 3600 (1 hour).

         SlurmctldTimeout=3600
         SlurmdTimeout=3600

      2. Copy /etc/slurm/slurm.conf to all nodes. If MUNGE authentication is used in the cluster as recommended, use these steps:

         1. Obtain the list of partitions in /etc/slurm/slurm.conf.

         2. Execute:

            # cp /etc/slurm/slurm.conf /etc/slurm/slurm.conf.update
            # sudo -u slurm /bin/bash -c 'cat /etc/slurm/slurm.conf.update \
             | pdsh -R mrsh -P partitions \
             "cat > /etc/slurm/slurm.conf"'
            # rm /etc/slurm/slurm.conf.update
            # scontrol reconfigure

         3. Verify that the reconfiguration took effect:

            # scontrol show config | grep Timeout

   3. Shut down any running slurmctld instances

      1. If applicable, shut down any backup controllers on the backup head nodes:

         backup: # systemctl stop slurmctld

      2. Shut down the master controller:

         master: # systemctl stop slurmctld

   4. Back up the slurmctld state files

      Also, it should be noted, that slurmctld maintains state information that is persistent. Almost every major version involves changes to the slurmctld state files. This state information will be upgraded as well if the upgrade remains within the supported version range and no data will be lost.

      However, if a downgrade should become necessary, state information from newer versions will not be recognized by an older version of slurmctld and thus will be discarded, resulting in a loss of all running and pending jobs. Thus it is useful to back up the old state in case an update needs to be rolled back.

      1. Determine the StateSaveLocation directory:

         # scontrol show config | grep StateSaveLocation

      2. Create a backup of the content of this directory to be able to roll back the update if an issue arises.

         Should a downgrade be required, make sure to restore the content of the StateSaveLocation directory from this backup.

   5. Shut down slurmd on the nodes

      # pdsh -R ssh -P partitions systemctl stop slurmd

   6. Update slurmctld on the master and backup nodes as well as slurmd on the compute nodes

      1. On the master/backup node(s), run:

         master: # zypper install \
          --force-resolution slurm_version

      2. On the master node, run:

         master: # pdsh -R ssh -P partitions \
          zypper install --force-resolution \
          slurm_version-node

   7. Replace deprecated options

      If deprecated options need to be replaced in the configuration files (see the list in Section 6.2.26.2, “Important Slurm Configuration Changes”), this can be performed before updating the services. These configuration files can be distributed to all controllers and nodes of the cluster by using the method described in Step 2.b.ii.

      

      Note: Memory Size Seen by slurmd Can Change on Update

      Under certain circumstances, the amount of memory seen by slurmd can change after an update. If this happens, slurmctld will put the nodes in a drained state. To check whether the amount of memory seem by slurmd will change after the update, run the following on a single compute node:

      node1: # slurmd -C

      Compare the output with the settings in slurm.conf. If required, correct the setting.

   8. Restart slurmd on all compute nodes

      On the master controller, run:

      master: # pdsh -R ssh -P partitions \
       systemctl start slurmd

      On the master, run:

      master: # systemctl start slurmctld

      Then execute the same on the backup controller(s).

   9. Verify whether the system operates properly

      1. Check the status of the controller(s).

         On the master and backup controllers, run:

         # systemctl status slurmctld

      2. Verify that the services are running without errors using:

         sinfo -R

         You will see whether there are any down, drained, failing, or failed nodes after the restart.

   10. Clean up

       Restore the SlurmdTimeout and SlurmctldTimeout values in /etc/slurm/slurm.conf on all nodes and run scontrol reconfigure (see Step 2.b).

Each service pack of SUSE Linux Enterprise High-Performance Computing includes a new major version of libslurm. The old version will not be uninstalled on upgrade. User-provided applications will work, as long as the library version used for building is no more than two major versions behind. We strongly recommend rebuilding local applications using libslurm—such as MPI libraries with Slurm support—as early as possible. This can require updating user application if new arguments were introduced to existing functions.

The pam_slurm_adopt module allows restricting access to compute nodes to those users that have jobs running on them. It can also take care of run-away processes from user's jobs and end these processes when the job has finished.

pam_slurm_adopt works by binding the login process of a user and all its child processes to the cgroup of a running job.

It can be enabled with following steps:

The memkind library is a user-extensible heap manager built on top of jemalloc which enables control of memory characteristics and a partitioning of the heap between kinds of memory. The kinds of memory are defined by operating system memory policies that have been applied to virtual address ranges. Memory characteristics supported by memkind without user extension include control of NUMA and page size features.

For more information, see:

This tool is only available for x86-64.

MUNGE allows for secure communication between different machines which share the same secret key. The most common use case is the slurm workload manager which uses MUNGE for the encryption of its messages. Another use case is authentication for the parallel shell mrsh.

MUNGE uses the UID/GID to uniquely identify and authenticate users. This care must be taken that the users who are to be authenticated across a network have unified UIDs and GIDs.

MUNGE credentials have a limited time-to-live. Therefore it must be ensured that the time is synchronized across the entire cluster.

Install MUNGE using zypper in munge. This will install all packages required at runtime. The package munge-devel can be used to build applications that require MUNGE authentication.

When installing MUNGE, a new key is generated on every system. However, the entire cluster needs to use the same MUNGE key. Therefore the key from one system needs to be copied to all other nodes in the cluster in a secure way. Make sure that the key is only readable by the munge user (permissions mask 0400).

pdsh (with SSH) can be used to do this:

Check permissions, owner, and file type of the file located under /etc/munge/munge.key key on the local system:

# stat --format "%F %a %G %U %n" /etc/munge/munge.key

The settings should be:

400 regular file munge munge /etc/munge/munge.key

Calculate the MD5 sum of munge.key:

# md5sum /etc/munge/munge.key

Copy the key to the listed nodes:

# pdcp -R ssh -w HOSTLIST /etc/munge/munge.key \
  /etc/munge/munge.key

Check the key settings in the remote host:

# pdsh -R ssh -w HOSTLIST stat --format \"%F %a %G %U %n\" \
  /etc/munge/munge.key
# pdsh -R ssh -w HOSTLIST md5sum /etc/munge/munge.key

Make sure they match.

munged needs to be run on all nodes where MUNGE authentication is to take place. If MUNGE is used for authentication across the network, it needs to run on each side of the communication.

To start the service and make sure it is started after every reboot, on each node, run:

systemctl enable munge.service
systemctl start munge.service

To perform this on multiple nodes, you can also use pdsh.

mrsh is a set of remote shell programs using the MUNGE authentication system instead of reserved ports for security.

It can be used as a drop-in replacement for rsh and rlogin.

To install mrsh, do the following:

To set up a cluster of machines allowing remote login from each other, first follow the instructions for setting up and starting MUNGE authentication in Section 7.14, “MUNGE Authentication”. After MUNGE has been successfully started, enable and start mrlogin on each machine on which the user will log in:

systemctl enable mrlogind.socket mrshd.socket
systemctl start mrlogind.socket mrshd.socket

To start mrsh support at boot, run:

systemctl enable munge.service
systemctl enable mrlogin.service

We do not recommend using mrsh when logged in as the user root. This is disabled by default. To enable it anyway, run:

echo "mrsh" >> /etc/securetty
echo "mrlogin" >> /etc/securetty

Boost is a set of portable C++ libraries which provide a reference implementation of “existing practices”. See the full release notes for Boost 1.71 at https://www.boost.org/users/history/version_1_71_0.html.

To load the highest available serial version of this module, run:

module load TOOLCHAIN boost

To use the MPI-specific Boost libraries, add an argument for the MPI flavor to the command:

module load TOOLCHAIN MPI_FLAVOR boost

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

Most Boost libraries do not depend on MPI flavors. However, Boost contains a set of libraries to abstract interactions with MPI. These libraries depend on the MPI flavor used.

List of MPI master packages:

Replace MPI_FLAVOR with one of the MPI flavors described in Section 8, “MPI Libraries”.

FFTW is a C subroutine library for computing the Discrete Fourier Transform (DFT) in one or more dimensions, of both real and complex data, and of arbitrary input size.

This library is available as both a serial and an MPI-enabled variant. This module requires a compiler toolchain module loaded. To select an MPI variant, the respective MPI module needs to be loaded beforehand. To load this module, run:

module load TOOLCHAIN fftw3

or

module load TOOLCHAIN MPI_FLAVOR fftw3

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

Replace MPI_FLAVOR with one of the MPI flavors described in Section 8, “MPI Libraries”.

The GNU Scientific Library (GSL) is a numerical library for C and C++ programmers.

The library provides a wide range of mathematical routines such as random number generators, special functions and least-squares fitting. There are over 1000 functions in total with an extensive test suite.

It is free software under the GNU General Public License.

For this library, a compiler toolchain needs to be loaded beforehand. To load gsl, run:

module load TOOLCHAIN gsl

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”.

List of master packages:

HYPRE is a library of linear solvers which aim to solve large and detailed simulations faster than traditional methods at large scales.

The library offers a comprehensive suite of scalable solvers for large-scale scientific simulation, featuring parallel multigrid methods for both structured and unstructured grid problems. HYPRE is highly portable, and supports a number of languages. It is developed at the Lawrence Livermore National Laboratory.

For this library, a compiler toolchain and an MPI flavor needs to be loaded beforehand. To load this module, run:

module load TOOLCHAIN MPI_FLAVOR hypre

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

METIS is a set of serial programs for partitioning graphs, partitioning finite element meshes, and producing fill reducing orderings for sparse matrices. The algorithms implemented in METIS are based on the multilevel recursive-bisection, multilevel k-way, and multi-constraint partitioning schemes.

Experiments on a wide range of graphs have shown that METIS is one to two orders of magnitude faster than other widely used partitioning algorithms. Graphs with several millions vertices can be partitioned in 256 parts in a few seconds on current generation systems.

The fill-reducing orderings produced by METIS are significantly better than those produced by other widely used algorithms including multiple minimum degree. For many classes of problems arising in scientific computations and linear programming, METIS is able to reduce the storage and computational requirements of sparse matrix factorization, by up to an order of magnitude. Moreover, unlike multiple minimum degree, the elimination trees produced by METIS are suitable for parallel direct factorization. Furthermore, METIS is able to compute these orderings very fast. Matrices with millions of rows can be reordered in just a few seconds on current generation workstations and PCs.

For this library, a compiler toolchain needs to be loaded beforehand. To load METIS, run:

module load TOOLCHAIN metis

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”.

List of master packages:

MUMPS (MUltifrontal Massively Parallel sparse direct Solver) solves a sparse system of linear equations A x = b using Gaussian elimination.

This library requires a compiler toolchain and an MPI flavor to be loaded beforehand. To load this module, run:

module load TOOLCHAIN MPI_FLAVOR mumps

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

NumPy is a general-purpose array-processing package designed to efficiently manipulate large multi-dimensional arrays of arbitrary records without sacrificing too much speed for small multi-dimensional arrays.

NumPy is built on the Numeric code base and adds features introduced by numarray as well as an extended C API and the ability to create arrays of arbitrary type which also makes NumPy suitable for interfacing with general-purpose data-base applications.

There are also basic facilities for discrete Fourier transform, basic linear algebra, and random number generation.

This package is available both for Python 2 and 3. The specific compiler toolchain module must be loaded for this library. The correct library module for the Python version used needs to be specified when loading this module. To load this module, run:

module load TOOLCHAIN pythonPYTHON_VERSION-numpy

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”.

List of master packages:

The Open Community Runtime project is an application-building framework that explores methods for high-core-count programming with focus on HPC applications.

This first reference implementation is a functionally complete implementation of the OCR 1.0.0 specification, with extensive tools and demonstration examples, running on both single-node and clusters.

This library is available both without and with MPI support. For this library a compiler toolchain and if applicable, an MPI flavor needs to be loaded beforehand. To load ocr, run:

module load TOOLCHAIN ocr

or

module load TOOLCHAIN MPI_FLAVOR ocr

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

OpenBLAS is an optimized BLAS (Basic Linear Algebra Subprograms) library based on GotoBLAS2 1.3, BSD version. It provides the BLAS API. It is shipped as a package enabled for environment modules and thus requires using Lmod to select a version. There are two variants of this library, an OpenMP-enabled variant and a pthreads variant.

The OpenMP variant covers all use cases:

When linking statically, ensure that libgomp.a is included by adding the linker flag -lgomp.

The pthreads variant of the OpenBLAS library can improve the performance of single-threaded programs. The number of threads used can be controlled with the environment variable OPENBLAS_NUM_THREADS.

This module requires loading a compiler toolchain beforehand. To select the latest version of this module provided, run:

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”.

List of master package:

PETSc is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations.

This module requires loading a compiler toolchain as well as an MPI library flavor beforehand. To load this module, run:

module load TOOLCHAIN MPI_FLAVOR petsc

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 8, “MPI Libraries”.

The library ScaLAPACK (short for Scalable LAPACK) includes a subset of LAPACK routines designed for distributed memory MIMD-parallel computers.

This library requires loading both a compiler toolchain and an MPI library flavor beforehand. To load this module, run:

module load TOOLCHAIN MPI_FLAVOR scalapack

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 8, “MPI Libraries”.

SCOTCH is a set of programs and libraries which implement the static mapping and sparse matrix reordering algorithms developed within the SCOTCH project.

Its purpose is to apply graph theory, with a divide and conquer approach, to scientific computing problems such as graph and mesh partitioning, static mapping, and sparse matrix ordering, in application domains ranging from structural mechanics to operating systems or bio-chemistry.

For this library a compiler toolchain and an MPI flavor needs to be loaded beforehand. To load this module, run:

module load TOOLCHAIN MPI_FLAVOR scotch

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

SciPy is a collection of mathematical algorithms and convenience functions built on the NumPy extension of Python. It adds significant power to the interactive Python session by providing the user with high-level commands and classes for manipulating and visualizing data. With SciPy, an interactive Python session becomes a data-processing and system-prototyping environment.

The additional benefit of basing SciPy on Python is that this also makes a powerful programming language available for use in developing sophisticated programs and specialized applications. Scientific applications using SciPy benefit from the development of additional modules in numerous niches of the software landscape by developers across the world. Everything from parallel programming to web and data-base subroutines and classes have been made available to the Python programmer. All of this power is available in addition to the mathematical libraries in SciPy.

This package is available both for Python 2 (up to version 1.2.0 only) and 3. The specific compiler toolchain modules must be loaded for this library. The correct library module for the Python version used needs to be specified when loading this module. To load this module, run:

module load TOOLCHAIN pythonPYTHON_VERSION-scipy

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”.

List of master packages:

SuperLU is a general purpose library for the direct solution of large, sparse, nonsymmetric systems of linear equations. The library is written in C and can be called from C and Fortran programs.

It uses MPI and OpenMP to support various forms of parallelism. It supports both real and complex data types, both single and double precision, and 64-bit integer indexing. The library routines performs an LU decomposition with partial pivoting and triangular system solves through forward and back substitution. The LU factorization routines can handle non-square matrices but the triangular solves are performed only for square matrices. The matrix columns can be preordered (before factorization) either through library or user-supplied routines. This preordering for sparsity is completely separate from the factorization. Working precision iterative refinement subroutines are provided for improved backward stability. Routines are also provided to equilibrate the system, estimate the condition number, calculate the relative backward error, and estimate error bounds for the refined solutions.

This library requires a compiler toolchain and an MPI flavor to be loaded beforehand. To load this module, run:

module load TOOLCHAIN superlu

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”.

List of master packages:

The Trilinos Project is an effort to develop algorithms and enabling technologies within an object-oriented software framework for the solution of large-scale, complex multi-physics engineering and scientific problems. A unique design feature of Trilinos is its focus on packages.

This library needs a compiler toolchain and and MPI flavor to be loaded beforehand. To load this module, run:

module load TOOLCHAIN MPI_FLAVOR trilinos

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

The Adaptable IO System (ADIOS) provides a simple, flexible way for scientists to describe the data in their code that may need to be written, read, or processed outside of the running simulation. For more information, see https://www.olcf.ornl.gov/center-projects/adios/.

module load TOOLCHAIN MPI_FLAVOR adios

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

Replace MPI_FLAVOR with one of the MPI flavors described in Section 8, “MPI Libraries”.

HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of data types, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable and extensible, allowing applications to evolve in their use of HDF5.

There are serial and MPI variants of this library available. All flavors require loading a compiler toolchain module beforehand. The MPI variants also require loading the correct MPI flavor module.

To load the highest available serial version of this module, run:

module load TOOLCHAIN hdf5

When an MPI flavor is loaded, the MPI version of this module can be loaded by:

module load TOOLCHAIN MPI_FLAVOR phpdf5

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 8, “MPI Libraries”.

The NetCDF software libraries for C, C++, FORTRAN, and Perl are a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

The packages with names starting with netcdf provide C bindings for the NetCDF API. These are available with and without MPI support.

There are serial and MPI variants of this library available. All flavors require loading a compiler toolchain module beforehand. The MPI variants also require loading the correct MPI flavor module.

The MPI variant becomes available when the MPI module is loaded. Both variants require loading a compiler toolchain module beforehand. To load the highest version of the non-MPI netcdf module, run:

module load TOOLCHAIN MPI_FLAVOR netcdf

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 8, “MPI Libraries”.

netcdf-cxx4 provides a C++ binding for the NetCDF API.

This module requires loading a compiler toolchain module beforehand. To load this module, run:

module load TOOLCHAIN netcdf-cxx4

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

The netcdf-fortran packages provide FORTRAN bindings for the NetCDF API, with and without MPI support.

There are serial and MPI variants of this library available. All flavors require loading a compiler toolchain module beforehand. The MPI variants also require loading the correct MPI flavor module.

The MPI variant becomes available when the MPI module is loaded. Both variants require loading a compiler toolchain module beforehand. To load the highest version of the non-MPI netcdf-fortran module, run:

module load TOOLCHAIN netcdf-fortran

or

module load TOOLCHAIN MPI_FLAVOR netcdf-fortran

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

Parallel NetCDF (PnetCDF) is a library providing high-performance I/O while still maintaining file-format compatibility with NetCDF by Unidata.

The package is available for the MPI Flavors: Open MPI 2 and 3, MVAPICH2 and MPICH.

To load the highest available serial version of this module, run:

module load TOOLCHAIN MPI_FLAVOR pnetcdf

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 8, “MPI Libraries”.

Intel* MPI Benchmarks provides a set of elementary benchmarks that conform to MPI-1, MPI-2, and MPI-3 standard. One can run all of the supported benchmarks, or a subset specified in the command line using one executable file. Use command-line parameters to specify various settings, such as time measurement, message lengths, and selection of communicators. For details, see the Intel* MPI Benchmarks User's Guide located at: https://software.intel.com/en-us/imb-user-guide.

For the IMB binaries to be found, a compiler toolchain and an MPI flavor needs to be loaded beforehand. To load this, run:

module load TOOLCHAIN MPI_FLAVOR imb

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

PAPI (package papi) provides a tool with a consistent interface and methodology for use of the performance counter hardware found in most major microprocessors.

This package works with all compiler toolchains and does not require a compiler toolchain to be selected. The latest version provided can be loaded by running:

module load TOOLCHAIN papi

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”.

List of master packages:

For general information about Lmod and modules, see Section 7.7, “Lmod — Lua-based Environment Modules”.

mpiP is a lightweight profiling library for MPI applications. Because it only collects statistical information about MPI functions, mpiP generates considerably less overhead and much less data than tracing tools. All the information captured by mpiP is task-local. It only uses communication during report generation, typically at the end of the experiment, to merge results from all of the tasks into one output file.

For this library a compiler toolchain and and MPI flavor needs to be loaded beforehand. To load this, run:

module load TOOLCHAIN MPI_FLAVOR mpip

For information about the toolchain to load, see Section 7.5, “GNU Compiler Collection for HPC”. For information about available MPI flavors, see Section 8, “MPI Libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 8, “MPI Libraries”.