Release Notes #

The template slurmrestd.service unit file now defaults to listen on both the Unix socket and the slurmrestd port.

The template slurmrestd.service unit file now defaults to enable auth/jwt and the munge unit is no longer a dependency by default.

Add extra “EnvironmentFile=-/etc/default/$service” setting to service files.

Allow jobs to pack onto nodes already rebooting with the desired features.

Reset job start time after nodes are rebooted, previously only done for cloud/power save boots.

Node features (if any) are passed to RebootProgram if run from slurmctld.

Fail srun when using invalid --cpu-bind options (e.g. --cpu-bind=map_cpu:99 when only 10 CPUs are allocated).

Storing batch scripts and env vars are now in indexed tables using substantially less disk space. Those storing scripts in 21.08 will all be moved and indexed automatically.

Run MailProg through slurmscriptd instead of directly fork+exec()'ing from slurmctld.

Add acct_gather_interconnect/sysfs plugin.

Future and Cloud nodes are treated as "Planned Down" in usage reports.

Add new shard plugin for sharing GPUs but not with mps.

Add support for Lenovo SD650 V2 in acct_gather_energy/xcc plugin.

Remove cgroup_allowed_devices_file.conf, since the default policy in modern kernels is to whitelist by default. Denying specific devices must be done through gres.conf.

Node state flags (DRAIN, FAILED, POWERING UP, etc.) will be cleared now if node state is updated to FUTURE.

srun will no longer read in SLURM_CPUS_PER_TASK. This means you will implicitly have to specify --cpus-per-task on your srun calls, or set the new SRUN_CPUS_PER_TASK environment variable to accomplish the same thing.

Remove connect_timeout and timeout options from JobCompParams as there’s no longer a connectivity check happening in the jobcomp/elasticsearch plugin when setting the location off of JobCompLoc.

Add support for hourly reoccurring reservations.

Allow nodes to be dynamically added and removed from the system. Configure MaxNodeCount to accomodate nodes created with dynamic node registrations (slurmd -Z --conf="") and scontrol.

Added support for Cgroup Version 2.

sacct - allocations made by srun will now always display the allocation and step(s). Previously, the allocation and step were combined when possible.

cons_tres - change definition of the "least loaded node" (LLN) to the node with the greatest ratio of available CPUs to total CPUs.

Add support to ship Include configuration files with configless.

Provide a detailed reason in the job log as to why it has been terminated when hitting a resource limit.

Pass and use alias_list through credential instead of environment variable.

Add ability to get host addresses from nss_slurm.

Enable reverse fanout for cloud+alias_list jobs.

Add support to delete/update nodes by specifying nodesets or the 'ALL' keyword alongside the delete/update node message nodelist expression (i.e. scontrol delete/update NodeName=ALL or scontrol delete/update NodeName=ns1,nodes[1-3]).

Expanded the set of environment variables accessible through Prolog/Epilog and PrologSlurmctld/EpilogSlurmctld to include SLURM_JOB_COMMENT, SLURM_JOB_STDERR, SLURM_JOB_STDIN, SLURM_JOB_STDOUT, SLURM_JOB_PARTITION, SLURM_JOB_ACCOUNT, SLURM_JOB_RESERVATION, SLURM_JOB_CONSTRAINTS, SLURM_JOB_NUM_HOSTS, SLURM_JOB_CPUS_PER_NODE, SLURM_JOB_NTASKS, and SLURM_JOB_RESTART_COUNT.

Attempt to requeue jobs terminated by slurm.conf changes (node vanish, node socket/core change, etc). Processes may still be running on excised nodes. Admin should take precautions when removing nodes that have jobs on running on them.

Add switch/hpe_slingshot plugin.

Add new SchedulerParameters option bf_licenses to track licenses as within the backfill scheduler.

AcctGatherEnergyType rsmi is now gpu.

TaskAffinity parameter was removed from cgroup.conf.

Fatal if the mutually-exclusive JobAcctGatherParams options of UsePss and NoShared are both defined.

KeepAliveTime has been moved into CommunicationParameters. The standalone option will be removed in a future version.

preempt/qos - add support for WITHIN mode to allow for preemption between jobs within the same QOS.

Fatal error if CgroupReleaseAgentDir is configured in cgroup.conf. The option has long been obsolete.

Fatal if more than one burst buffer plugin is configured.

Added keepaliveinterval and keepaliveprobes to CommunicationParameters.

Added new max_token_lifespan=<seconds> to AuthAltParameters to allow sites to restrict the lifespan of any requested ticket by an unprivileged user.

Disallow slurm.conf node configurations with NodeName=ALL.

Remove support for (non-functional) --cpu-bind=boards.

Added --prefer option at job submission to allow for 'soft' constraints.

Add condflags=open to sacctmgr show events to return open/currently down events.

sacct -f flag implies -c flag.

srun --overlap now allows the step to share all resources (CPUs, memory, and GRES), where previously --overlap only allowed the step to share CPUs with other steps.

openapi/v0.0.35 - Plugin has been removed.

burst_buffer plugins - err_msg added to bb_p_job_validate().

openapi - added flags to slurm_openapi_p_get_specification(). Existing plugins only need to update their prototype for the function as manipulating the flags pointer is optional.

openapi - Added OAS_FLAG_MANGLE_OPID to allow plugins to request that the operationId of path methods be mangled with the full path to ensure uniqueness.

openapi/[db]v0.0.36 - Plugins have been marked as deprecated and will be removed in the next major release.

switch plugins - add switch_g_job_complete() function.

Removed gres/mic plugin used to support Xeon Phi coprocessors.

Add LimitFactor to the QOS. A float that is factored into an associations GrpTRES limits. For example, if the LimitFactor is 2, then an association with a GrpTRES of 30 CPUs, would be allowed to allocate 60 CPUs when running under this QOS.

A job’s next_step_id counter now resets to 0 after being requeued. Previously, the step id’s would continue from the job’s last run.

API change: Removed slurm_kill_job_msg and modified the function signature for slurm_kill_job2. slurm_kill_job2 should be used instead of slurm_kill_job_msg.

AccountingStoreFlags=job_script allows you to store the job’s batch script.

AccountingStoreFlags=job_env allows you to store the job’s env vars.

Removed sched/hold plugin.

cli_filter/lua, jobcomp/lua, job_submit/lua now load their scripts from the same directory as the slurm.conf file (and thus now will respect changes to the SLURM_CONF environment variable).

SPANK - call slurm_spank_init if defined without slurm_spank_slurmd_exit in slurmd context.

Add new PLANNED state to a node to represent when the backfill scheduler has it planned to be used in the future instead of showing as IDLE. sreport also has changed it’s cluster utilization report column name from 'Reserved' to 'Planned' to match this nomenclature.

Put node into INVAL state upon registering with an invalid node configuration. Node must register with a valid configuration to continue.

Remove SLURM_DIST_LLLP environment variable in favor of just SLURM_DISTRIBUTION.

Make --cpu-bind=threads default for --threads-per-core — can be overridden by the CLI or an environment variable.

slurmd - allow multiple comma-separated controllers to be specified in configless mode with --conf-server

Manually powering down of nodes with scontrol now ignores SuspendExc<Nodes|Parts>.

Distinguish queued reboot requests (REBOOT@) from issued reboots (REBOOT^).

auth/jwt - add support for RS256 tokens. Also permit the username in the 'username' field in addition to the 'sun' (Slurm UserName) field.

service files - change dependency to network-online rather than just network to ensure DNS and other services are available.

Add "Extra" field to node to store extra information other than a comment.

Add ResumeTimeout, SuspendTimeout and SuspendTime to Partitions.

The memory.force_empty parameters is no longer set by jobacct_gather/cgroup when deleting the cgroup`. This previously caused a significant delay (~2s) when terminating a job, and is not believed to have provided any perceivable benefit. However, this may lead to slightly higher reported kernel mem page cache usage since the kernel cgroup memory is no longer freed immediately.

TaskPluginParam=verbose is now treated as a default. Previously it would be applied regardless of the job specifying a --cpu-bind.

Add node_reg_mem_percent SlurmctldParameter to define percentage of memory nodes are allowed to register with.

Define and separate node power state transitions. Previously a powering down node was in both states, POWERING_OFF and POWERED_OFF. These are now separated. e.g. IDLE+POWERED_OFF (IDLE~) → IDLE+POWERING_UP (IDLE#) - Manual power up or allocation → IDLE → IDLE+POWER_DOWN (IDLE!) - Node waiting for power down → IDLE+POWERING_DOWN (IDLE%) - Node powering down → IDLE+POWERED_OFF (IDLE~) - Powered off

Some node state flag names have changed. These would be noticeable for example if using a state flag to filter nodes with sinfo. e.g. POWER_UP → POWERING_UP POWER_DOWN → POWERED_DOWN POWER_DOWN now represents a node pending power down

Create a new process called slurmscriptd which runs PrologSlurmctld and EpilogSlurmctld. This avoids fork() calls from slurmctld, and can avoid performance issues if the slurmctld has a large memory footprint.

Pass JSON of job to node mappings to ResumeProgram.

QOS accrue limits only apply to the job QOS, not partition QOS.

Any return code from SPANK plugin or SPANK function that is not SLURM_SUCCESS (zero) will be considered to be an error. Previously, only negative return codes were considered an error.

Add support for automatically detecting and broadcasting executable shared object dependencies for sbcast and srun --bcast.

All SPANK error codes now start at 3000. Where previously SPANK would give a return code of 1, it will now return 3000. This change will break ABI compatibility with SPANK plugins compiled against older version of Slurm.

SPANK plugins are now required to match the current Slurm release, and must be recompiled for each new Slurm major release. (They do not need to be recompiled when upgrading between maintenance releases.)

SLURM_NODE_ALIASES now has brackets around the node’s address to be able to distinguish IPv6 addresses. e.g. <node_name>:[<node_addr>]:<node_hostname>

The job_container/tmpfs plugin now requires PrologFlags=contain to be set in slurm.conf.

Limit max_script_size to 512 MB.

Errors detected in the parser handlers due to invalid configurations are now propagated and can lead to fatal (and thus exit) the calling process.

Enforce a valid configuration for AccountingStorageEnforce in slurm.conf. If the configuration is invalid, then an error message will be printed and the command or daemon (including slurmctld) will not run.

Removed AccountingStoreJobComment option. Please update your config to use AccountingStoreFlags=job_comment instead.

Removed DefaultStorage{Host,Loc,Pass,Port,Type,User} options.

Removed CacheGroups, CheckpointType, JobCheckpointDir, MemLimitEnforce, SchedulerPort, SchedulerRootFilter options.

Added Script to DebugFlags for debugging slurmscriptd (the process that runs slurmctld scripts such as PrologSlurmctld and EpilogSlurmctld).

Rename SbcastParameters to BcastParameters.

systemd service files - add new “-s” option to each daemon which will change the working directory even with the -D option. (Ensures any core files are placed in an accessible location, rather than /.)

Added BcastParameters=send_libs and BcastExclude options.

Remove the (incomplete) burst_buffer/generic plugin.

Make SelectTypeParameters=CR_Core_Memory default for cons_tres and cons_res.

Remove support for TaskAffinity=yes in cgroup.conf. Adding task/affinity to TaskPlugins in slurm.conf is strongly recommended instead.

Changed the --format handling for negative field widths (left justified) to apply to the column headers as well as the printed fields.

Invalidate multiple partition requests when using partition based associations.

scrontab - create the temporary file under the TMPDIR environment variable (if set), otherwise continue to use TmpFS as configured in slurm.conf.

sbcast / srun --bcast - removed support for zlib compression. lz4 is vastly superior in performance, and (counter-intuitively) zlib could provide worse performance than no compression at all on many systems.

sacctmgr - changed column headings to ParentID and ParentName instead of Par ID and "Par Name` respectively.

SALLOC_THREADS_PER_CORE and SBATCH_THREADS_PER_CORE have been added as input environment variables for salloc and sbatch, respectively. They do the same thing as --threads-per-core.

Don’t display node’s comment with scontrol show nodes unless set.

Added SLURM_GPUS_ON_NODE environment variable within each job/step.

sreport - change to sorting TopUsage by the --tres option.

slurmrestd - do not run allow operation as SlurmUser/root by default.

scontrol show node now shows State as base_state+flags instead of shortened state with flags appended. eg. IDLE# → IDLE+POWERING_UP. Also POWER state flag string is POWERED_DOWN.

scrontab - add ability to update crontab from a file or standard input.

scrontab - added ability to set and expand variables.

Make srun sensitive to BcastParameters.

Added sbcast/srun --send-libs, sbcast --exclude and srun --bcast-exclude.

Changed ReqMem field in sacct to match memory from ReqTRES. It now shows the requested memory of the whole job with a letter appended indicating units (M for megabytes, G for gigabytes, etc.). ReqMem is only displayed for the job, since the step does not have requested TRES. Previously ReqMem was also displayed for the step but was just displaying ReqMem for the job.

jobcomp plugin: change plugin API to jobcomp_p_*().

sched plugin: change plugin API to sched_p_*() and remove slurm_sched_p_initial_priority() call.

step_ctx code has been removed from the api.

slurm_stepd_get_info()/stepd_get_info() has been removed from the api.

The v0.0.35 OpenAPI plugin has now been marked as deprecated. Please convert your requests to the v0.0.37 OpenAPI plugin.

Log messages enabled by the various DebugFlags have been overhauled, and will all print at the verbose() level, and prepend the flag name that is associated with a given log message

accounting_storage/filetxt has been removed as an option. Consider using accounting_storage/slurmdbd as an alternative.

Setting of number of Sockets per node was standardized for configuration line with and without Boards=. Specifically in case of Boards=1 and #CPUs given the default value of Sockets will be set to #CPUs/#Cores/#Threads

Dynamic Future Nodes: slurmds started with -F[<feature>] will be associated with a node name in Slurm that matches the same hardware configuration

SlurmctldParameters=cloud_reg_addrsa: Cloud nodes automatically get NodeAddr and NodeHostname set from slurmd registration

SlurmctldParameters=power_save[_min]_interval: Configure how often the power save module looks to do work`

By default, a step started with srun will be granted exclusive (or non-overlapping) access to the resources assigned to that step. No other parallel step will be allowed to run on the same resources at the same time. This replaces one facet of the --exclusive option’s behavior, but does not imply the --exact option described below. To get the previous default behavior, which allowed parallel steps to share all resources, use the new srun --overlap option.

In conjunction to this non-overlapping step allocation behavior being the new default, there is an additional new option for step management --exact, which will allow a step access to only those resources requested by the step. This is the second half of the --exclusive behavior. Otherwise, by default all non-gres resources on each node in the allocation will be used by the step, making it so no other parallel step will have access to those resources unless --overlap is specified for both steps.

The option --threads-per-core now influences task layout/binding, not just allocation.

AutoDetect in gres.conf can now be specified for some nodes while not for others via the NodeName option.

gres.conf: Add new MultipleFiles configuration entry to allow a single GRES to manage multiple device files simultaneously.

The option SallocDefaultCommand has been removed.

Support for an "Interactive Step" has been added, designed to be used with salloc to launch a terminal on an allocated compute node automatically. Enable this by setting use_interactive_step as part of LaunchParameters.

IPv6 support has been added. This must be explicitly enabled with EnableIPv6 in CommunicationParameters. IPv4 support can be disabled with DisableIPv4.

Allow use of a target directory with srun --bcast, and change the default file name to include the node name as well.

The new option --mail-type=INVALID_DEPEND has been added to salloc, sbatch, and srun.

Differences between hardware (memory size, number of CPUs) discovered on node vs configured in slurm.conf will now throw an error only when the node state is set to drain. Previously it was done on every node registration, those messages were demoted to debug level.

The command scrontab has been added. It permits crontab-compatible job scripts to be defined. These scripts will recur automatically (at most) on the intervals described.

Enable the -lnodes=:gpus= in #PBS/qsub -l nodes syntax.

Any user with AdminLevel equal or higher than Operator can see any hidden partition by default, as SlurmUser or root already did.

select/linear will now allocate up to nodes RealMemory as spedified in slurm.conf when configured with SelectTypeParameters=CR_Memory and --mem=0 specified. Previous behavior was no memory accouted and no memory limits implied to job.

slurmrestd, an API to interface with slurmdbd.

The option --ntasks-per-gpu has been added to sbatch and srun.

The --gpu-bind=single option has been added to sbatch and srun.

Fix: scontrol takeover [backup] hangs when specifying a backup > 1 have been fixed. All slurmctlds below the "backup" will be shutdown.

sacct: get the UID from database instead of from the user name and a system call. Add --use-local-uid option to sacct to use the previous behavior.

sbatch: the %s format in -e/-i/-o options will expand to batch rather than 4294967294.

squeue: added pendingtime as a option for --Format.

sacct: AllocGres and ReqGres were removed. Alloc/ReqTres should be used instead.

scontrol: added the "Reserved" license count to scontrol show licenses.

squeue: put sorted start times of N/A or 0 at the end of the list.

scontrol: Change scontrol reboot ASAP to use next_state=resume logic.

scontrol: added an admin-settable "Comment" field to each Node.

squeue and sinfo: -O no longer repeat the last suffix specified.

salloc: wait for PrologSlurmctld to finish before entering the shell.

Add time specification: now-X (that is, subtract X from the present)

slurm_ctl_conf_t has been renamed to slurm_conf_t.

slurm_free_kvs_comm_set() has been renamed to slurm_pmi_free_kvs_comm_set(), slurm_get_kvs_comm_set() has been renamed to slurm_pmi_get_kvs_comm_set().

slurm_job_step_layout_get() parameters has changed to use slurm_step_id_t see slurm.h for new implementation. If not running hetsteps just put NO_VAL as the value for step_het_comp.

slurm_job_step_stat() parameters has changed to use slurm_step_id_t see slurm.h for new implementation. If not running hetsteps just put NO_VAL as the value for step_het_comp.

slurm_job_step_get_pids() parameters has changed to use slurm_step_id_t see slurm.h for new implementation. If not running hetsteps just put NO_VAL as the value for step_het_comp.

slurm_job_step_get_pids() parameters has changed to use slurm_step_id_t see slurm.h for new implementation. If you are not running hetsteps, use NO_VAL as the value for step_het_comp.

slurmdb_selected_step_t has been renamed slurm_selected_step_t.

slurm_sbcast_lookup() arguments have changed. It now takes a populated slurm_selected_step_t instead of job_id, het_job_offset, step_id.

Due to internal restructuring ahead of the 20.11 release, applications calling libslurm must call slurm_init(NULL) before any API calls. Otherwise the API call is likely to fail due to libslurm’s internal configuration not being available.

buildcache create: make file exists less verbose

spack mirror create: don’t change paths to urls

Improve error message for requirements

uninstall: fix accidental cubic complexity

scons: fix signature for install_args

Fix combine_phase_logs text encoding issues

Use a module-like object to propagate changes in the MRO, when setting build env

PackageBase should not define builder legacy attributes

Forward lookup of the run_tests attribute

Bugfix for timers

Fix path handling in prefix inspections

Fix libtool filter for Fujitsu compilers

FileCache: delete the new cache file on exception

Propagate exceptions from Spack python console

Tests: Fix a bug/typo in a config_values.py fixture

Various CI fixes

Docs: remove monitors and analyzers, typos

bump release version for tutorial command

spack spec prints dependencies more legibly. Dependencies in the output now appear at the earliest level of indentation possible

You can override package.py attributes like url, directly in packages.yaml ( docs)

There are a number of new architecture-related format strings you can use in Spack configuration files to specify paths ( docs)

Major performance improvements for installation from binary caches

Test suite can now be parallelized using xdist (used in GitHub Actions)

Reduce lock contention for parallel builds in environments

We now build nearly all of E4S with oneapi in our buildcache

Added 3 new machine learning-centric stacks to binary cache: x86_64_v3, CUDA, ROCm

Support for Python 3.5 is dropped . Only Python 2.7 and 3.6+ are officially supported.

This is the last Spack release that will support Python 2 . Spack v0.19 will emit a deprecation warning if you run it with Python 2, and Python 2 support will soon be removed from the develop branch.

LD_LIBRARY_PATH is no longer set by default by spack load or module loads.

Setting LD_LIBRARY_PATH in Spack environments/modules can cause binaries from outside of Spack to crash, and Spack’s own builds use RPATH and do not need LD_LIBRARY_PATH set in order to run. If you still want the old behavior, you can run these commands to configure Spack to set LD_LIBRARY_PATH:

spack config add modules:prefix_inspections:lib64:[LD_LIBRARY_PATH]
spack config add modules:prefix_inspections:lib:[LD_LIBRARY_PATH]

The spack:concretization:[together|separately] has been removed after being deprecated in v0.18. Use concretizer:unify:[true|false].

config:module_roots is no longer supported after being deprecated in v0.18. Use configuration in module sets instead ( docs).

spack activate and spack deactivate are no longer supported, having been deprecated in v0.18. Use an environment with a view instead of activating/deactivating (docs).

The old YAML format for buildcaches is now deprecated . If you are using an old buildcache with YAML metadata you will need to regenerate it with JSON metadata.

spack bootstrap trust and spack bootstrap untrust are deprecated in favor of spack bootstrap enable and spack bootstrap disable and will be removed in v0.20.

The graviton2 architecture has been renamed to neoverse_n1, and graviton3 is now neoverse_v1. Buildcaches using the old architecture names will need to be rebuilt.

The terms blacklist and whitelist have been replaced with include and exclude in all configuration files . You can use spack config update to automatically fix your configuration files.

Permission setting on installation now handles effective uid properly

buildable:true for an MPI implementation now overrides buildable:false for mpi

Improved error messages when attempting to use an unconfigured compiler

Do not punish explicitly requested compiler mismatches in the solver

spack stage: add missing –fresh and –reuse

Fixes for adding build system executables like cmake to package scope

Bugfix for binary relocation with aliased strings produced by newer binutils

Fix several bugs related to bootstrapping

Fix a regression that was causing spec hashes to differ between Python 2 and Python 3

Fixed compiler flags for oneAPI and DPC++

Fixed several issues related to concretization

Improved support for Cray manifest file and spack external find

Assign a version to openSUSE Tumbleweed according to the GLIBC version in the system

Improved Dockerfile generation for spack containerize

Fixed a few bugs related to concurrent execution of commands

WarpX: add v22.06, fixed libs property

openPMD: add v0.14.5, update recipe for @develop

Environment views can optionally link only run dependencies with link:run

spack external find --all finds library-only packages in addition to build dependencies

Customizable config:license_dir option

spack external find --path PATH takes a custom search path

spack spec has a new --format argument like spack find

spack concretize --quiet skips printing concretized specs

spack info now has cleaner output and displays test info

Package-level submodule option for git commit versions

Using /hash syntax to refer to concrete specs in an environment now works even if /hash is not installed.

full hash (see above)

new develop versioning scheme 0.19.0-dev0

Allow for multiple dependencies/dependents from the same package

Splice differing virtual packages

Concretization of large environments with unify: when_possible is much faster than concretizing separately (see above)

Single-pass view generation algorithm is 2.6x faster

oneapi and dpcpp flag support

better support for M1 and a64fx

Spack no longer supports Python 2.6

Removed deprecated --run-tests option of spack install; use spack test

Removed deprecated spack flake8; use spack style

Deprecate spack:concretization config option; use concretizer:unify

Deprecate top-level module configuration; use module sets

spack activate and spack deactivate are deprecated in favor of environments; will be removed in 0.19.0

Fix bug that broke locks with many parallel builds

Many bugfixes and consistency improvements for the new concretizer and --reuse

CMakePackage uses CMAKE_INSTALL_RPATH_USE_LINK_PATH

Refactored lua support: lua-lang virtual supports both lua and luajit via new LuaPackage build system

PythonPackage: now installs packages with pip

Python: improve site_packages_dir handling

Extends: support spec, not just package name

Use stable URLs and ?full_index=1 for all github patches