Slow boot on systems with lots of SCSI devices and dm-multipath

SUSE Linux Enterprise Server 15 SP5
SUSE Linux Enterprise Server 15 SP4
SUSE Linux Enterprise Server 15 SP3
SUSE Linux Enterprise Server 15 SP2
SUSE Linux Enterprise Server 15 SP1

On certain very large deployments in a SAN, with thousands of SCSI logical units forming multipath maps,
system boot may be very slow, and may in the worst case stall completely or fail with boot timeout.

For SLE-15-SP4, the following maintenance updates addressing this issue were released May/2023:

multipath-tools-0.9.0+117+suse.78cc20b-150400.4.13.1
udev-249.16-150400.8.28.3
systemd-249.16-150400.8.28.3


For SLE-15-SP5, the multipath-tools and udev containing the patch are already in the GM release.
The systemd package (systemd-249.16-150400.8.28.3) was released as maintenance update June/2023.


On older SLE releases, and as long as these updates can't be installed on SP4, the following workaround can be applied:

 - copy /usr/lib/udev/rules.d/58-scsi-sg3_symlink.rules to /etc/udev/rules.d
 - open /etc/udev/rules.d/58-scsi-sg3_symlink.rules in editor
 - comment out or delete the following lines:

    ENV{SCSI_IDENT_SERIAL}=="?*", ENV{DEVTYPE}=="disk", SYMLINK+="disk/by-id/scsi-S$env{SCSI_VENDOR}_$env{SCSI_MODEL}_$env{SCSI_IDENT_SERIAL}"
    ENV{SCSI_IDENT_SERIAL}=="?*", ENV{DEVTYPE}=="partition", SYMLINK+="disk/by-id/scsi-S$env{SCSI_VENDOR}_$env{SCSI_MODEL}_$env{SCSI_IDENT_SERIAL}-part%n"
    ENV{SCSI_IDENT_LUN_VENDOR}=="?*", ENV{DEVTYPE}=="disk", SYMLINK+="disk/by-id/scsi-0$env{SCSI_VENDOR}_$env{SCSI_MODEL}_$env{SCSI_IDENT_LUN_VENDOR}"

 - save the file

Note that this workaround assumes that no symlinks of the form `/dev/disk/by-id/scsi-S*` and `/dev/disk/by-id/scsi-0*` are used anywhere in the system to identify devices.

On large systems that exhibit this problem, these symlinks are not useful for device identification anyway, so this will most probably not be an issue.
If the system has multipath hardware but uses a non-multipath root disk, a generally recommended workaround is to create a blacklist entry in `/etc/multipath.conf` for the non-multipath root device:

    blacklist {
        wwid 3600508e000000000c540303d733f200c
    }

(replace the wwid value by the actual WWID of the root disk of the affected system).

See also the SLE Storage Administration Guide, section "multipath installation types"

During device discovery, udev creates device nodes and various symlinks for SCSI devices.
Most of these symlinks are found after boot under `/dev/disk/by-id`, and can be used to identify devices uniquely.
However, depending on the properties of the hardware, certain symlinks are not unique.
In particular, symlinks with the pattern `/dev/disk/by-id/scsi-S*` are assembled from the vendor, product, and serial number from the SCSI INQUIRY; for example, `/dev/disk/by-id/SFUJITSU_ETERNUS_DXL_280805`.
If a storage array reports the same serial number for all logical units, these symlink will be the same for every logical unit. It is thus not useful for identifying individual volumes.

When udev encounters a situation like this, it will see many devices all claiming this same symlink for themselves.
Udev needs to determine which device should own the symlink.
This is done by assigning a "link priority" value to the symlinks claimed by a device (see man udev(7)).

But all SCSI devices have the same priority; only multipath or RAID devices would have a higher priority. Finding the highest-priority devices among many contenders may take a long time.

An additional complication occurs if multipathd is is configured to use "find_multipaths greedy" (default on SUSE, see man multipath.conf(5)), but is configured only in the booted system, not in the initial RAM disk.

On such systems, all SCSI devices will initially be classified as multipath device, and will be forwarded to multipathd to create a multipath map from them.

But if the device has been treated as non-multipath in the initial RAM disk before and is mounted or otherwise in use, multipathd will fail to set up the map, and will re-trigger a uevent for the device in order to make sure it will then be correctly classified as non-multipath.
While this works well in principle, it can take a long time if udev is still busy working through a queue of uevents for lots of SCSI devices competing for symlinks.
In the worst case, this may cause a timeout mounting certain file systems during boot.

The problem is generic and can't be completely eliminated.


The steps described in the "Solution" section above tackle various aspects of the problem:

- The systemd update improves udev's symlink selection algorithm, speeding up symlink creation significantly in the scenario described above.
- The workaround that changes the udev rules files completely eliminates the ambigous `/dev/disk/by-id/scsi-S*` symlinks (and also `/dev/disk/by-id/scsi-0*`, which is another type of symlink that's often non-unique).
- The multipath-tools update avoids an already mounted disk to be mis-classified as a multipath member, and thus avoids the additional uevent as described above.
- Blacklisting non-multipath SCSI disks avoids misclassification of SCSI disks, and is generally recommenced best practice.