kABI breakage causing driver load failures (error -22)

This document (000021137) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 Service Pack 4

Situation

Drivers fail to load. Errors similar to one of the following examples may be observed:

e.g.

          [   22.284818] PM: Image not found (code -22)
          [   42.196496][  T647] mlx5_core 0000:27:00.0: probe_one:2139:(pid 647): mlx5_init_one failed with error code -22
          [   42.231501][ T1656] mlx5_core: probe of 0000:27:00.0 failed with error -22

e.g.

          [  184.993975][ T1401] bnxt_en 0000:86:00.0 (unnamed net_device) (uninitialized): devlink_params_register failed. rc=-22
          [  185.256768][ T1674] bnxt_en: probe of 0000:86:00.0 failed with error -22

Resolution

The kABI issue has been corrected in the released kernel version 5.14.21-150400.24.66.1 (June 2023)

Information appearing in kernel changelog:

    * Tue May 16 2023 tbogendoerfer@suse.de
    - Refresh
      patches.suse/net-ice-Add-support-for-enable_iwarp-and-enable_roce.patch.
    - Delete
      patches.suse/devlink-Add-enable_iwarp-generic-device-param.patch.
      Fixed broken kABI (bsc#1208050 bsc#1211414).
    - commit 118de8c

Cause

kABI issue was introduced in kernel 5.14.21-150400.24.11.1 (after initial GA kernel release) by the following backported patch:

   * Tue Jun 14 2022 tbogendoerfer@suse.de
   - net/ice: Remove unused enum (bsc#1200502).
   - net/ice: Fix boolean assignment (bsc#1200502).
   - net/ice: Add support for enable_iwarp and enable_roce devlink
     param (bsc#1200502).
   - devlink: Add 'enable_iwarp' generic device param (bsc#1200502).
   - commit 220523b

Additional Information

All kernels from 5.14.21-150400.24.11.1 (July 2022) to 5.14.21-150400.24.63.1 (May 2023) (inclusive) are affected.

The faulty backport broke the kABI in such a way that drivers built against the SLES15 SP4 GA kernel release and then used in conjunction with one of the kernel versions containing the faulty backport, may return -EINVAL ("Invalid Argument") from devlink_params_register() or devlink_param_verify() resulting in the adapters being non-operational.

The GA released kernel is not affected: 5.14.21-150400.22

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.