node.session.timeo.replacement_timeout/recovery_tmo always changes from 120 to 5 after a while

This document (000021146) is provided subject to the disclaimer at the end of this document.

Environment

SUSE linux Enterprise Server all Service Packs

Situation

iSCSI option node.session.timeo.replacement_timeout is set to default value 120:

n1:~ # iscsiadm -m node --targetname iqn.shared_storage:12sp5 -o show | grep node.session.timeo.replacement_timeout
node.session.timeo.replacement_timeout = 120
node.session.timeo.replacement_timeout = 120
node.session.timeo.replacement_timeout = 120

yet, during target outage, kernel reports I/O errors to offline device after 5 seconds:

Jul 21 12:42:36 n1 kernel:  session2: session recovery timed out after 5 secs
Jul 21 12:42:37 n1 kernel:  session1: session recovery timed out after 5 secs
Jul 21 12:42:37 n1 kernel: sd 7:0:0:0: rejecting I/O to offline device
Jul 21 12:42:37 n1 kernel: sd 7:0:0:0: [sdb] killing request
Jul 21 12:42:37 n1 kernel: sd 7:0:0:0: [sdb] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jul 21 12:42:37 n1 kernel: sd 7:0:0:0: [sdb] CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
Jul 21 12:42:37 n1 kernel: print_req_error: I/O error, dev sdb, sector 0
Jul 21 12:42:37 n1 kernel: sd 7:0:0:0: rejecting I/O to offline device

Resolution

If /etc/multipath.conf doesn't exist, create it and add following content:

devices {       
        device {
        vendor IET
        product VIRTUAL-DISK
        fast_io_fail_tmo 120
}

replace vendor and product to appropriate values for your storage.
The values can be obtained from lsscsi command:

n1:~ # lsscsi
[7:0:0:0]    disk    IET      VIRTUAL-DISK     0     /dev/sda 
[7:0:0:1]    disk    IET      VIRTUAL-DISK     0     /dev/sdb 
[7:0:0:2]    disk    IET      VIRTUAL-DISK     0     /dev/sdd 
[8:0:0:0]    disk    IET      VIRTUAL-DISK     0     /dev/sdc

Apply the new settings with multipath command, for example: multipath -d

Cause

multipath command resets the default iSCSI timeout from 120 to 5:

n1:~ # cat /sys/class/iscsi_session/session*/recovery_tmo
120
120

n1:~ # multipath -v 3 -d

n1:~ # cat /sys/class/iscsi_session/session*/recovery_tmo
5
5

Even though multipath is not used, the command multipath -v3 -d is executed by supportconfig, which support usually requires for analysis.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.