Configure multiple iSCSI ifaces and use multipathing
Objective: To provide a quick overview of the steps needed to bind different NICS to iSCSI ifaces and use those ifaces to setup multipathing.
SLES 11 SP3 server with 5 NICS (1 on public network, 3 on private network dedicated to iscsi).
eth0 – 22.214.171.124
eth1 – 10.1.10.1
eth2 – 10.1.10.2
eth3 – 10.1.10.3
iscsi-target – 10.1.10.100 (iqn.my-target.com)
- Install open-iscsi
zypper install open-iscsi
- Run “yast iscsi-client”, do nothing but let yast configure the initiator name.
Yast has specific parameters regarding the syntax of the initiator name. Rather than manually configuring the initiator name and run into problems with Yast later, let Yast set the initiator name up front and save time later on.
- Check the network setup and verify that all IP addresses and subnet masks are correct.
- Check network connectivity by pinging from the different interfaces.
syntax: ping -I ethX IP.of.target
example: ping -I eth1 10.1.1.100
- Set /proc/sys/net/ipv4/conf/rp_filter to 0
The rp_filter setting impacts packet filtering/routing and can cause problems with iSCSI connections on a multi-homed system.
SETUP ISCSI IFACE FILES
- Configure iface devices
syntax: iscsiadm -m iface -I iface# –op=new (dash dash op=new)
example: iscsiadm -m iface -I iface1 –op=new (dash dash op=new)
Run this command to create an iface file for each desired interface. As of today, 08/22/2014, Yast cannot create iface files thus this step must be done manually.
- Modify iface file to uniquely identify one NIC. There are several different parameters that can be used (iface.hwaddress, iface.net_ifacename, iface.ipaddress, etc) see the README for a more complete list of parameters.
syntax: iscsiadm -m iface -I iface# –op=update -n iface.hwaddress -v MAC:of:NIC (dash dash op=update)
example: iscsiadm -m iface -I iface1 –op=update -n iface.hwaddress -v 00:0c:29:d9:6f:75 (dash dash op=update)
A word of caution: use the minimum number of parameters to correlate the iface and the NIC. Adding more parameters increases the likely-hood of typos or other mistakes that can cause iscsi not to work as expected.
- Check the contents of the iface files to ensure accuracy. Either cat the file or use iscsiadm to view the files
syntax: cat /etc/iscsi/ifaces/iface#
example: cat /etc/iscsi/ifaces/iface1
syntax: iscsiadm -m iface -I iface#
example: iscsiadm -m iface -I iface1
- Set the initiator name.
If not done earlier set the initiator name. As noted above Yast has some specific syntax rule for the initiator names. Run “yast iscsi-client” and let Yast set the initiator name and then exit before doing any discovery.
- The easiest method to have iscsid scan the target device and then use the /etc/iscsi/ifaces files create records for each target and iface combination is:
syntax: iscsiadm -m discoverydb -t st -p IP.of.target.nic:3260 –discover (dash dash discover)
example: iscsiadm -m discoverydb -t st -p 10.1.10.100:3260 –discover (dash dash discover)
It is also possible to just discover a target on one iface.
syntax: iscsiadm -m discovery -t st -p IP.of.target.nic:3260 –interface=iface# –discover (dash dash discover)
example: iscsiadm -m discovery -t st -p 10.1.10.100:3260 –interface=iface1 –discover (dash dash discover)
Once iscsid has built the send_targets database a login must be initiated to access the target device. Yast or the cli can be used to manage the iscsi connections and login to each node.
With the iface files created in /etc/iscsi/ifaces and the target discovered YaST will read the send_targets database and display each connection between the iface and the target and will list them in the Discovered Targets tab. From there log-in options are available via the log-in button. If authentication is needed YaST may be an easier option over the cli.
With the targets discovered for the desired ifaces the initiator now needs to log into the target.
syntax: iscsiadm -m node -l
example: iscsiadm -m node -l
This will log the initiator into all of the nodes that are in the database. Please note this example assumes no authentication is needed. If authentication is needed use YaST or consult the documentation. More granular options, such as logging into only one node, can be found in the documentation.
There are many settings relating to each iscsi node. The startup method and authentication methods to name a few. Several settings are available in YaST or manually manipulating the file is an option; however, misconfiguration of the node settings may cause problems with log-in and operation.
Once everything is configured as desired verify that chkconfig open-iscsi is set to on.
- Start multipathing
- Check mpio paths
If only one path is listed under multipath -ll double check the value of /proc/sys/net/ipv4/conf/all/rp_filter.
For help with multipath configuration please see http://support.novell.com/techcenter/sdb/en/2005/04/sles_multipathing.html
The README for iscsi can be found at /usr/share/doc/packages/open-iscsi/README
You’ll need to make a few additional modifications or you’re going to hit some issue
1. You’ll want to set the MTU=9000 to enable jumbo frame for your iscsi NIC’s
2. You need to disable the FW_KERNEL_SECURITY firewall setting or it will override rp_filter. It’s easier to reboot after making this change since it modifies lots of kernel setting.
sysconf_addword /etc/sysconfig/SuSEfirewall2 FW_KERNEL_SECURITY no
3. Preventing ARP Flux. Without this only a single interface will be able to communicate with the SAN due to arp flux. This needs to be done in addition to setting net.ipv4.conf.all.rp_filter=0.
I have two iscsi interfaces and my system was only using one interface. iscsi now uses both interfaces and using multipath after i followed deadpoint’s recommendations under step 3.
AUTO LOGIN AFTER REBOOT?
Thanks to this article and the comment by deadpoint about ARP flux I was able to get this to work on my system.
HOWEVER, after the client reboots the iscsi sessions are gone. I then have to manually login again to the iscsi volume(s) after the system comes back up.
Is there a way to have open-iscsi auto login during startup?