Holy Supportconfig File – Analysis: Part 1

Let’s talk about Supportutils tool again? Sure thing, why not!

If you have read my previous post about Supportutils here, you know by now how important this tool is for our work as TSEs (Technical Support Engineer). Today, i’d like to show you some of the important information that this little amazing tool can gather for us when we troubleshoot our customers/partners’ problems.

The Holy Supportconfig File gathers all these nice information about the system and put them in text files:

mlimardo@nts_hostname_170405_1159>ll
total 4488912
-rw------- 1 mlimardo users 2659 Apr 5 11:59 basic-environment.txt
-rw------- 1 mlimardo users 176817 Apr 5 11:59 basic-health-check.txt
-rw------- 1 mlimardo users 186789888 Apr 6 12:06 boot.txt
-rw------- 1 mlimardo users 172 Apr 5 14:13 cimom.txt
-rw------- 1 mlimardo users 5661 Apr 5 12:00 crash.txt
-rw------- 1 mlimardo users 24610 Apr 5 14:14 cron.txt
-rw------- 1 mlimardo users 185 Apr 5 14:14 dhcp.txt
-rw------- 1 mlimardo users 277 Apr 5 14:14 dns.txt
-rw------- 1 mlimardo users 83 Apr 5 14:14 docker.txt
-rw------- 1 mlimardo users 443575 Apr 5 14:13 env.txt
-rw------- 1 mlimardo users 807676 Apr 5 14:13 etc.txt
-rw------- 1 mlimardo users 13865 Apr 5 14:14 fs-autofs.txt
-rw------- 1 mlimardo users 22902 Apr 5 12:00 fs-btrfs.txt
-rw------- 1 mlimardo users 327149 Apr 5 12:00 fs-diskio.txt
-rw------- 1 mlimardo users 14425 Apr 5 14:14 fs-iscsi.txt
-rw------- 1 mlimardo users 553 Apr 5 14:14 fs-softraid.txt
-rw------- 1 mlimardo users 3734881 Apr 5 14:15 hardware.txt
-rw------- 1 mlimardo users 86 Apr 5 14:13 ha.txt
-rw------- 1 mlimardo users 81 Apr 5 14:14 ib.txt
-rw------- 1 mlimardo users 165 Apr 5 14:14 kvm.txt
-rw------- 1 mlimardo users 1125 Apr 5 14:13 ldap.txt
-rw------- 1 mlimardo users 4533459 Apr 5 14:14 lvm.txt
-rw------- 1 mlimardo users 182 Apr 5 14:14 lxc.txt
-rw------- 1 mlimardo users 7265328 Apr 5 12:00 memory.txt
-rw------- 1 mlimardo users 38546320 Apr 5 14:15 messages-20170223.txt
-rw------- 1 mlimardo users 4645746 Apr 5 14:15 messages-20170224.txt
-rw------- 1 mlimardo users 4859734 Apr 5 14:15 messages-20170308.txt
-rw------- 1 mlimardo users 246125003 Apr 5 14:15 messages-20170313.txt
-rw------- 1 mlimardo users 357044984 Apr 5 14:15 messages-20170314.txt
-rw------- 1 mlimardo users 4578230 Apr 5 14:15 messages-20170320.txt
-rw------- 1 mlimardo users 202142286 Apr 5 14:15 messages-20170321.txt
-rw------- 1 mlimardo users 130368890 Apr 5 14:15 messages-20170322.txt
-rw------- 1 mlimardo users 165620408 Apr 5 14:15 messages-20170323.txt
-rw------- 1 mlimardo users 195113895 Apr 5 14:15 messages-20170324.txt
-rw------- 1 mlimardo users 148348587 Apr 5 14:15 messages-20170325.txt
-rw------- 1 mlimardo users 93239228 Apr 5 14:15 messages-20170326.txt
-rw------- 1 mlimardo users 93499447 Apr 5 14:15 messages-20170327.txt
-rw------- 1 mlimardo users 92053712 Apr 5 14:15 messages-20170328.txt
-rw------- 1 mlimardo users 92082236 Apr 5 14:15 messages-20170329.txt
-rw------- 1 mlimardo users 188351393 Apr 5 14:15 messages-20170330.txt
-rw------- 1 mlimardo users 136865059 Apr 5 14:15 messages-20170331.txt
-rw------- 1 mlimardo users 137331900 Apr 5 14:15 messages-20170401.txt
-rw------- 1 mlimardo users 267305527 Apr 5 14:15 messages-20170402.txt
-rw------- 1 mlimardo users 153961683 Apr 5 14:15 messages-20170403.txt
-rw------- 1 mlimardo users 154299752 Apr 5 14:15 messages-20170404.txt
-rw------- 1 mlimardo users 153061660 Apr 5 14:15 messages-20170405.txt
-rw------- 1 mlimardo users 10441873 Apr 5 14:15 messages.txt
-rw------- 1 mlimardo users 320789 Apr 5 14:15 modules.txt
-rw------- 1 mlimardo users 1392075 Apr 5 14:14 mpio.txt
-rw------- 1 mlimardo users 1615071 Apr 5 14:14 network.txt
-rw------- 1 mlimardo users 9991 Apr 5 14:14 nfs.txt
-rw-r--r-- 1 mlimardo users 1904 Apr 6 12:06 nsa.html
-rw------- 1 mlimardo users 26392 Apr 5 12:00 ntp.txt
-rw------- 1 mlimardo users 88 Apr 5 14:13 ocfs2.txt
-rw------- 1 mlimardo users 16225871 Apr 5 14:13 open-files.txt
-rw------- 1 mlimardo users 58177 Apr 5 14:13 pam.txt
-rw------- 1 mlimardo users 15177 Apr 5 14:14 print.txt
-rw------- 1 mlimardo users 2148060 Apr 5 12:02 proc.txt
-rw------- 1 mlimardo users 262980 Apr 5 11:59 rpm.txt
-rw------- 1 mlimardo users 24003 Apr 5 14:14 samba.txt
-rw------- 1 mlimardo users 85 Apr 5 14:15 sam.txt
-rw------- 1 mlimardo users 120 Apr 5 14:14 sar.txt
-rw------- 1 mlimardo users 18929 Apr 5 14:14 security-apparmor.txt
-rw------- 1 mlimardo users 3095 Apr 5 12:00 security-audit.txt
-rw------- 1 mlimardo users 86 Apr 5 14:12 slert.txt
-rw------- 1 mlimardo users 21483 Apr 5 14:14 slp.txt
-rw------- 1 mlimardo users 80 Apr 5 14:13 smt.txt
-rw------- 1 mlimardo users 10176 Apr 5 14:14 ssh.txt
-rw------- 1 mlimardo users 81 Apr 5 14:13 sssd.txt
-rw------- 1 mlimardo users 3335 Apr 5 14:15 summary.xml
-rw------- 1 mlimardo users 6701 Apr 5 14:15 supportconfig.txt
-rw------- 1 mlimardo users 278769 Apr 5 14:13 sysconfig.txt
-rw------- 1 mlimardo users 26216199 Apr 5 14:13 sysfs.txt
drwxr-xr-x 15 mlimardo users 197 Apr 6 10:04 system
-rw------- 1 mlimardo users 2290725 Apr 5 14:14 systemd.txt
-rw------- 1 mlimardo users 1371669 Apr 5 14:14 udev.txt
-rw------- 1 mlimardo users 11774230 Apr 5 14:13 updates.txt
-rw------- 1 mlimardo users 35587851 Apr 5 14:15 warn-20170223.txt
-rw------- 1 mlimardo users 4309774 Apr 5 14:15 warn-20170309.txt
-rw------- 1 mlimardo users 240189734 Apr 5 14:15 warn-20170313.txt
-rw------- 1 mlimardo users 343051059 Apr 5 14:15 warn-20170314.txt
-rw------- 1 mlimardo users 89089380 Apr 5 14:15 warn-20170321.txt
-rw------- 1 mlimardo users 12911522 Apr 5 14:15 warn-20170322.txt
-rw------- 1 mlimardo users 53898064 Apr 5 14:15 warn-20170323.txt
-rw------- 1 mlimardo users 97073894 Apr 5 14:15 warn-20170324.txt
-rw------- 1 mlimardo users 66114855 Apr 5 14:15 warn-20170325.txt
-rw------- 1 mlimardo users 6993923 Apr 5 14:15 warn-20170326.txt
-rw------- 1 mlimardo users 7055319 Apr 5 14:15 warn-20170327.txt
-rw------- 1 mlimardo users 7026144 Apr 5 14:15 warn-20170328.txt
-rw------- 1 mlimardo users 7028008 Apr 5 14:15 warn-20170329.txt
-rw------- 1 mlimardo users 74375827 Apr 5 14:15 warn-20170330.txt
-rw------- 1 mlimardo users 11870521 Apr 5 14:15 warn-20170331.txt
-rw------- 1 mlimardo users 11902952 Apr 5 14:15 warn-20170401.txt
-rw------- 1 mlimardo users 151080579 Apr 5 14:15 warn-20170402.txt
-rw------- 1 mlimardo users 11923045 Apr 5 14:15 warn-20170403.txt
-rw------- 1 mlimardo users 11907839 Apr 5 14:15 warn-20170404.txt
-rw------- 1 mlimardo users 11923863 Apr 5 14:15 warn-20170405.txt
-rw------- 1 mlimardo users 84 Apr 5 14:14 web.txt
-rw------- 1 mlimardo users 80 Apr 5 14:14 xen.txt
-rw------- 1 mlimardo users 159254 Apr 5 14:14 x.txt
-rw------- 1 mlimardo users 2772778 Apr 5 12:00 y2log.txt

Usually we don’t need to check all these files, everything depends on the problem that it is been troubleshooted but some of them are must-see:

• basic-environment.txt
• updates.txt

In basic-environment.txt, you will find information about the kernel version, the architecture, the SUSE product(s) and if the system is a virtual machine or physical:

#==[ Command ]======================================#
# /bin/uname -a
Linux hostname 3.12.59-60.45-default #1 SMP Sat Jun 25 06:19:03 UTC 2016 (396c69d) x86_64 x86_64 x86_64 GNU/Linux

#==[ System ]=======================================#
# Virtualization
Manufacturer: VMware, Inc.
Hardware: VMware Virtual Platform
Hypervisor: VMware (hardware platform)
Identity: Virtual Machine (hardware platform)

#==[ Configuration File ]===========================#
# /etc/os-release
NAME="SLES_SAP"
VERSION="12-SP1"
VERSION_ID="12.1.0.1"
PRETTY_NAME="SUSE Linux Enterprise Server for SAP Applications 12 SP1"
ID="sles_sap"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles_sap:12:sp1"

This is the first step for troubleshooting: we MUST know which product that particular problematic server is running, which kernel flavor and version, that’s just the starting point.

Then we always check updates.txt file which also provides a lot of useful info, for example the registration status:

#==[ Command ]======================================#
# /usr/sbin/SUSEConnect --status
[{"identifier":"SLES_SAP","version":"12.1","arch":"x86_64","status":"Registered"}]

Which repositories are configured:

#==[ Command ]======================================#
# /usr/bin/zypper --non-interactive --no-gpg-checks repos
# | Alias | Name | Enabled | GPG Check | Refresh
--+-----------------------------------------------------+------------------------+---------+-----------+--------
1 | SLE-12-SP1-SAP-12.1-0 | SLE-12-SP1-SAP-12.1-0 | Yes | (r ) Yes | No
2 | SMT-http_hostname:SLE-12-SP1-SAP-Updates | SLE-12-SP1-SAP-Updates | Yes | ( p) Yes | Yes
3 | SMT-http_hostname:SLE-HA12-SP1-Pool | SLE-HA12-SP1-Pool | Yes | (r ) Yes | No
4 | SMT-http_hostname:SLE-HA12-SP1-Updates | SLE-HA12-SP1-Updates | Yes | ( p) Yes | Yes
5 | SMT-http_hostname:SLE12-SP1-SAP-Pool | SLE12-SP1-SAP-Pool | Yes | (r ) Yes | No
6 | SMT-http_hostname:SLES12-SP1-Pool | SLES12-SP1-Pool | Yes | (r ) Yes | No
7 | SMT-http_hostname:SLES12-SP1-Updates | SLES12-SP1-Updates | Yes | ( p) Yes | Yes

From the information above, we have learnt that our system is a virtual machine running SLES for SAP 12 SP1, its kernel version is slightly old (from 2016); it is using an SMT server, is properly registered and all the needed repos are in place and enabled. And that’s already enough information to troubleshoot some kind of problems.

But there is more! One of the most common problem that land on our desks is booting issue hence, in this case, i would firstly check the file boot.txt where i can find the content of very important files related to Grub like /boot/grub/device.map or /boot/grub/menu.lst:

#==[ Configuration File ]===========================#

# /boot/grub/device.map
(hd2) /dev/sdd
(hd3) /dev/sde
(hd0) /dev/disk/by-id/scsi-36c81f660d5673d001a59930b1aaaef43
(hd4) /dev/disk/by-id/scsi-360000970000298700730533031324245
(hd1) /dev/sdc

#==[ Configuration File ]===========================#
# /boot/grub/menu.lst
default 0
timeout 8
gfxmenu (hd0,0)/message
title SUSE Linux Enterprise Server 11 SP4 - 3.0.101-100
 root (hd0,0)
 kernel /vmlinuz-3.0.101-100-default root=/dev/disk/by-id/scsi-36c81f660d5673d001a59930b1aaaef43-part3 resume=/dev/disk/by-id/scsi-36c81
f660d5673d001a59930b1aaaef43-part2 splash=nosplash crashkernel=256M-:128M cpufreq=no splash=silent showopts vga=0x31a
 initrd /initrd-3.0.101-100-default
title Failsafe -- SUSE Linux Enterprise Server 11 SP4 - 3.0.101-100
 root (hd0,0)
 kernel /vmlinuz-3.0.101-100-default root=/dev/disk/by-id/scsi-36c81f660d5673d001a59930b1aaaef43-part3 showopts ide=nodma apm=off noresu
me edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 nomodeset x11failsafe vga=0x31a
 initrd /initrd-3.0.101-100-default

If multipath is involved, i could check the mpio.txt file where i can verify which version of multipath package is running:

#==[ Verification ]=================================#
# rpm -V multipath-tools-0.4.9-122.1
# Verification Status: Passed

Verify my paths:

# /sbin/multipath -ll
mpathe (360000970000298700730533031333531) dm-2 EMC,SYMMETRIX
size=1.1G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
 |- 7:0:0:2 sdd 8:48 active ready running
 |- 7:0:1:2 sdg 8:96 active ready running
 |- 8:0:0:2 sdj 8:144 active ready running
 `- 8:0:1:2 sdm 8:192 active ready running
mpathd (360000970000298700730533031333446) dm-1 EMC,SYMMETRIX
size=103G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
 |- 7:0:0:1 sdc 8:32 active ready running

or verify the configurations in multipath.conf:

#==[ Configuration File ]===========================#
# /etc/multipath.conf
defaults {
 polling_interval 10
 path_selector "service-time 0"
 uid_attribute "ID_SERIAL"
 path_grouping_policy multibus
 prio const
 path_checker directio
 flush_on_last_del no
 max_fds max
 rr_min_io_rq 1
 rr_weight priorities
 failback immediate
 no_path_retry fail
 queue_without_daemon no
 user_friendly_names yes
 bindings_file "/etc/multipath/bindings"
 mode 644
 uid 0
 gid disk
}
blacklist {
 devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
}

The information in these text files are tremendously useful for our work but there is more and more to be explored. I hope you enjoyed reading my post but this is not the end..

..TO BE CONTINUED..