Configuring fadump on SLES12 SP3 and later

This document (7023277) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12 SP3 and later

SUSE Linux Enterprise Server 15 GA and later

Situation

Firmware assisted dump (fadump) is a dump capturing mechanism provided as a reliable alternative to kdump on IBM POWER systems.

'yast2 kdump' (YaST2 - Kernel Kdump) can be used to enable and configure it.

For a step-by-step guide, how to enable and test the settings, see the 'Resolution' tab.

Resolution

Enabling fadump:

1. On a text based terminal run as user root

yast2 kdump

to open the dump configuration tool.

2. In the Start-Up tab:

   a. select 'Enable Kdump'
   b. select 'Use Firmware-Assisted Dump'
   c. update 'Kdump Memory' according to the kernel's memory requirements
   d. select 'OK' to apply the settings

3. Reboot the system for the settings to take effect. After reboot check if the kdump service is up and running with:

systemctl status kdump.service -l

The output should show that the kdump.service is 'loaded' and 'active':

kdump.service - Load kdump kernel and initrd
Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: disabled)
Active: active (exited) since Wed 2018-08-15 12:39:18 PDT; 1 day 3h ago
Process: 66117 ExecStop=/lib/kdump/unload.sh (code=exited, status=0/SUCCESS)
Process: 66124 ExecStart=/lib/kdump/load.sh --update (code=exited, status=0/SUCCESS)
Main PID: 66124 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 4096)
CGroup: /system.slice/kdump.service

Aug 15 12:39:18 <hostname> systemd[1]: Starting Load kdump kernel and initrd...

4. Invoke a kernel crash, for example with

echo c > /proc/sysrq-trigger

The kernel should boot with an active dump, capture the vmcore & reboot into the production kernel.

5. For testing purposes: run the crash tool on the dumpfile/vmcore to check if the filtered vmcore could be analyzed.

6. Repeat steps 2c & 4 when cmdline or sysctl parameters are changed as memory requirements could change with a change in such parameters.

Disabling fadump:

1. Run as user root

yast2 kdump

to open the dump configuration tool.

2. In the Start-Up tab:
a. deselect 'Use Firmware-Assisted Dump'
b. select 'OK' to apply the settings

3. Reboot the system for the settings to take effect.

Note:

1. The crashkernel is used to reserve memory for fadump starting with SLES12 SP3. For recommended settings see Additional Notes.

2. YaST2 support to configure fadump was added starting with SLES12 SP1.
3. Always use the latest dump capture related packages (maintenance updates) for stability.

The packages typically are kdump, yast2-kdump and makedumpfile.

Additional Information

Starting with SLES12 SP3 - crashkernel is used to reserve memory for fadump and fadump_reserve_mem was deprecated. Recommended settings are:

System Memory	crashkernel setting (crashkernel=)
4 GB - 16 GB	768 MB
16 GB - 64 GB	1024 MB
64 GB - 128 GB	2 GB
128 GB - 1 TB	4 GB
1 TB - 2 TB	6 GB
2 TB - 4 TB	12 GB
4 TB - 8 TB	20 GB
8 TB - 16 TB	36 GB
16 TB - 32 TB	64 GB
32 TB - 64 TB	128 GB
64 TB & above	180 GB

Update the bootloader to pass the crashkernel setting

'crashkernel=<value>'

(e.g. crashkernel=2048M to the kernel

In the /etc/default/grub file, append 'crashkernel=<value> (e.g.: crashkernel=2048M) and if not there yet 'fadump=on' at the end of the GRUB_CMDLINE_LINUX_DEFAULT entries.

Run as user root

grub2-mkconfig -o /boot/grub2/grub.cfg

afterwards to update /boot/grub2/grub.cfg.

Note: The above recommended crashkernel values were arrived at testing with standard system configuration settings/resources. The values can vary depending on the system resources such as memory, CPU, disks and services and might need to be adjusted accordingly.

Note:
On Power10 systems with LMB (Logical Memory Block) size greater than 256MB there's a possibility for a kernel vmcore corruption.

Additional details:
IBM's Support document"Kernel dump corruption with Firmware-Assisted Dump (FADump)"
SUSE Knowledge Base Article - 000021429 - "fadump: Crash info header is not valid"

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.