How to obtain systemd service core dumps

This document (7017137) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 SP4
SUSE Linux Enterprise Server 15 SP3
SUSE Linux Enterprise Server 15 SP2
SUSE Linux Enterprise Server 15 SP1
SUSE Linux Enterprise Server 15 GA

SUSE Linux Enterprise Server 12 SP5
SUSE Linux Enterprise Server 12 SP4
SUSE Linux Enterprise Server 12 SP3
SUSE Linux Enterprise Server 12 SP2
SUSE Linux Enterprise Server 12 SP1
SUSE Linux Enterprise Server 12 GA

Situation

After following the steps outlined in TID3054866 - How to obtain application core dumps, the systemd services still will not write a core file to disk.

Some systemd units are currently in a failed state, shown when running "systemctl --failed" and it is needed to capture a core dump the next time the unit fails.

Resolution

For SLES12 to SLES12 SP2

Temporary configuration:
echo '|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e' > /proc/sys/kernel/core_pattern
or persistent configuration:
  1. define kernel.core_pattern configuration
 
echo 'kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e' > /etc/sysctl.d/50-coredump.conf
  1. reboot

Run sysctl -a | grep kernel.core_pattern to confirm the correct core_pattern command.

In order to see coredumps, run:
 
sles12sp1:~ # systemd-coredumpctl list 
TIME                           PID   UID   GID SIG EXE
Fri 2016-01-08 15:01:10 MST   1088     0     0   6 /usr/sbin/sshd
Fri 2016-01-08 15:34:04 MST   1655     0     0   6 /usr/sbin/sshd
Fri 2016-01-08 15:34:05 MST  21491     0     0   6 /usr/sbin/sshd
Fri 2016-01-08 15:35:27 MST   1361     0     0   6 /usr/sbin/cron
Fri 2016-01-08 15:35:36 MST  21501     0     0   6 /usr/sbin/sshd
Fri 2016-01-08 15:35:39 MST  21530     0     0   6 /usr/sbin/cron

To get the most recent sshd coredump, run:
sles12sp1:~ # systemd-coredumpctl -o core.sshd dump /usr/sbin/sshd
TIME                                         PID   UID   GID SIG EXE
              Fri 2016-01-08 15:35:36 MST  21501     0     0   6 /usr/sbin/sshd
More than one entry matches, ignoring rest.

sles12sp1:~ # ls -l core.sshd
-rw-r--r-- 1 root root 954368 Jan  8 15:49 core.sshd

To get other coredumps, by PID# run:
 

systemd-coredumpctl -o FileName dump <#pid>
systemd-coredumpctl -o core.sshd dump 21491

 

NOTE: You must run systemd-coredumpctl dump to extract any core dumps you want out of the systemd journal before rebooting the server. The core dumps stored in the systemd journal will not persist after a server reboot. See systemd-coredumpctl(1) for more options.

Application cores can be stored in /var/lib/systemd/coredump depending on the storage setting in /etc/systemd/coredump.conf, see the coredump.conf(5) man page for options.


For SLES12SP3 to SLES15SP3

NOTE: The systemd command to manage core files changed from "systemd-coredumpctl" to simply "coredumpctl" and the default core pattern is already properly set to:
 

|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e



Also note that systemd is already by default configured to write core files to disk (/var/lib/systemd/coredump/). It should thus be possible to simply use getappcore against any existing application core files without the need to first use coredumpctl.


getappcore

Usually uploading  core file and list of installed packages (list of installed packages in included in supportconfig ) is enough for SUSE Technical Support. But getappcore(8) could be used to creates an archive containing the application core, and all files required to analyze it - including the binary which created the core, and all required shared libraries, plus RPM version information for further investigation by SUSE.

Run genappcore with core file path if using non-compressed core file:
 

getappcore <corefile>


of if using compressed core file then add path to binary of the process which crashed:
 

# coredumpctl info 1141
           PID: 1141 (sshd)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Tue 2021-11-09 10:33:54 CET (18min ago)
  Command Line: sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
    Executable: /usr/sbin/sshd
 Control Group: /system.slice/sshd.service
          Unit: sshd.service
         Slice: system.slice
       Boot ID: 752fe79f770444d4892c84daea7409e2
    Machine ID: cf02db235794ef8c4a358db66183e547
      Hostname: 192
       Storage: /var/lib/systemd/coredump/core.sshd.0.752fe79f770444d4892c84daea7409e2.1141.1636450434000000.zst
       Message: Process 1141 (sshd) of user 0 dumped core.
                
                Stack trace of thread 1141:
                #0  0x00007f59bef5f2c6 __select (libc.so.6 + 0xf72c6)
                #1  0x0000556956cc4815 n/a (sshd + 0x10815)
                #2  0x00007f59bee8d34d __libc_start_main (libc.so.6 + 0x2534d)
                #3  0x0000556956cc5eea n/a (sshd + 0x11eea)


# getappcore -b /usr/sbin/sshd /var/lib/systemd/coredump/core.sshd.0.752fe79f770444d4892c84daea7409e2.1141.1636450434000000.zst 
####################################################################
Get Application Core Tool, v1.52.8_dev2
Date:     11/09/21, 10:53:37
Server:   192
OS:       SUSE Linux Enterprise Server 15 SP3
Kernel:   5.3.18-57-default (x86_64)
Corefile: /var/lib/systemd/coredump/core.sshd.0.752fe79f770444d4892c84daea7409e2.1141.1636450434000000.zst
####################################################################

Local configuration file... None
Validating core file... Done
Validating binary file... Done
Checking Source Binary with chkbin... Done
Building list of required libraries... Done
Building list of required RPMs... rpm: no arguments given for query
Done
Building list of debuginfo RPMs...  Done
Setting gdb environment variables... Done
Creating gdb startup files... Done
Creating core archive... Done

Affected Binary: /usr/sbin/sshd
Coredump File:   /var/lib/systemd/coredump/core.sshd.0.752fe79f770444d4892c84daea7409e2.1141.1636450434000000.zst
Archive Name:    /var/log/scc_192_sshd_211109_105337_appcore.txz
Removing required files and directories ... Done

Finished!

 

Cause

Prior to SLES12 SP3 the systemd architecture by default does not write service unit core files to disk, it writes them to the systemd journal. The systemd-coredumpctl command can then be used to extract core dumps from the journal and write them to disk.
 
The core dumps stored in the journal are not persistent across reboots.

Additional Information

If "systemd-coredumpctl" shows no core dumps.

command "dmesg" shows:
[750.396864] systemd-journald[574]: Failed to write entry (25 items, 104362632 bytes) despite vacuuming, ignoring: Argument list too long
 
  • memory is insufficient, at least 2Gb of memory
  • also adjust the /etc/systemd/journald.conf changing the below parameters and reboot:
 
RuntimeMaxUse=1G
RuntimeMaxFileSize=1G

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7017137
  • Creation Date: 08-Jan-2016
  • Modified Date:12-Aug-2022
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center