My Favorites

Close

Please to see your favorites.

  • Bookmark
  • Email Document
  • Printer Friendly
  • Favorite
  • Rating:

How to obtain systemd service core dumps

This document (7017137) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 12 Service Pack 1 (SLES 12 SP1)
SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2)

Situation

After following the steps outlined in TID3054866 - How to obtain application core dumps, the systemd services still will not write a core file to disk.

You noticed some systemd units are currently in a failed state after running systemctl --failed and want to capture a core dump the next time the unit fails.

Resolution

Temporary configuration:
echo '|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e' > /proc/sys/kernel/core_pattern

or

Persistent configuration:
1. echo 'kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e' > /etc/sysctl.d/50-coredump.conf
2. reboot

Run sysctl -a | grep 'kernel.core_pattern' to confirm the correct core_pattern command.

UPDATE NOTE: The command "systemd-coredumpctl" has changed in SLES12-SP2 and above. In SLES12-SP2/SP3 the command is "coredumpctl".

In order to see coredumps, run:
sles12sp1:~ # systemd-coredumpctl list
TIME                                         PID   UID   GID SIG EXE
              Fri 2016-01-08 15:01:10 MST   1088     0     0   6 /usr/sbin/sshd
              Fri 2016-01-08 15:34:04 MST   1655     0     0   6 /usr/sbin/sshd
              Fri 2016-01-08 15:34:05 MST  21491     0     0   6 /usr/sbin/sshd
              Fri 2016-01-08 15:35:27 MST   1361     0     0   6 /usr/sbin/cron
              Fri 2016-01-08 15:35:36 MST  21501     0     0   6 /usr/sbin/sshd
              Fri 2016-01-08 15:35:39 MST  21530     0     0   6 /usr/sbin/cron
To get the most recent sshd coredump, run:
sles12sp1:~ # systemd-coredumpctl -o core.sshd dump /usr/sbin/sshd
TIME                                         PID   UID   GID SIG EXE
              Fri 2016-01-08 15:35:36 MST  21501     0     0   6 /usr/sbin/sshd
More than one entry matches, ignoring rest.

sles12sp1:~ # ls -l core.sshd
-rw-r--r-- 1 root root 954368 Jan  8 15:49 core.sshd

To get other coredumps, by PID# run:
coredumpctl -o FileName dump PID#
coredumpctl -o core.sshd dump 21491

NOTE:
You must run systemd-coredumpctl dump to extract any core dumps you want out of the systemd journal before rebooting the server. The core dumps stored in the systemd journal will not persist after a server reboot. See systemd-coredumpctl(1) for more options.

Application core can be stored in /var/lib/systemd/coredump depending on the storage setting in /etc/systemd/coredump.conf, see the coredump.conf(5) man page for options.

Run getappcore /path/to/core.sshd to prepare the core dump to upload to SUSE.
sles12sp1:~ # getappcore core.sshd
####################################################################
Get Application Core Tool, v1.28
Date:   01/08/16, 15:50:51
Server: sles12sp1
OS: SUSE Linux Enterprise Server 12  - SP1
Kernel: 3.12.49-11-default  (x86_64)
Corefile: core.sshd
####################################################################

Binary file not provided, trying to determine source binary using gdb... Done (/usr/sbin/sshd)
Checking Source Binary with chkbin... Done
Building list of required libraries with gdb... Done
Building list of required RPMs... Done
Building list of debuginfo RPMs...

     glibc-2.19-31.9.x86_64.rpm   -->   glibc-debuginfo-2.19-31.9.x86_64.rpm
     krb5-1.12.1-19.1.x86_64.rpm   -->   krb5-debuginfo-1.12.1-19.1.x86_64.rpm
     libaudit1-2.3.6-3.103.x86_64.rpm   -->   audit-debuginfo-2.3.6-3.103.x86_64.rpm
     libcom_err2-1.42.11-7.1.x86_64.rpm   -->   e2fsprogs-debuginfo-1.42.11-7.1.x86_64.rpm
     libkeyutils1-1.5.9-3.29.x86_64.rpm   -->   keyutils-debuginfo-1.5.9-3.29.x86_64.rpm
     libopenssl1_0_0-1.0.1i-34.1.x86_64.rpm   -->   openssl-debuginfo-1.0.1i-34.1.x86_64.rpm
     libpcre1-8.33-3.314.x86_64.rpm   -->   pcre-debuginfo-8.33-3.314.x86_64.rpm
     libselinux1-2.3-2.75.x86_64.rpm   -->   libselinux-debuginfo-2.3-2.75.x86_64.rpm
     libwrap0-7.6-886.3.x86_64.rpm   -->   tcpd-debuginfo-7.6-886.3.x86_64.rpm
     libz1-1.2.8-5.1.x86_64.rpm   -->   zlib-debuginfo-1.2.8-5.1.x86_64.rpm
     openssh-6.6p1-29.1.x86_64.rpm   -->   openssh-debuginfo-6.6p1-29.1.x86_64.rpm
     pam-1.1.8-14.1.x86_64.rpm   -->   pam-debuginfo-1.1.8-14.1.x86_64.rpm

                               ... Done
Setting gdb environment variables... Done
Creating gdb startup files... Done
Creating core archive... Done
        Created archive as:  /var/log/nts_sles12sp1_sshd_160108_1550_appcore.tbz
Removing required files and directories ... Done

Finished!

Cause

The systemd architecture does not write service unit core files to disk. It writes them to the systemd journal. You then use the systemd-coredumpctl command to extract core dumps from the journal and write them to disk. The core dumps stored in the journal are not persistent across reboots.


Additional Information

If "systemd-coredumpctl" shows no core dumps.

command "dmesg" shows:
[  750.396864] systemd-journald[574]: Failed to write entry (25 items, 104362632 bytes) despite vacuuming, ignoring: Argument list too long

- memory is insufficient - at least 2Gb of memory
- also adjusted the /etc/systemd/journald.conf changing parameters and reboot:
RuntimeMaxUse=1G
RuntimeMaxFileSize=1G

Disclaimer

This Support Knowledgebase provides a valuable tool for NetIQ/Novell/SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7017137
  • Creation Date:08-JAN-16
  • Modified Date:19-SEP-18
    • SUSESUSE Linux Enterprise Server
< Back to Support Search

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center