Troubleshooting systemd jobs that are hung or stuck

This document (000020261) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise High Availability Extension 15
SUSE Linux Enterprise High Availability Extension 12
SUSE Linux Enterprise Server for SAP Applications 15
SUSE Linux Enterprise Server for SAP Applications 12

Situation

The purpose of this TID is to give a troubleshooting example of a systemd service that appears stuck or hung.

The server is taking longer than usual to boot. Some system services are not starting as expected. The systemctl list-jobs command shows running and waiting jobs. For example:
> sudo /bin/systemctl --no-pager list-jobs
JOB UNIT                                 TYPE  STATE  
208 getty@tty1.service                   start waiting
216 systemd-user-sessions.service        start waiting
213 hostinfo.service                     start running
207 getty.target                         start waiting
117 multi-user.target                    start waiting
219 systemd-update-utmp-runlevel.service start waiting
255 after-local.service                  start waiting
235 plymouth-quit-wait.service           start waiting
253 issue-generator.service              start waiting
240 plymouth-quit.service                start waiting

10 jobs listed.

 

Resolution

Usually the running job is holding up others. In this example the hostinfo.service is running and should be the focus of troubleshooting efforts. Show the services that depend on the running job. They cannot start until the running job completes.
> sudo /bin/systemctl --no-pager --after list-jobs
JOB UNIT                                 TYPE  STATE  
208 getty@tty1.service                   start waiting
    waiting for job 207 (getty.target/start)
216 systemd-user-sessions.service        start waiting
    waiting for job 117 (multi-user.target/start)
    waiting for job 208 (getty@tty1.service/start)
    waiting for job 235 (plymouth-quit-wait.service/start)
    waiting for job 240 (plymouth-quit.service/start)
213 hostinfo.service                     start running
    waiting for job 117 (multi-user.target/start)
    waiting for job 253 (issue-generator.service/start)
207 getty.target                         start waiting
    waiting for job 117 (multi-user.target/start)
    waiting for job 255 (after-local.service/start)
117 multi-user.target                    start waiting
    waiting for job 219 (systemd-update-utmp-runlevel.service/start)
219 systemd-update-utmp-runlevel.service start waiting
255 after-local.service                  start waiting
    waiting for job 117 (multi-user.target/start)
235 plymouth-quit-wait.service           start waiting
    waiting for job 117 (multi-user.target/start)
    waiting for job 208 (getty@tty1.service/start)
253 issue-generator.service              start waiting
    waiting for job 117 (multi-user.target/start)
    waiting for job 216 (systemd-user-sessions.service/start)
240 plymouth-quit.service                start waiting
    waiting for job 117 (multi-user.target/start)

10 jobs listed.
The multi-user.target and issue-generator.service cannot start until the hostinfo.service completes. Find out the service application hostinfo.service starts. If you have locate installed, you can use it to search for the hostinfo.service unit file. Many systemd units are kept in /usr/lib/systemd/system or /etc/systemd/system.
# locate hostinfo.service
/etc/systemd/system/default.target.wants/hostinfo.service
/usr/lib/systemd/system/hostinfo.service

# rpm -qf /usr/lib/systemd/system/hostinfo.service
hostinfo-1.0-18.15.noarch

# grep ExecStart /usr/lib/systemd/system/hostinfo.service
ExecStart=/usr/sbin/hostinfo

# rpm -qf /usr/sbin/hostinfo
hostinfo-1.0-18.15.noarch

# chkbin /usr/sbin/hostinfo
# cat /var/log/scc_chkbin_hostinfo_1937.txt 
####################################################################
Binary Check Tool, v1.02.1
Date:   05/27/21, 12:24:39
Kernel: 5.3.18-24.64-default, Hardware: x86_64
####################################################################

Checking Binary Ownership                                   ... Done
Checking for Shared Libraries                               ... Done
Validating Unique RPMs                                      ... Warning

#--[ Checking File Ownership ]-------------------------------------#
/usr/sbin/hostinfo                  - from RPM: hostinfo-1.0-18.15.noarch
 :Shell script

#--[ Validating Unique RPMs ]--------------------------------------#
Validating RPM: hostinfo-1.0-18.15.noarch            [   Warning   ]
.......T.  c /etc/hostinfo.conf
S.5....T.    /usr/sbin/hostinfo
In this case the size, md5sum and timestamp have all changed since /usr/sbin/hostinfo was installed. Reinstall the hostinfo RPM package and test again. If you have a server with the same service working correctly, compare the chkbin output from the working and affected servers.

If no issues are reported with the application, you might consider opening a support case with SUSE and include a supportconfig after running chkbin on the running services.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020261
  • Creation Date: 27-May-2021
  • Modified Date:27-May-2021
    • SUSE Linux Enterprise High Availability Extension
    • SUSE Linux Enterprise Server
    • SUSE Linux Enterprise Server for SAP Applications

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center