How to tell if system got rebooted or crashed

This document (000020272) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 15

Situation

System got rebooted however with default setup, we cannot tell if the system got gracefully rebooted or it was crashed/powered-off/fenced from outside. 

Knowing if system got crashed could help with certain Application/Database startup/recovery procedure. 

Resolution

To determine if the system got rebooted or crashed, we have two method: 
  1. Looking at previous systemd journal logs. 
  2. Writing two systemd services to track system status. 
To use systemd journal, journal logging should be persistent. For full documentation, check the link below. Simply: 
  • set Storage=persistent in /etc/systemd/journald.conf
  • then systemctl restart systemd-journald
When system get rebooted, this will be written to previous journal log. For example: 
 
s15sp2:~ # journalctl -b -1 | tail -6
Jun 02 14:16:57 s15sp2 systemd[1]: Reached target Shutdown.
Jun 02 14:16:57 s15sp2 systemd[1]: Reached target Final Step.
Jun 02 14:16:57 s15sp2 systemd[1]: Starting Reboot...
Jun 02 14:16:57 s15sp2 systemd[1]: Shutting down.
Jun 02 14:16:57 s15sp2 systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Jun 02 14:16:57 s15sp2 systemd-journald[415]: Journal stopped

To use systemd, we need to create two services. One to touch a file on shutdown and the other service to check file on system startup. The systemd service can be adjusted to run any script. Here is an example; just copy-paste it: 
cat <<EOF > /etc/systemd/system/graceful-shutdown.service
[Unit]
Description=graceful-shutdown.service
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/true
ExecStop=/bin/touch "/graceful-shutdown.flag"
RemainAfterExit=true

[Install]
WantedBy=multi-user.target
EOF

cat <<EOF > /etc/systemd/system/check-graceful-shutdown-flag.service
[Unit]
Description=check-graceful-shutdown-flag.service
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'test -f /graceful-shutdown.flag && { /usr/bin/logger "System was shutdown gracefully"; rm /graceful-shutdown.flag; } || /usr/bin/logger "System crashed"'
RemainAfterExit=true

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl enable graceful-shutdown.service check-graceful-shutdown-flag.service

When system get rebooted e.g. \reboot : 
 
s15sp2:~ # journalctl -b -u check-graceful-shutdown-flag.service
-- Logs begin at Tue 2021-01-19 10:48:02 CET, end at Fri 2021-06-04 13:57:48 CEST. --
Jun 04 13:57:42 s15sp2 systemd[1]: Starting check-graceful-shutdown-flag.service...
Jun 04 13:57:42 s15sp2 root[1967]: System was shutdown gracefully
Jun 04 13:57:42 s15sp2 systemd[1]: Started check-graceful-shutdown-flag.service.

When system get crashed e.g. echo c > /proc/sysrq-trigger : 
 
s15sp2:~ # journalctl -b -u check-graceful-shutdown-flag.service
-- Logs begin at Tue 2021-01-19 10:48:02 CET, end at Fri 2021-06-04 13:56:56 CEST. --
Jun 04 13:56:49 s15sp2 systemd[1]: Starting check-graceful-shutdown-flag.service...
Jun 04 13:56:49 s15sp2 root[1944]: System crashed
Jun 04 13:56:49 s15sp2 systemd[1]: Started check-graceful-shutdown-flag.service.

 

Additional Information

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020272
  • Creation Date: 04-Jun-2021
  • Modified Date:11-Jun-2021
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center