Unmounting Runtime Directory Failed During Shutdown

This document (7023942) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12

Situation

A system was upgraded from SLES 11 SP4 to SLES 12 during shutdown the following error can be seen: 

dracut: Disassembling device-mapper devices , Command failed

this error can be seen on s390x but also on x86 when enabling debugging during shutdown. Detailed output: 

Apr 25 16:32:38 linux-aoie systemd[1]: Unmounting Runtime Directory...
Apr 25 16:32:38 linux-aoie systemd[1]: Unmounted Lock Directory.
Apr 25 16:32:38 linux-aoie umount[2531]: umount: /var/run: target is busy
Apr 25 16:32:38 linux-aoie umount[2531]:         (In some cases useful info abou
Apr 25 16:32:38 linux-aoie umount[2531]:          use the device is found by lso
Apr 25 16:32:38 linux-aoie systemd[1]: var-run.mount: Mount process exited, code
Apr 25 16:32:38 linux-aoie systemd[1]: Failed unmounting Runtime Directory.


Resolution

To fix such problem we need to make sure /var/run is pointing to /run
This can be done via two method: 

     1) Interactive by booting the system from SLES DVD and select "Rescue Mode
then mounting the root disk to /mnt 
then remove /mnt/var/run 
next, create a link /mnt/var/run -> /mnt/run 

Note  in the case /var is separate volume then mount var instead of / and adjust above paths 

     2) Automated by using convertfs dracut module: 
run the following command :
dracut --add convertfs -f 

add the following two kernel command line parameters to the kernel Command Line (use 'yast bootloader'): 
rd.convertfs rootflags=rw  systemctl reboot 

After doing above, the result should look something similar to below :

linux-aoie# ll -ld /var/run*
lrwxrwxrwx  1 root root    6 May 13 09:22 /var/run -> ../run
drwxr-xr-x 22 root root 4096 Apr 30 11:05 /var/run.runmove~

And the var-run.mount unit file should not be running: 

linux-aoie# systemctl status var-run.mount 
● var-run.mount - Runtime Directory
   Loaded: loaded (/usr/lib/systemd/system/var-run.mount; static; vendor preset: disabled)
   Active: inactive (dead

Cause

The cause of this issue is duplicated tmpfs Filesystem. The upgraded system end up with two run-time directories:

linux-aoie# mount | grep run 
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /var/run type tmpfs (rw,nosuid,nodev,mode=755)


In SLES 11 SP4 /var/run is used as tmpfs ; In SLES 12 /run is used as tmpfs. /var/run is a link to /run

linux-aoie# ll /var/run
lrwxrwxrwx 1 root root 4 May  2  2017 /var/run -> /run


But when /var/run is a directory (like with SLES 11), var-run.mount systemd unit gets activated and mounts /var/run as tmpfs 

linux-aoie# systemctl status var-run.mount
● var-run.mount - Runtime Directory
   Loaded: loaded (/usr/lib/systemd/system/var-run.mount; static; vendor preset: disabled)
   Active: active (mounted) since Thu 2019-04-25 16:39:46 CEST; 13min ago
    Where: /var/run
     What: tmpfs
  Process: 678 ExecMount=/usr/bin/mount /run /var/run -t bind -o bind (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 512)


This could be seen also in journalctl output: 

Apr 25 16:39:46 linux-aoie systemd[1]: var-run.mount: Directory /var/run to mount over is not empty, mounting anyway.
Apr 25 16:39:46 linux-aoie systemd[1]: Mounting Runtime Directory...
Apr 25 16:39:46 linux-aoie systemd[1]: Mounted Runtime Directory.


The following lines prevent systemd from mounting /var/run as tmpfs but systemd still mounting /var/run as tmpfs because its a directory. 

linux-aoie# grep Condition /usr/lib/systemd/system/var-run.mount
ConditionPathIsDirectory=/var/run
ConditionPathIsSymbolicLink=!/var/run

Additional Information

Above is a workaround for this problem. And should fix the issue. However SUSE Engineering is working on a fix that might be included in SLES12 SP5 DVD. 

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7023942
  • Creation Date: 13-Jun-2019
  • Modified Date:03-Mar-2020
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center