How to monitor /dev/null for attribute changes

This document (7021124) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 12
audit

Situation

An unknown process turns /dev/null from a character device into a regular file. /usr/bin/lsof -b +M -n -l shows processes complaining /dev/null being deleted:

[...]
systemd       1              0    2   unknown                          /dev/null (deleted) lstat: Resource temporarily unavailable) (stat: Resource temporarily unavailable)
dmeventd    649              0    0   unknown                          /dev/null (deleted) lstat: Resource temporarily unavailable) (stat: Resource temporarily unavailable)
haveged     656              0    0   unknown                          /dev/null (deleted) lstat: Resource temporarily unavailable) (stat: Resource temporarily unavailable)
lvmetad    1773              0    0   unknown                          /dev/null (deleted) lstat: Resource temporarily unavailable) (stat: Resource temporarily unavailable)
nscd       1899  1901      496    2   unknown                          /dev/null (deleted) lstat: Resource temporarily unavailable) (stat: Resource temporarily unavailable)

[...]

The goal is to monitor /dev/null to identify the accessing program.

Resolution

To recreate /dev/null please run the following commands:

rm /dev/null
mknod /dev/null -m 666 c 1 3

The Linux kernel auditing daemon provides a way to monitor attribute changes for given paths. For the monitoring to take place, please install the package audit and enable the daemon:

SLES 11 SP4:
insserv auditd
rcauditd start

SLE 12:
systemctl enable auditd
systemctl start auditd #(rcauditd <option> would work as well)

A non reboot persistent monitor may be set up easily using:

auditctl -w /dev/null -p a -k <monitorname>

For <monitorname> just pass a name that will help to identify the entries in the log file, e.g:

auditctl -w /dev/null -p a -k nullmonitor

If a reboot persistent solution is required, please edit /etc/audit/audit.rules and add the rule here:

# Feel free to add below this line. See auditctl man page

-w /dev/null -p a -k nullmonitor

When done, reload the auditd daemon:

rcauditd reload

Monitor /var/log/audit/audit.log for occurrences of the key "nullmonitor" and identify the process changing /dev/null.

Additional Information

A possible testcase

  1. Create a character device to monitor. For this use case the device /dev/nulltest will be created:

    mknod /dev/nulltest -m 666 -c 1 100
    sles12sp2:~ # ls -l /dev/nulltest
    crw-rw-rw- 1 root root 1, 100 Jul 27 11:57 /dev/nulltest

    Note the c at the start which identifies /dev/null as a character device.
    In order to avoid conflicts with already existing charater devices, please lookup existing character device numbers in /sys/dev/char/1\:*
    This will show a list of character devices currently managed by udev.

  2. Make sure the audit daemon was started and setup the monitor rule:

    rcauditd status
    auditctl -w /dev/nulltest -p a -k nullmonitor


  3. Now let's do something to /dev/nulltest:

    cd /root
    touch nulltest
    mv /root/nulltest /dev/
    sles12sp2:~ # ls -l /dev/nulltest
    -rw-r--r-- 1 root root 0 Jul 27 11:58 /dev/nulltest

    So we turned /dev/nulltest from a character device into a regular file.

  4. /var/log/audit/audit.log shows:

    rizzo:~ # grep nullmonitor /var/log/audit/audit.log
    type=CONFIG_CHANGE msg=audit(1501148396.172:3): auid=0 ses=1 op="add_rule" key="nullmonitor" list=4 res=1
    type=CONFIG_CHANGE msg=audit(1501148479.964:4): auid=0 ses=1 op="updated_rules" path="/dev/nulltest" key="nullmonitor" list=4 res=1
    type=CONFIG_CHANGE msg=audit(1501148479.964:5): auid=0 ses=1 op="updated_rules" path="/dev/nulltest" key="nullmonitor" list=4 res=1
    type=SYSCALL msg=audit(1501148479.964:6): arch=c000003e syscall=190 success=yes exit=0 a0=4 a1=7f00620200df a2=1a09080 a3=1c items=1 ppid=2304 pid=2843 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="mv" exe="/usr/bin/mv" key="nullmonitor"


Here we see /usr/bin/mv accessed /dev/nulltest. The syscall executed may get identified using ausyscall <arch> <number>, e.g. ausyscall x86_64 190:

sles12sp2:~ # ausyscall x86_64 190
fsetxattr

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7021124
  • Creation Date: 27-Jul-2017
  • Modified Date:03-Mar-2020
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center