Processes in an Uninterruptible Sleep (D) State

This document (7002725) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10
 

Situation

Applications seem slow to respond or appear hung. Slow server performance. Processes stuck in a "D" state.

#==[ Checking Health of Processes ]=================#
# egrep " D| Z" /root/scc_server_210831_0955/psout.yKkLeVtM
                                            vv

root      3744  3654  0.0  0.0  16420   768 D+   00:00:00 ls -l /data
root      3964  3962  0.0  0.0   4480   668 D+   00:00:00 /bin/df -h

                                            ^^
 

Resolution

Processes in a "D" or uninterruptible sleep state are usually waiting on I/O. The ps command shows a "D" on processes in an uninterruptible sleep state. The vmstat command also shows the current processes that are "blocked" or waiting on I/O. The vmstat and ps will not agree on the number of processes in a "D" state, so don't be too concerned. You cannot kill "D" state processes, even with SIGKILL or kill -9. As the name implies, they are uninterruptible. You can only clear them by rebooting the server or waiting for the I/O to respond. It is normal to see processes in a "D" state when the server performs I/O intensive operations.

If performance becomes an issue, you may need to check the health of your disks. Make sure your firmware and kernel disk drivers are updated.

In the example above, the NFS server had failed while the client still had an active mount to it. Restarting the NFS server would resolve this particular issue.

There are two ways to find more about the processes in D state.

1. ps -eo ppid,pid,user,stat,pcpu,comm,wchan:32
This prints a list of all processes where in the last column either a '-' is displayed when the process is running or the name of the kernel function in which the process is sleeping if the process is currently sleeping. This includes also processes which are interruptible. Processes that are in uninterruptible sleep can be determined via the fourth column which would then show a D.

# ps -eo ppid,pid,user,stat,pcpu,comm,wchan:32 | grep " D"
 3654  3744 root     D+    0.0 ls              -
    1  3964 root     D     0.0 df              -


2. echo w > /proc/sysrq-trigger
This command produces a report and a list of all processes in D state and a full kernel stack trace to /var/log/messages. This shows much more information than the first option described above.

# echo w > /proc/sysrq-trigger
# dmesg -T
[Tue Aug 31 10:44:05 2021] task:ls              state:D stack:    0 pid: 3744 ppid:  3654 flags:0x00004006
[Tue Aug 31 10:44:05 2021] Call Trace:
[Tue Aug 31 10:44:05 2021]  __schedule+0x2ff/0x760
[Tue Aug 31 10:44:05 2021]  ? __rpc_wait_for_completion_task+0x30/0x30 [sunrpc]
[Tue Aug 31 10:44:05 2021]  ? rpc_exit+0x30/0x30 [sunrpc]
[Tue Aug 31 10:44:05 2021]  schedule+0x2f/0xa0
[Tue Aug 31 10:44:05 2021]  rpc_wait_bit_killable+0x1e/0xa0 [sunrpc]
[Tue Aug 31 10:44:05 2021]  __wait_on_bit+0x2c/0x90
[Tue Aug 31 10:44:05 2021]  out_of_line_wait_on_bit+0x91/0xb0
[Tue Aug 31 10:44:05 2021]  ? init_wait_var_entry+0x50/0x50
[Tue Aug 31 10:44:05 2021]  __rpc_execute+0x10c/0x420 [sunrpc]
[Tue Aug 31 10:44:05 2021]  ? nfs_weak_revalidate+0x67/0xd0 [nfs]
[Tue Aug 31 10:44:05 2021]  rpc_execute+0x71/0xd0 [sunrpc]
[Tue Aug 31 10:44:05 2021]  rpc_run_task+0x10d/0x140 [sunrpc]
[Tue Aug 31 10:44:05 2021]  rpc_call_sync+0x50/0x90 [sunrpc]
[Tue Aug 31 10:44:05 2021]  nfs3_rpc_wrapper.constprop.13+0x2f/0xa0 [nfsv3]
[Tue Aug 31 10:44:05 2021]  nfs3_proc_getattr+0x58/0xb0 [nfsv3]
[Tue Aug 31 10:44:05 2021]  __nfs_revalidate_inode+0xff/0x330 [nfs]
[Tue Aug 31 10:44:05 2021]  nfs_getattr+0x140/0x2d0 [nfs]
[Tue Aug 31 10:44:05 2021]  vfs_statx+0x89/0xe0
[Tue Aug 31 10:44:05 2021]  __do_sys_statx+0x3b/0x80
[Tue Aug 31 10:44:05 2021]  do_syscall_64+0x5b/0x1e0
[Tue Aug 31 10:44:05 2021]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Tue Aug 31 10:44:05 2021] RIP: 0033:0x7f0c9822943b
[Tue Aug 31 10:44:05 2021] Code: Bad RIP value.
[Tue Aug 31 10:44:05 2021] RSP: 002b:00007fff051af2c0 EFLAGS: 00000206 ORIG_RAX: 000000000000014c
[Tue Aug 31 10:44:05 2021] RAX: ffffffffffffffda RBX: 00007fff051b0796 RCX: 00007f0c9822943b
[Tue Aug 31 10:44:05 2021] RDX: 0000000000000100 RSI: 00007fff051b0796 RDI: 00000000ffffff9c
[Tue Aug 31 10:44:05 2021] RBP: 0000000000000100 R08: 00007fff051af2e0 R09: 00000000ffffff9c
[Tue Aug 31 10:44:05 2021] R10: 000000000000025e R11: 0000000000000206 R12: 00007fff051af2e0
[Tue Aug 31 10:44:05 2021] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000002
[Tue Aug 31 10:44:05 2021] task:df              state:D stack:    0 pid: 3964 ppid:     1 flags:0x00004002
[Tue Aug 31 10:44:05 2021] Call Trace:
[Tue Aug 31 10:44:05 2021]  __schedule+0x2ff/0x760
[Tue Aug 31 10:44:05 2021]  ? __rpc_wait_for_completion_task+0x30/0x30 [sunrpc]
[Tue Aug 31 10:44:05 2021]  ? rpc_exit+0x30/0x30 [sunrpc]
[Tue Aug 31 10:44:05 2021]  schedule+0x2f/0xa0
[Tue Aug 31 10:44:05 2021]  rpc_wait_bit_killable+0x1e/0xa0 [sunrpc]
[Tue Aug 31 10:44:05 2021]  __wait_on_bit+0x2c/0x90
[Tue Aug 31 10:44:05 2021]  out_of_line_wait_on_bit+0x91/0xb0
[Tue Aug 31 10:44:05 2021]  ? init_wait_var_entry+0x50/0x50
[Tue Aug 31 10:44:05 2021]  __rpc_execute+0x10c/0x420 [sunrpc]
[Tue Aug 31 10:44:05 2021]  ? nfs_weak_revalidate+0x67/0xd0 [nfs]
[Tue Aug 31 10:44:05 2021]  rpc_execute+0x71/0xd0 [sunrpc]
[Tue Aug 31 10:44:05 2021]  rpc_run_task+0x10d/0x140 [sunrpc]
[Tue Aug 31 10:44:05 2021]  rpc_call_sync+0x50/0x90 [sunrpc]
[Tue Aug 31 10:44:05 2021]  nfs3_rpc_wrapper.constprop.13+0x2f/0xa0 [nfsv3]
[Tue Aug 31 10:44:05 2021]  nfs3_proc_getattr+0x58/0xb0 [nfsv3]
[Tue Aug 31 10:44:05 2021]  __nfs_revalidate_inode+0xff/0x330 [nfs]
[Tue Aug 31 10:44:05 2021]  nfs_getattr+0x140/0x2d0 [nfs]
[Tue Aug 31 10:44:05 2021]  vfs_statx+0x89/0xe0
[Tue Aug 31 10:44:05 2021]  __do_sys_newstat+0x39/0x70
[Tue Aug 31 10:44:05 2021]  do_syscall_64+0x5b/0x1e0
[Tue Aug 31 10:44:05 2021]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Tue Aug 31 10:44:05 2021] RIP: 0033:0x7f0505d71095
[Tue Aug 31 10:44:05 2021] Code: Bad RIP value.
[Tue Aug 31 10:44:05 2021] RSP: 002b:00007ffd3c786828 EFLAGS: 00000246 ORIG_RAX: 0000000000000004
[Tue Aug 31 10:44:05 2021] RAX: ffffffffffffffda RBX: 00005636b25c87a0 RCX: 00007f0505d71095
[Tue Aug 31 10:44:05 2021] RDX: 00007ffd3c7868d0 RSI: 00007ffd3c7868d0 RDI: 00005636b25c7be0
[Tue Aug 31 10:44:05 2021] RBP: 0000000000000000 R08: 00005636b25c87a0 R09: 0000000000000003
[Tue Aug 31 10:44:05 2021] R10: 00005636b25c5010 R11: 0000000000000246 R12: 00005636b25c7c20
[Tue Aug 31 10:44:05 2021] R13: 00005636b25c7b80 R14: 0000000000000000 R15: 00007ffd3c7868b0

 

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7002725
  • Creation Date: 21-Feb-2009
  • Modified Date:31-Aug-2021
    • SUSE Linux Enterprise Desktop
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center