System hangs and shows soft lockup error with kauditd call trace

This document (000021152) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12 SP5

Situation

The system is hanging and showing soft lockup error with kauditd call trace randomly.

An example of the call trace is shown below:

... kernel: [8621353.443255] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kauditd:52]
... kernel: [8621353.443532] CPU: 0 PID: 52 Comm: kauditd Tainted: P        W   EL     4.12.14-122.153-default #1 SLE12-SP5
... kernel: [8621353.443533] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
... kernel: [8621353.443534] task: ffff9afd4f6b8dc0 task.stack: ffffb6c7838e8000
... kernel: [8621353.443543] RIP: 0010:_raw_spin_lock_irqsave+0x8/0x40
... kernel: [8621353.443544] RSP: 0018:ffffb6c7838ebe38 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
... kernel: [8621353.443545] RAX: 0000000000000286 RBX: ffffffff9b8b5780 RCX: ffff9afa7561d300
... kernel: [8621353.443546] RDX: ffffffff9b8b5780 RSI: ffff9afa7561d900 RDI: ffffffff9b8b5794
... kernel: [8621353.443546] RBP: ffff9afa7561d900 R08: 0000000000000472 R09: 0000000000000001
... kernel: [8621353.443547] R10: 000000017fd147c6 R11: ffffb6c783a87d08 R12: ffffffff9b8b5794
... kernel: [8621353.443547] R13: 0000000000000000 R14: ffff9afa7561d9e4 R15: ffff9afa7561d900
... kernel: [8621353.443549] FS:  0000000000000000(0000) GS:ffff9afd7fc00000(0000) knlGS:0000000000000000
... kernel: [8621353.443549] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
... kernel: [8621353.443550] CR2: 00007f3a786a3000 CR3: 00000002ce00a002 CR4: 00000000007606f0
... kernel: [8621353.443558] PKRU: 55555554
... kernel: [8621353.443558] Call Trace:
... kernel: [8621353.443570]  skb_queue_head+0x1b/0x40
... kernel: [8621353.443574]  kauditd_send_queue+0xc1/0xf0
... kernel: [8621353.443575]  ? kauditd_retry_skb+0x20/0x20
... kernel: [8621353.443577]  ? kauditd_send_multicast_skb+0x80/0x80
... kernel: [8621353.443578]  kauditd_thread+0x90/0x210
... kernel: [8621353.443583]  ? wait_woken+0x80/0x80
... kernel: [8621353.443588]  kthread+0xf6/0x130
... kernel: [8621353.443589]  ? auditd_reset+0xb0/0xb0
... kernel: [8621353.443591]  ? kthread_bind+0x10/0x10
... kernel: [8621353.443592]  ret_from_fork+0x1f/0x40

Resolution

This bug is already fixed in SLES 12 SP5 kernel version 4.12.14-122.159.1 or later versions.

Cause

This issue is caused by a bug in the audit feature.

Additional Information

The release notes for the kernel patch "SUSE-SLE-SERVER-12-SP5-2023-2163" :
- audit: improve audit queue handling when "audit=1" on cmdline (bsc#1209969).

The changelog for the fix:
# rpm -q --changelog kernel-default
...
* Wed Apr 26 2023 ematsumiya@suse.de
- audit: improve audit queue handling when "audit=1" on cmdline
(bsc#1209969).
- commit 05326be

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.