pidstat exists occasionally with segmentation fault
This document (000020577) is provided subject to the disclaimer at the end of this document.
Environment
SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 12
Situation
In environments monitoring processes using pidstat -d -t (e.g. running via a cronjob), pidstat crashes occasionally with a segmentation fault which can be found in the kernel buffer or system log:
[Thu Sep 16 09:46:07 2021] pidstat[2103]: segfault at 7f00f408d0c4 ip 00007f00f41bce0b sp 00007ffc090c3cb8 error 6 in libc-2.22.so[7f00f408d000+19b000]
[Tue Oct 5 09:46:09 2021] pidstat[27248]: segfault at 7f28c16d00c4 ip 00007f28c17ffded sp 00007ffe0a290528 error 7 in libc-2.22.so[7f28c16d0000+19b000]
[Tue Nov 9 08:00:16 2021] pidstat[27920]: segfault at 7f72ffbb8084 ip 00007f72ffce7ded sp 00007ffcccd301b8 error 7 in libc-2.22.so[7f72ffbb8000+19b000]
[Wed Dec 1 05:16:19 2021] pidstat[31775]: segfault at 7fcda7765024 ip 00007fcda7894e0b sp 00007ffff19542a8 error 7 in libc-2.22.so[7fcda7765000+19b000]
[Wed Jan 12 06:54:29 2022] pidstat[19232]: segfault at 7f451ac970c4 ip 00007f451adc6ded sp 00007ffd3111fab8 error 7 in libc-2.22.so[7f451ac97000+19b000]
The next execution of pidstat exists again just fine.
[Thu Sep 16 09:46:07 2021] pidstat[2103]: segfault at 7f00f408d0c4 ip 00007f00f41bce0b sp 00007ffc090c3cb8 error 6 in libc-2.22.so[7f00f408d000+19b000]
[Tue Oct 5 09:46:09 2021] pidstat[27248]: segfault at 7f28c16d00c4 ip 00007f28c17ffded sp 00007ffe0a290528 error 7 in libc-2.22.so[7f28c16d0000+19b000]
[Tue Nov 9 08:00:16 2021] pidstat[27920]: segfault at 7f72ffbb8084 ip 00007f72ffce7ded sp 00007ffcccd301b8 error 7 in libc-2.22.so[7f72ffbb8000+19b000]
[Wed Dec 1 05:16:19 2021] pidstat[31775]: segfault at 7fcda7765024 ip 00007fcda7894e0b sp 00007ffff19542a8 error 7 in libc-2.22.so[7fcda7765000+19b000]
[Wed Jan 12 06:54:29 2022] pidstat[19232]: segfault at 7f451ac970c4 ip 00007f451adc6ded sp 00007ffd3111fab8 error 7 in libc-2.22.so[7f451ac97000+19b000]
The next execution of pidstat exists again just fine.
Resolution
Please use
zypper lp -a --bug=1194679
to locate the respective patch containing the fix and
zypper patch <name-of-the-patch>
to apply.
zypper lp -a --bug=1194679
to locate the respective patch containing the fix and
zypper patch <name-of-the-patch>
to apply.
Cause
The issue is experienced under the following conditions:
1.) Using -t, to display tasks alongside processes.
2.) Having more than 100 tasks/processes.
3.) Because the `st_pid_list` array grows by doubling the size when it's reallocating, trying to write a task data into the latest bucket (if it's a process, instead, it reallocates fine, making even more unlikely that it happens again in the next iteration).
4.) pidstat trying to write task information right after, then writing to an unallocated location.
5.) All this happening because a thread just spawned right before pidstat started up or since the last iteration, if an interval is specified.
1.) Using -t, to display tasks alongside processes.
2.) Having more than 100 tasks/processes.
3.) Because the `st_pid_list` array grows by doubling the size when it's reallocating, trying to write a task data into the latest bucket (if it's a process, instead, it reallocates fine, making even more unlikely that it happens again in the next iteration).
4.) pidstat trying to write task information right after, then writing to an unallocated location.
5.) All this happening because a thread just spawned right before pidstat started up or since the last iteration, if an interval is specified.
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000020577
- Creation Date: 21-Feb-2022
- Modified Date:21-Feb-2022
-
- SUSE Linux Enterprise Server
- SUSE Linux Enterprise Server for SAP Applications
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com
