mysqld segfault when the system is under stress

This document (000020354) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 SP3
SUSE Linux Enterprise Server 15 SP2

Situation

When running SLES 15 SP2 with MySQL DB, MariaDB and Samba File under stress, the following segfault message is observed.

2021-06-02T04:44:46.960342+05:30 MonzaEST-P2-C1 kernel: [377264.064652] mysqld[8071]: segfault at 0 ip 000056306d8ca984 sp 00007f54ccff0ad0 error 6 in mysqld[56306cb70000+1392000]

Along with this error message a dip in the performance is seen when the segfault is observed.

The MariaDB log will also contain a message:

[ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.

Note that the impact of the segfault results in:
- mysql service becomes temporarily unavailable and restarts automatically.
- no data loss or data corruption observed.

Resolution

It may be possible to reduce the likelihood of exceeding the threshold by increasing the value of innodb_fatal_semaphore_wait_threshold.

This can be done through command-line parameters via

--innodb-fatal-semaphore-wait-threshold=<value>

Through my.cnf by setting

innodb-fatal-semaphore-wait-threshold = "<value>"

or through the client using the SET command:

SET GLOBAL innodb-fatal-semaphore-wait-threshold="<value>";

where <value> is the desired value.

More information on innodb_fatal_semaphore_wait_threshold can be found at
https://mariadb.com/kb/en/innodb-system-variables/#innodb_fatal_semaphore_wait_threshold

Other methods would involve increasing storage bandwidth or reducing the database workload.

Additional Information

Under conditions in which the storage bandwidth is saturated and iowait time is high, MariaDB semaphore wait time may periodically exceed the configured threshold (default 600 seconds) causing MariaDB to intentionally abort under the belief that it is hung. If this occurs, the MariaDB log will contain a message such as:

[ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.