mysqld segfault when the system is under stress

This document (000020354) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 SP3
SUSE Linux Enterprise Server 15 SP2

Situation

When running SLES 15 SP2 with MySQL DB, MariaDB and Samba File under stress,  the following segfault message is observed.
2021-06-02T04:44:46.960342+05:30 MonzaEST-P2-C1 kernel: [377264.064652] mysqld[8071]: segfault at 0 ip 000056306d8ca984 sp 00007f54ccff0ad0 error 6 in mysqld[56306cb70000+1392000]
Along with this error message a dip in the performance is seen when the segfault is observed.

The MariaDB log will also contain a message:
[ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.

Note that the impact of the segfault results in:
- mysql service becomes temporarily unavailable and restarts automatically.
- no data loss or data corruption observed.

Resolution

It may be possible to reduce the likelihood of exceeding the threshold by increasing the value of innodb_fatal_semaphore_wait_threshold.
  • This can be done through command-line parameters via
--innodb-fatal-semaphore-wait-threshold=<value>
  • Through my.cnf by setting
innodb-fatal-semaphore-wait-threshold = "<value>"
  • or through the client using the SET command:
SET GLOBAL innodb-fatal-semaphore-wait-threshold="<value>";
where <value> is the desired value.

More information on innodb_fatal_semaphore_wait_threshold can be found at
https://mariadb.com/kb/en/innodb-system-variables/#innodb_fatal_semaphore_wait_threshold

Other methods would involve increasing storage bandwidth or reducing the database workload.

Additional Information

Under conditions in which the storage bandwidth is saturated and iowait time is high, MariaDB semaphore wait time may periodically exceed the configured threshold (default 600 seconds) causing MariaDB to intentionally abort under the belief that it is hung. If this occurs, the MariaDB log will contain a message such as:
[ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020354
  • Creation Date: 11-Aug-2021
  • Modified Date:11-Aug-2021
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center