rsyslog discards/delays local messages - if syslog remote target is down

This document (7023589) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 15

Situation

The SLE (SUSE Linux Enterprise) Server (rsyslog client) is configured to forward certain messages to a remote
syslog server via TCP (Transmisson Control Protocol) /Port 514.

The setup done with a disk-assisted memory queue.

In the issue scenario, remote syslog server becomes unreachable via network.

Once the messages queue (no matter, what size) becomes full, local messages are not
written any longer to e.g. /var/log/messages (important system messages can become 
invisible for an investigating system engineer)


Resolution

For a functional setup and to prevent main/local message queue to be impacted by a remote syslog server outage,
proper message discard rules have to be implemented, e.g.:

l9931022:~ # cat /etc/rsyslog.d/ora_audit.conf
if ( $syslogfacility-text == 'local1' ) and ( $syslogseverity == 4 /* warning */ )  then {
        $WorkDirectory /var/spool/rsyslog       # where to place spool files
        $ActionQueueFileName RemoteQueue        # unique name prefix for spool files
        $ActionQueueMaxDiskSpace 1G             # 1gb space limit (use as much as possible)
        $ActionQueueSaveOnShutdown on           # save messages to disk on shutdown
        $ActionQueueType LinkedList             # run asynchronously
        $ActionResumeRetryCount -1              # infinite retries if host is down
        $ActionQueueTimeoutEnqueue 0            # causes that the message will be discarded immediately if the queue is full
        *.* @@aaa.bbb.ccc.ddd:514               # IP of remote syslog server:port 514
        stop
}

Above setup discards any messages, if queue size limit is reached (in this setup 1 GB) immediately !

Cause

The initial customer configuration requested a disk-assisted ActionQueue with limited 
disk space and did not specify, what client-side rsyslog daemon should do, when the queue 
becomes full. 

With default values, the queue is trying not to drop any message unless necessary and as 
a consequence is not able to quickly handle the big load from reproducer.

Rsyslog has one main message queue for incoming messages. They are processed and stored in 
ActionQueues belonging to each action configured in rsyslog.conf.

If one of the ActionQueues is full and rsyslog cannot put a message there, the message is 
by default not discarded immediately, but it's placed back to main queue and rsyslog tries 
to deliver it later. So, a single full ActionQueue can significantly slow down the whole 
rsyslog daemon. Then it tries to throttle the data senders and free some queue space in 
the meantime. The throttling can slow down the whole server or even make it unresponsible, 
when processes like sshd are waiting on write to syslog socket.

To prevent that, the administrator has to configure proper discard rules. 
(as described in the "Resolution" example)

Rsyslog has fairly extensive options for specifying, what to discard and when.

For example, "$ActionQueueTimeoutEnqueue 0" option causes, that the message will be discarded 
immediately, if the queue is full, "$ActionQueueDiscardSeverity" could be set to drop all 
queued low-priority messages, if the treshold  "$ActionQueueDiscardMark" is reached or even 
increase the maximum number of threads, handling this queue, using "$ActionQueueWorkerThreads" 
to improve performance. 

Additional Information

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7023589
  • Creation Date: 14-Dec-2018
  • Modified Date:03-Mar-2020
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center