Collecting file logs with Host Tailer on the Logging Operator

This document (000020992) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Rancher 2.6.9
SUSE Rancher 2.7.x

Situation

Pre-requisites

Rancher Logging operator

To retrieve logs not written to the stdout (e.g. kube-audit logs), one can use the Banzai Host Tailer CRD provided by https://banzaicloud.com/. From the documentation: "HostTailer’s main goal is to tail custom files and transmit their changes to stdout". This way, the logging-operator can process them. Example usage is here. Similarly, you can use the file-tailer if you know the log file name.

Resolution

1. Create a HostTailer to retrieve the audit logs

apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: kubeaudit-hosttailer-sample
spec:
  fileTailers:
    - name: kube-audit
      path: /var/log/kube-audit/audit-log.json
      disabled: false

Note: It will be created in the "default" namespace if none is specified.

1.2 Verify the HostTailer object

kubectl get hosttailer -n  default

1.3 Verify the daemonset created by the HostTailer

kubectl get daemonsets -n default

1.4 Verify the HostTailer pod labels

$ kubectl get pods --show-labels -n default | grep -i hosttailer
kubeaudit-hosttailer-sample-host-tailer-n8h8q   1/1     Running   0          9d    app.kubernetes.io/instance=kubeaudit-hosttailer-sample-host-tailer,app.kubernetes.io/name=host-tailer

The label “app.kubernetes.io/name=host-tailer” will be used in the ClusterFlow object.

2. Modify or update if there is a ClusterFlow in place.

  match:
  - select:
      labels:
        app.kubernetes.io/name: host-tailer

Otherwise, create a new ClusterFlow and ClusterOutput following these examples.

2.1 Create ClusterFlow

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: flowhostfile
  namespace: cattle-logging-system
spec:
  globalOutputRefs:
  - output-test-hostfile
  match:
  - select:
      labels:
        app.kubernetes.io/name: host-tailer

For testing purposes, the ClusterOutput will output on a file on the Fluentd container.

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: output-test-hostfile
  namespace: cattle-logging-system
spec:
  file:
    path: /tmp/${tag}

Note the ${tag} on the path, required by Fluentd. In this case, the logs will be available on the /tmp folder of the pod rancher-logging-root-fluentd-0 in the cattle-logging-system namespace.

Cause

The underlying issue is that kube-audit logs are not written to stdout so the logging operator cant process them. This is why the Banzai Host Tailer CRD is needed to tail custom files and transmit their changes to stdout.

Additional Information

Rancher Integration with Logging Services

https://ranchermanager.docs.rancher.com/v2.7/pages-for-subheaders/logging

Flow and Outputs

RKE2 Kube-audit logs

https://www.suse.com/support/kb/doc/?id=000021171

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.