How to test rancher-logging Flows and Outputs with local file output

• A Rancher v2.5+ managed Kubernetes clusters
• rancher-logging installed in the cluster

When configuring rancher-logging on a cluster, it may be necessary to troubleshoot Flows and Outputs. This guide provides a way to test these without any third-party tools involved (e.g. an external logging destination such as Elasticsearch).

1. The first step will be to create an Output on the cluster-logging-system namespace. You can do so by exploring the cluster within the Rancher UI, navigating to Logging > Outputs in the resources menu on the left side, and then clicking Create (alternatively, you can apply the manifest via the kubectl CLI). The following example manifest will create a basic Output that you can customize later. It will send the resulting logs to the /tmp/logs/ folder on the FluentD pod (rancher-logging-root-fluentd-0) in the cluster-logging-system Namespace:

   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Output
   metadata:
    name: test-output
    namespace: cattle-logging-system
   spec:
    file:
       path: /tmp/logs/${tag}/%Y/%m/%d/%H.%M
       buffer:
         timekey: 1m
         timekey_wait: 10s
         timekey_use_utc: true
   

2. The next step is to create a Flow that targets the pod created in step 3, using a label selector, and routes logs to the Output created in step 1. Again, this can be done from the UI by clicking on the Logging > Flows tab and clicking Create (or via kubectl):

   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Flow
   metadata:
     name: test-flow
     namespace: cattle-logging-system
   spec:
     localOutputRefs:
       - test-output
     match:
       - select:
           labels:
             logging-test: 'true'
   

3. Next, a dummy pod will be deployed to generate log messages with which to test the Flow and Output. In this example the swiss-army-knife debug image created by Rancher Support is used, but you can replace this as required by your testing. You can create this Pod via the Import YAML function at the top-right of Rancher the UI or via the kubectl CLI:

   apiVersion: v1
   kind: Pod
   metadata:
     name: test-logging-pod
     namespace: cattle-logging-system
     labels:
       logging-test: "true"
   spec:
     containers:
     - name: sak
       image: rancherlabs/swiss-army-knife:latest
   

4. With the Output, Flow, and Pod in place, you can now test the setup by executing a shell in the test pod and sending any desired test log messages to STDOUT (before doing this, be sure that the fluentd-configcheck pod in the cattle-logging-system Namespace is in status Completed):

   $ kubectl -n cattle-logging-system exec -it test-logging-pod -- /bin/bash
   # echo 'Hello World from Rancher Logging!' > /proc/1/fd/1
   

5. To view the results, you have to go to the /tmp/logs folder that will be created in the Fluentd pod (you might have to wait up to a few minutes for the log file to be populated, due to container log scraping and buffering by Fluent Bit). Inside the /tmp/logs directory will be a directory for the test Pod, and the log file will be in a child directory under the day, month and year:

6. $ kubectl -n cattle-logging-system exec -it rancher-logging-root-fluentd-0 -- /bin/sh
   
   $ ls -lrt /tmp/logs
   total 4
   drwxr-xr-x    3 fluent   fluent        4096 Feb 28 15:06 kubernetes.var.log.containers.test-logging-pod_cattle-logging-system_sak-8611b0ee3869dc245a4921aac63242901b104f77c86c7d58a32caafafe59aaaa.log
   
   $ cat 15.05_0.log 
   2024-02-28T15:05:50+00:00       kubernetes.var.log.containers.test-logging-pod_cattle-logging-system_sak-8611b0ee3869dc245a4921aac63242901b104f77c86c7d58a32caafafe59aaaa.log   {"log":"Hello World from Rancher Logging!\n","stream":"stdout","time":"2024-02-28T15:05:50.766535534Z","kubernetes":{"pod_name":"test-logging-pod","namespace_name":"cattle-logging-system","pod_id":"1d16a19e-f565-447b-b9f0-39bdf559eaf6","labels".....
   

7. You can now freely change your Output and Flow configuration to test and troubleshoot configurations, filters and parsers etc. The results will appear in the /tmp/logs/ folder of the Fluentd pod.