How to increase the log level for Canal components in a Rancher Kubernetes Engine (RKE) or Rancher v2.x provisioned Kubernetes cluster

Task

During network troubleshooting it may be useful to increase the log level of the Canal components. This article details how to set verbose debug-level Canal component logging, in Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes clusters.

Pre-requisites

• A Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes cluster with the Canal Network Provider

Resolution

N.B. As these instructions involve editing the Canal DaemonSet directly, the change will not persist cluster update events, i.e. invocations of rke up for RKE CLI provisioned clusters, or changes to the cluster configuration for a Rancher provisioned cluster. As a result cluster updates should be avoided whilst collecting the debug level logs for troubleshooting.

Via the Rancher UI

For a Rancher v2.x managed cluster, the Canal component log level can be adjusted via the Rancher UI, per the following process:

1. Navigate to the System project of the relevant cluster within the Rancher UI.
2. Locate the canal DaemonSet workload within the kube-system namespace, click the vertical elipses (⋮) and select Edit.
3. Click to Edit the calico-node container.
4. Add CALICO_STARTUP_LOGLEVEL = DEBUG in the Environment Variables section and click Save.
5. Click Edit for the canal DaemonSet again.
6. This time click to Edit the kube-flannel container.
7. Click Show advanced options.
8. In the Command section add --v=10 to the Entrypoint e.g.: /opt/bin/flanneld --ip-masq --kube-subnet-mgr --v=10, and click Save.

Via kubectl

With a Kube Config file sourced for the relevant cluster, for a user with permission to edit the System project, the Canal component log level can be adjusted via kubectl, per the following process:

1. Run kubectl -n kube-system edit daemonset canal.
2. In the env definition for the calico-node container add an environment variable with the name CALICO_STARTUP_LOGLEVEL and value DEBUG, e.g.:

   [...]
         containers:
         - env:
           [...]
           - name: CALICO_STARTUP_LOGLEVEL
             value: DEBUG
   [...]

3. In the command definition for the kube-flannel container add --v=10 to the command, e.g.:

   [...]
         - commmand:
           - /opt/bin/flanneld
           - --ip-masq
           - --kube-subnet-mgr
           - --v=10
   [...]

4. Save the file.

Further reading

• Calico configuration documentation
• Flannel troubleshooting documentation