How to collect a stacktrace from the Rancher server process in Rancher v2.x

Task

During troubleshooting it may be useful to collect a stacktrace from a running Rancher instance, and this article details the steps for creating one.

Pre-requisites

• A Rancher v2.x instance, either a single Docker container or a Highly Available (HA) installation in Kubernetes

Steps

A single Docker container installation of Rancher

1. Open two shell sessions on the server running the single container installation of Rancher.

2. In one of the shell sessions, run docker ps, to locate and note the ID or name of the Rancher server container.

3. In the same shell, start streaming the Docker container logs to file with the command below, replacing <RANCHER_CONTAINER_ID_OR_NAME> with the Rancher container ID or name identified in step 2.

   docker logs -f <RANCHER_CONTAINER_ID_OR_NAME> > rancher.log 2>&1

4. In the other shell, send a SIGABRT signal to the rancher process in the container, to trigger a stacktrace, with the command below:

   docker exec <RANCHER_CONTAINER_ID_OR_NAME> bash -c 'kill -SIGABRT $(pgrep -x rancher)'

5. When the stacktrace generation is complete the Rancher container will restart and the docker logs command in step 3. will exit. You can then provide the rancher.log file, containing the trace, to Rancher Support.

A Highly Available (HA) installation of Rancher in a Kubernetes cluster

1. Open two shell sessions and source an admin kubeconfig for the Rancher server cluster in both.

2. In one of the shell sessions, determine the name (e.g. rancher-7b9f4764f5-rs2jx) of the Rancher leader pod per the steps in the article here.

3. In the same shell, start streaming the Rancher leader pod's logs to file, replacing with the pod name identified in step 2.:

   kubectl logs -n cattle-system --tail=-1 -f <LEADER_POD_NAME> > rancher.log

4. In the other shell, send a SIGABRT signal to the rancher process in the leader pod, with the command below, to trigger a stacktrace, replacing with the pod name identified in step 2.:

   kubectl -n cattle-system exec <LEADER_POD_NAME> -- bash -c 'kill -SIGABRT  $(pgrep -x rancher)'

5. 1. When the stacktrace generation is complete the Rancher leader pod will restart and the kubectl logs command in step 3. will exit. You can then provide the rancher.log file, containing the trace, to Rancher Support.

Validating the success of the stacktrace

If you want to validate the stacktrace was successfully generated, you can confirm the presence of the SIGABRT signal and goroutine traces in the rancher.log file as below:

SIGABRT: abort
PC=0x461781 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x7370308, 0x80, 0x0, 0x0, 0xc000000000, 0x7ffdd633d8f0, 0x435863, 0xc000446848, 0x7ffdd633d910, 0x40b04f, ...)
    /usr/local/go/src/runtime/sys_linux_amd64.s:535 +0x21
runtime.futexsleep(0x7370308, 0x7ffd00000000, 0xffffffffffffffff)
    /usr/local/go/src/runtime/os_linux.go:44 +0x46
runtime.notesleep(0x7370308)
    /usr/local/go/src/runtime/lock_futex.go:151 +0x9f
runtime.stoplockedm()
    /usr/local/go/src/runtime/proc.go:2068 +0x88
runtime.schedule()
    /usr/local/go/src/runtime/proc.go:2469 +0x485
runtime.park_m(0xc000be4780)
    /usr/local/go/src/runtime/proc.go:2610 +0x9d
runtime.mcall(0x0)
    /usr/local/go/src/runtime/asm_amd64.s:318 +0x5b

This should end with a section similar to the following after the goroutine traces:

rax    0xca
rbx    0x73701c0
rcx    0x461783
rdx    0x0
rdi    0x7370308
rsi    0x80
rbp    0x7ffdd633d8d8
rsp    0x7ffdd633d890
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x1
r14    0xc000dde7e0
r15    0x0
rip    0x461781
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

Further reading

• Rancher Documentation on Troubleshooting the Rancher Server Kubernetes Cluster