How to collect a stacktrace from the Rancher server process in Rancher v2.x
This document (000020039) is provided subject to the disclaimer at the end of this document.
Situation
Task
During troubleshooting it may be useful to collect a stacktrace from a running Rancher instance, and this article details the steps for creating one.
Pre-requisites
- A Rancher v2.x instance, either a single Docker container or a Highly Available (HA) installation in Kubernetes
Steps
A single Docker container installation of Rancher
-
Open two shell sessions on the server running the single container installation of Rancher.
-
In one of the shell sessions, run
docker ps
, to locate and note the ID or name of the Rancher server container. -
In the same shell, start streaming the Docker container logs to file with the command below, replacing
<RANCHER_CONTAINER_ID_OR_NAME>
with the Rancher container ID or name identified in step 2.docker logs -f <RANCHER_CONTAINER_ID_OR_NAME> > rancher.log 2>&1
-
In the other shell, send a SIGABRT signal to the rancher process in the container, to trigger a stacktrace, with the command below:
docker exec <RANCHER_CONTAINER_ID_OR_NAME> bash -c 'kill -SIGABRT $(pgrep -x rancher)'
-
When the stacktrace generation is complete the Rancher container will restart and the
docker logs
command in step 3. will exit. You can then provide therancher.log
file, containing the trace, to Rancher Support.
A Highly Available (HA) installation of Rancher in a Kubernetes cluster
-
Open two shell sessions and source an admin kubeconfig for the Rancher server cluster in both.
-
In one of the shell sessions, determine the name (e.g.
rancher-7b9f4764f5-rs2jx
) of the Rancher leader pod per the steps in the article here. -
In the same shell, start streaming the Rancher leader pod's logs to file, replacing with the pod name identified in step 2.:
kubectl logs -n cattle-system --tail=-1 -f <LEADER_POD_NAME> > rancher.log
-
In the other shell, send a SIGABRT signal to the rancher process in the leader pod, with the command below, to trigger a stacktrace, replacing with the pod name identified in step 2.:
kubectl -n cattle-system exec <LEADER_POD_NAME> -- bash -c 'kill -SIGABRT $(pgrep -x rancher)'
-
- When the stacktrace generation is complete the Rancher leader pod will restart and the
kubectl logs
command in step 3. will exit. You can then provide therancher.log
file, containing the trace, to Rancher Support.
- When the stacktrace generation is complete the Rancher leader pod will restart and the
Validating the success of the stacktrace
If you want to validate the stacktrace was successfully generated, you can confirm the presence of the SIGABRT
signal and goroutine
traces in the rancher.log
file as below:
SIGABRT: abort
PC=0x461781 m=0 sigcode=0
goroutine 0 [idle]:
runtime.futex(0x7370308, 0x80, 0x0, 0x0, 0xc000000000, 0x7ffdd633d8f0, 0x435863, 0xc000446848, 0x7ffdd633d910, 0x40b04f, ...)
/usr/local/go/src/runtime/sys_linux_amd64.s:535 +0x21
runtime.futexsleep(0x7370308, 0x7ffd00000000, 0xffffffffffffffff)
/usr/local/go/src/runtime/os_linux.go:44 +0x46
runtime.notesleep(0x7370308)
/usr/local/go/src/runtime/lock_futex.go:151 +0x9f
runtime.stoplockedm()
/usr/local/go/src/runtime/proc.go:2068 +0x88
runtime.schedule()
/usr/local/go/src/runtime/proc.go:2469 +0x485
runtime.park_m(0xc000be4780)
/usr/local/go/src/runtime/proc.go:2610 +0x9d
runtime.mcall(0x0)
/usr/local/go/src/runtime/asm_amd64.s:318 +0x5b
This should end with a section similar to the following after the goroutine traces:
rax 0xca
rbx 0x73701c0
rcx 0x461783
rdx 0x0
rdi 0x7370308
rsi 0x80
rbp 0x7ffdd633d8d8
rsp 0x7ffdd633d890
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x0
r13 0x1
r14 0xc000dde7e0
r15 0x0
rip 0x461781
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Further reading
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000020039
- Creation Date: 06-May-2021
- Modified Date:06-May-2021
-
- SUSE Rancher
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com