Running "crash" to analyze dump data on SLES11
Crash is a powerful tool that will analyze core dumps after crashing, to help you with troubleshooting or forensics analysis.
Coredump files after an incident are stored in:
/var/crash/$DATE
There is a README file in this path with basic info of the core dump file:
sles-beta:/var/crash/2013-03-27-10:32 # cat README.txt Kernel crashdump ---------------- Crash time : 2013-03-27 10:31 (-0600) Kernel version : 2.6.32.12-0.7-default Host : linux Dump level : 0 Dump format : compressed sles-beta:/var/crash/2013-03-27-10:32 #
The way to run crash is:
crash vmlinux vmcore
NOTE: crash will store vmlinux compressed, we need to uncompress first:
# gzip -d vmlinux-2.6.32.12-0.7-default.gz
Then we run crash and we may have this error:
crash: vmlinux-2.6.32.12-0.7-default: no debugging data available
crash: vmlinux-2.6.32.12-0.7-default.debug: debuginfo file not found
crash: either install the appropriate kernel debuginfo package, or
copy vmlinux-2.6.32.12-0.7-default.debug to this machine
Notice that we said that we need a package, lets see:
sles-beta:/var/crash/2013-03-27-10:32 # rpm -qa | grep debug sles-beta:/var/crash/2013-03-27-10:32 #
Indeed, we have no debug package, logic tells us that would be no problem installing it, see:
# zypper search *debug*
Apparently these are not the packages we need. What’s wrong?
Let’s see the repo list:
# zypper lr
That’s the problem, we have not activated the necessary repos, according to our distribution proceed to activate.
In this case it is SLES11 SP2:
# zypper mr --enable nu_novell_com:SLE11-SP2-Debuginfo-Core
# zypper mr --enable nu_novell_com:SLE11-SP2-Debuginfo-Updates
Then refresh references and zypper repos:
# zypper ref -s
# zypper refresh
And search again:
# zypper search debug
Now we list several packages related to the keyword debug, be more specific using the version of our kernel:
# uname -r
NOTE: If we are on the same server that generated the crash, we use that version of it, if we are doing the analysis from another server we need to use the exact version that was built with. This is seen in the README.txt.
sles-beta:/var/crash/2013-03-27-10:32 # grep "version" README.txt Kernel version : 2.6.32.12-0.7-default sles-beta:/var/crash/2013-03-27-10:32 #
Search for the correct Kernel debuginfo package and install it:
# zypper search -s kernel-*-debuginfo*
In this example it is: kernel-default-debuginfo-2.6.32.12-0.7.1, after install we are ready to run crash again.
crash vmlinux-2.6.32.12-0.7-default vmcore
Remember, we decompressed vmlinux at first.
Voilá!
In the first screen we have useful information, process name, pid, status, cpu, etc etc etc.
Now we can analyze the core dump using ‘backtrace‘, ‘files‘, ‘ps‘, ‘log‘ etc. And do the analysis as long and deep as desired.
Happy debugging!
No comments yet