SUSE Conversations


Running “crash” to analyze dump data on SLES11

debianized

By: debianized

April 16, 2013 3:56 pm

Reads:2674

Comments:0

Rating:5.0

Crash is a powerful tool that will analyze core dumps after crashing, to help you with troubleshooting or forensics analysis.

Coredump files after an incident are stored in:

/var/crash/$DATE

There is a README file in this path with basic info of the core dump file:

sles-beta:/var/crash/2013-03-27-10:32 # cat README.txt
Kernel crashdump
----------------

Crash time : 2013-03-27 10:31 (-0600)
Kernel version : 2.6.32.12-0.7-default
Host : linux
Dump level : 0
Dump format : compressed

sles-beta:/var/crash/2013-03-27-10:32 #

The way to run crash is:

crash vmlinux vmcore

NOTE: crash will store vmlinux compressed, we need to uncompress first:

# gzip -d vmlinux-2.6.32.12-0.7-default.gz

Then we run crash and we may have this error:

crash: vmlinux-2.6.32.12-0.7-default: no debugging data available
crash: vmlinux-2.6.32.12-0.7-default.debug: debuginfo file not found

crash: either install the appropriate kernel debuginfo package, or
copy vmlinux-2.6.32.12-0.7-default.debug to this machine

Notice that we said that we need a package, lets see:

sles-beta:/var/crash/2013-03-27-10:32 # rpm -qa | grep debug
sles-beta:/var/crash/2013-03-27-10:32 #

Indeed, we have no debug package, logic tells us that would be no problem installing it, see:

# zypper search *debug*

Apparently these are not the packages we need. What’s wrong?

Let’s see the repo list:

# zypper lr

That’s the problem, we have not activated the necessary repos, according to our distribution proceed to activate.

In this case it is SLES11 SP2:

# zypper mr --enable nu_novell_com:SLE11-SP2-Debuginfo-Core
# zypper mr --enable nu_novell_com:SLE11-SP2-Debuginfo-Updates

Then refresh references and zypper repos:

# zypper ref -s
# zypper refresh

And search again:

# zypper search debug

Now we list several packages related to the keyword debug, be more specific using the version of our kernel:

# uname -r

NOTE: If we are on the same server that generated the crash, we use that version of it, if we are doing the analysis from another server we need to use the exact version that was built with. This is seen in the README.txt.

sles-beta:/var/crash/2013-03-27-10:32 # grep "version" README.txt
Kernel version : 2.6.32.12-0.7-default
sles-beta:/var/crash/2013-03-27-10:32 #

Search for the correct Kernel debuginfo package and install it:

# zypper search -s kernel-*-debuginfo*

In this example it is: kernel-default-debuginfo-2.6.32.12-0.7.1, after install we are ready to run crash again.

crash vmlinux-2.6.32.12-0.7-default vmcore

Remember, we decompressed vmlinux at first.

Voilá!

In the first screen we have useful information, process name, pid, status, cpu, etc etc etc.

Now we can analyze the core dump using ‘backtrace‘, ‘files‘, ‘ps‘, ‘log‘ etc. And do the analysis as long and deep as desired.

Happy debugging!

VN:D [1.9.22_1171]
Rating: 5.0/5 (5 votes cast)
Running "crash" to analyze dump data on SLES11, 5.0 out of 5 based on 5 ratings

Tags:
Categories: Enterprise Linux, openSUSE, Server, SUSE Linux Enterprise, Technical Solutions

Disclaimer: As with everything else at SUSE Conversations, this content is definitely not supported by SUSE (so don't even think of calling Support if you try something and it blows up).  It was contributed by a community member and is published "as is." It seems to have worked for at least one person, and might work for you. But please be sure to test, test, test before you do anything drastic with it.

Comment

RSS