Contents:

Environment

SUSE Linux Enterprise Server 10 Service Pack 1

Problem

Kernel boot messages need to be saved after a server crash/hang but access to a serial console is not available.

Solution

Netconsole can send boot kernel printk messages to another host on the network using UDP pakets.

Limitations

  • Only IP networks, UDP pakets and ethernet devices are supported.
  • The network card driver has to support the netpoll api.
  • Early kernel panics might not be captured.

Test systems

Server: (sending kernel messages) Client: (receiving kernel messages)
IP: 10.0.0.1 IP: 10.0.0.2
Source port: 4444 Receiving port: 6666
MAC: 00:1A:A0:D2:55:66

Server configuration

  1. Determine client MAC address

An easy way to do this is by pinging the client from the server with:

ping -c 1 10.0.0.2; clear; /sbin/arp -n |awk '/10.0.0.2 /{ print $3}'

Note: In case the remote logging client is on a separate subnet than the sender, you will need to know the MAC address of the default gateway.

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth0
0.0.0.0         10.0.0.254      0.0.0.0         UG    0      0        0 eth0

The default gateway is the one where “Destination” is 0.0.0.0. Determine the MAC address by using the ping command in step 1 with the IP of the “Gateway”, shown here as 10.0.0.254.

  1. Load the netconsole module at boot timeEdit /etc/sysconfig/kernel and change:
    MODULES_LOADED_ON_BOOT="netconsole"

    Run:

    SuSEconfig
  2. Netconsole module optionsEdit /etc/modprobe.conf.local and add:
    options netconsole netconsole=4444@10.0.0.1/eth0,6666@10.0.0.2/00:1A:A0:D2:55:66

    Explanation

    4444              => Server source port
    10.0.0.1          => Server IP
    eth0              => Server NIC
    6666              => Client listening port
    10.0.0.2          => Client IP
    00:1A:A0:D2:55:66 => Client MAC

Client configuration

  1. Open firewall port:

Edit /etc/sysconfig/SuSEfirewall2 and change:

FW_ALLOW_INCOMING_HIGHPORTS_UDP="6666"

The firewall needs to be restarted for the changes to take effect.

rcSuSEfirewall2 restart

If the client is already behind a firewall and security is not a concern,
you can alternatively disable SuSEfirewall2:

rcSuSEfirewall2 stop
  1. Receive kernel messagesThere are several methods how to read the kernel messages on the client.

    2.1 On screen reading:

    netcat -l -u -p 6666

    2.2 Log to a file:

    netcat -l -p 6666 -u 2>&1 | tee /var/log/netconsole &

    2.3 Syslog-ng log:

    Edit /etc/syslog-ng/syslog-ng and add:

    source net { udp(ip("0.0.0.0") port(6666)); };
    destination netconsole { file("/var/log/$HOST-netconsole.log"); };
    log { source(net); destination(netconsole); };

    Restart syslog-ng:

    service syslog restart

    You will find the logs under /var/log/$hostname-netconsole.log

    ls /var/log |grep netconsole
    10.0.0.1-netconsole.log

    A sample of the boot messages would look similar to this:

    head -15 /var/log/10.0.0.1-netconsole.log
    May  2 19:00:00 10.0.0.1 Linux version 2.6.16.54-0.2.5-bigsmp (geeko@buildhost) (gcc version 4.1.2 20070115 (prerelease) (SUSE Linux)) #1 SMP Mon Jan 21 13:29:51 UTC 2008
    May  2 19:00:00 10.0.0.1 BIOS-provided physical RAM map:
    May  2 19:00:00 10.0.0.1 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 0000000000100000 - 00000000efff0000 (usable)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000efff0000 - 00000000efffec00 (ACPI data)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000efffec00 - 00000000effff000 (reserved)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
    May  2 19:00:00 10.0.0.1 2943MB HIGHMEM available.
    May  2 19:00:00 10.0.0.1 895MB LOWMEM available.
    May  2 19:00:00 10.0.0.1 found SMP MP-table at 000fe710
    May  2 19:00:00 10.0.0.1 DMI 2.3 present.
    May  2 19:00:00 10.0.0.1 Using APIC driver default
    May  2 19:00:00 10.0.0.1 ACPI: PM-Timer IO Port: 0x808

Alternative solutions

Kernel messages can be redirected to a serial console if a serial port and a null modem cable are available.

See knowledge base article “Configuring a Remote Serial Console for SLES“.

Conclusion

Netconsole is a helpful tool for debugging kernel issues where you need to capture and analyze kernel messages from a crashed or hanging server when disk logging or serial consoles are unavailable.

External links

Netconsole kernel documentation

(Visited 1 times, 1 visits today)
Tags:
Category: SUSE Linux Enterprise Server, Technical Solutions
This entry was posted Monday, 12 May, 2008 at 1:18 pm
You can follow any responses to this entry via RSS.

Comments

  • AustinJoseph says:

    Jozef nice writeup very useful in situations where sysadmins dont have access to serial console , just like in our situation.

  • vbotka says:

    With SLED11 the following setup works for me:

    On the client:
    netcat -u -l -p 6666

    On the server:
    echo 9 4 1 7 > /proc/sys/kernel/printk
    modprobe netconsole netconsole=@/,@10.20.1.64/

  • ronglux says:

    On RHEL platform i can’t not capture the boot message,but the boot message can be captured on SLES(D), Someone could tell me why

  • Leave a Reply

    Your email address will not be published. Required fields are marked *