SUSE Conversations


Netconsole howto: send kernel boot messages over ethernet



By: jslezacek

May 12, 2008 1:18 pm

Reads:617

Comments:3

Rating:0

Contents:

Environment

SUSE Linux Enterprise Server 10 Service Pack 1

Problem

Kernel boot messages need to be saved after a server crash/hang but access to a serial console is not available.

Solution

Netconsole can send boot kernel printk messages to another host on the network using UDP pakets.

Limitations

  • Only IP networks, UDP pakets and ethernet devices are supported.
  • The network card driver has to support the netpoll api.
  • Early kernel panics might not be captured.

Test systems

Server: (sending kernel messages) Client: (receiving kernel messages)
IP: 10.0.0.1 IP: 10.0.0.2
Source port: 4444 Receiving port: 6666
MAC: 00:1A:A0:D2:55:66

Server configuration

  1. Determine client MAC address

An easy way to do this is by pinging the client from the server with:

ping -c 1 10.0.0.2; clear; /sbin/arp -n |awk '/10.0.0.2 /{ print $3}'

Note: In case the remote logging client is on a separate subnet than the sender, you will need to know the MAC address of the default gateway.

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth0
0.0.0.0         10.0.0.254      0.0.0.0         UG    0      0        0 eth0

The default gateway is the one where “Destination” is 0.0.0.0. Determine the MAC address by using the ping command in step 1 with the IP of the “Gateway”, shown here as 10.0.0.254.

  1. Load the netconsole module at boot timeEdit /etc/sysconfig/kernel and change:
    MODULES_LOADED_ON_BOOT="netconsole"

    Run:

    SuSEconfig
  2. Netconsole module optionsEdit /etc/modprobe.conf.local and add:
    options netconsole netconsole=4444@10.0.0.1/eth0,6666@10.0.0.2/00:1A:A0:D2:55:66

    Explanation

    4444              => Server source port
    10.0.0.1          => Server IP
    eth0              => Server NIC
    6666              => Client listening port
    10.0.0.2          => Client IP
    00:1A:A0:D2:55:66 => Client MAC

Client configuration

  1. Open firewall port:

Edit /etc/sysconfig/SuSEfirewall2 and change:

FW_ALLOW_INCOMING_HIGHPORTS_UDP="6666"

The firewall needs to be restarted for the changes to take effect.

rcSuSEfirewall2 restart

If the client is already behind a firewall and security is not a concern,
you can alternatively disable SuSEfirewall2:

rcSuSEfirewall2 stop
  1. Receive kernel messagesThere are several methods how to read the kernel messages on the client.

    2.1 On screen reading:

    netcat -l -u -p 6666

    2.2 Log to a file:

    netcat -l -p 6666 -u 2>&1 | tee /var/log/netconsole &

    2.3 Syslog-ng log:

    Edit /etc/syslog-ng/syslog-ng and add:

    source net { udp(ip("0.0.0.0") port(6666)); };
    destination netconsole { file("/var/log/$HOST-netconsole.log"); };
    log { source(net); destination(netconsole); };

    Restart syslog-ng:

    service syslog restart

    You will find the logs under /var/log/$hostname-netconsole.log

    ls /var/log |grep netconsole
    10.0.0.1-netconsole.log

    A sample of the boot messages would look similar to this:

    head -15 /var/log/10.0.0.1-netconsole.log
    May  2 19:00:00 10.0.0.1 Linux version 2.6.16.54-0.2.5-bigsmp (geeko@buildhost) (gcc version 4.1.2 20070115 (prerelease) (SUSE Linux)) #1 SMP Mon Jan 21 13:29:51 UTC 2008
    May  2 19:00:00 10.0.0.1 BIOS-provided physical RAM map:
    May  2 19:00:00 10.0.0.1 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 0000000000100000 - 00000000efff0000 (usable)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000efff0000 - 00000000efffec00 (ACPI data)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000efffec00 - 00000000effff000 (reserved)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
    May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
    May  2 19:00:00 10.0.0.1 2943MB HIGHMEM available.
    May  2 19:00:00 10.0.0.1 895MB LOWMEM available.
    May  2 19:00:00 10.0.0.1 found SMP MP-table at 000fe710
    May  2 19:00:00 10.0.0.1 DMI 2.3 present.
    May  2 19:00:00 10.0.0.1 Using APIC driver default
    May  2 19:00:00 10.0.0.1 ACPI: PM-Timer IO Port: 0x808

Alternative solutions

Kernel messages can be redirected to a serial console if a serial port and a null modem cable are available.

See knowledge base article “Configuring a Remote Serial Console for SLES“.

Conclusion

Netconsole is a helpful tool for debugging kernel issues where you need to capture and analyze kernel messages from a crashed or hanging server when disk logging or serial consoles are unavailable.

External links

Netconsole kernel documentation

VN:F [1.9.22_1171]
Rating: 0.0/5 (0 votes cast)

Tags:
Categories: SUSE Linux Enterprise Server, Technical Solutions

Disclaimer: As with everything else at SUSE Conversations, this content is definitely not supported by SUSE (so don't even think of calling Support if you try something and it blows up).  It was contributed by a community member and is published "as is." It seems to have worked for at least one person, and might work for you. But please be sure to test, test, test before you do anything drastic with it.

3 Comments

  1. By:AustinJoseph

    Jozef nice writeup very useful in situations where sysadmins dont have access to serial console , just like in our situation.

  2. By:vbotka

    With SLED11 the following setup works for me:

    On the client:
    netcat -u -l -p 6666

    On the server:
    echo 9 4 1 7 > /proc/sys/kernel/printk
    modprobe netconsole netconsole=@/,@10.20.1.64/

  3. By:ronglux

    On RHEL platform i can’t not capture the boot message,but the boot message can be captured on SLES(D), Someone could tell me why

Comment

RSS