Troubleshooting duplicate IP on Harvester node

Harvester 1.4.0

Harvester 1.4.1

Harvester 1.4.2

Harvester node will not getting IP address, throwing a message similar to this:

wickedd[]: mgmt-br.XXXX: IPv4 duplicate address ipv4 10.X.XXX.2/24 detected (in use by ec:XX:XX:XX:XX:XX)!

To troubleshoot IP duplicate issues, we can create debug services and files on Harvester node using the following commands:

cat > /oem/92_wicked_debug.yaml <<EOT
name: Enable Wicked Debug
stages:
    initramfs:
        - commands:
            - sed -i 's/^WICKED_DEBUG.*/WICKED_DEBUG="all"/' /etc/sysconfig/network/config
            - sed -i 's/^WICKED_LOG_LEVEL.*/WICKED_LOG_LEVEL="debug"/' /etc/sysconfig/network/config
EOT

cat > /etc/systemd/system/tcpdump-arp-raw.service <<EOT
[Unit]
Description=capture raw arp messages
After=network-pre.target
Before=network.service
 
[Service]
Type=simple
ExecStart=/usr/sbin/tcpdump -w /root/tcpdump-arp.pcap --print -Uvvvnfi any arp
SyslogIdentifier=tcpdump-arp-raw
 
[Install]
WantedBy=multi-user.target
EOT

systemctl enable tcpdump-arp-raw.service

mkdir -p /etc/systemd/journald.conf.d

cat > '/etc/systemd/journald.conf.d/99-no-ratelimit.conf' <<EOT
[Journal]
        RateLimitInterval=0
        RateLimitIntervalSec=0
        RateLimitBurst=0
EOT

systemctl daemon-reload

and reboot the Harvester node to apply the configurations.

The outputs and files that can be useful for troubleshooting are:

# provides the output of
journalctl -b -o short-precise > journal.log
# send us this file
/tmp/tcpdump-arp.pcap
# collect the configuration
tar -czf config-ifcfg.tgz /etc/sysconfig/network
wicked show-config > config-dump.log
# collect the status
wicked ifstatus --verbose all > status.log
ip addr show > ip_addr.log
ip route show table all > routes.log
ip neigh show > ip_neigh_show.log
rpm -qa | grep wicked > wicked-version.log
# collect network card information
hwinfo --netcard > hwinfo-netcard.log

If the captures were usefull, these are the commands to disable all the debug stuff we enabled before:

systemctl stop tcpdump-arp-raw.service
systemctl disable tcpdump-arp-raw.service
rm /etc/systemd/journald.conf.d/99-no-ratelimit.conf
systemctl daemon-reload

mv /oem/92_wicked_debug.yaml /oem/92_wicked_debug.yaml.off

then reboot the Harvester node to apply the changes

The code in wicked (network manager) has been improved significantly on Harvester v1.4.X OS images.

arp: increase arp-send retry value to avoid address configuration failure due to ENOBUF reported by kernel while duplicate address detection with underlying bonding in 802.3ad mode reporting link "up & running" too early

This means that the probability to detect duplicates is much much higher than with older versions, especially compared to Harvester v1.3.X OS images.

For completely disable the IP duplicate check feature, you can run this commands on Harvester node:

cat > /oem/93_wicked_duplicate.yaml <<EOT
name: Disable wicked check duplicate
stages:
    initramfs:
        - commands:
            - sed -i 's/^CHECK_DUPLICATE_IP.*/CHECK_DUPLICATE_IP="no"/' /etc/sysconfig/network/config
EOT