How to Simulate Network Outages and Wrong-wired Network Environments for Xen DomUs

by Fabian Herschel

Problem:

For some situations you need to test system and software behaviour in special error cases like network outages. Clusters, also demo clusters, always have to be tested in case of failing network connections. Using XEN to run systems in domUs makes it hard just to “plug off” the network cable. So how do you simulate a network failure for domUs?

Solution:

Each administrative step is done as user root on dom0. In parallel you should have access to the serial console of the domU and should be logged in as any user you want. This is just to test that the network outage works fine.

1. First figure out the names of your domains (you can skip this if you already know the domain names)

   xm list

   In our example the command gives the following result:

   Name                                      ID   Mem VCPUs      State   Time(s)
   Domain-0                                   0  2234     2     r-----    489.8
   sles10                                     4   512     1     -b----     10.3
   sles10-1                                       512     1               196.2

   Where sles10 is the domU, where we want to plug off the network cable.

2. Now we have to evaluate the network configuration of that domU.

   xm network-list <domU>

   In our Example: xm network-list sles10

   Following our example the command lists these information:

   Idx BE     MAC Addr.     handle state evt-ch tx-/rx-ring-ref BE-path
   0   0  00:16:3e:5c:5a:c5    0     4      8     1544 /1545    /local/domain/0/backend/vif/4/0
   1   0  00:16:3e:14:f9:04    1     4      9     1546 /1547    /local/domain/0/backend/vif/4/1
   2   0  00:16:3e:0f:da:6d    2     4      10    1548 /1549    /local/domain/0/backend/vif/4/2
   ...

   Following the BE-path value of the interface with index (Idx) 0, we find the interface vif4.0, which is a abbreviation of the BE-path:

   /local/domain/0/backend/vif/4/0 -> vif4.0

3. Before we can run the bridge configuration command, we need to figure out the bridge name the interface is assigned to:

   brctl show

   In our case the command returns:

   bridge name     bridge id               STP enabled     interfaces
   xenbr0          8000.feffffffffff       no              vif0.0
                                                           peth0
                                                           vif4.4
                                                           vif4.0
   ...

   In our example this means, the interface vif4.0 is assigned to xenbr0. We need both values to “plug out” the network connection from this bridge and to create a planned network outage for our tests.

4. Now it’s time to run a ping command inside the serial console. If you do not know how to start a serial console to a domU, use YaST and select virtualization and machine management. You can ping the default gateway or any other machine which will be reached via the xenbr0 network.

Click to view.

1. To create a network outage for that interface, we run the brctl delif command on the dom0.

   brctl delif <bridge> <interface>

   Example: brctl delif xenbr0 vif4.0

   The ping on the domU “sles10” seams to hang, sometimes later we get the message “Destination Host Unreachable”. This is the result of our planned network outage. If you test a cluster, the cluster links status should show reaction.

2. It’s easy to damage something, but we need a way back and to replug the interface into the right network bridge. Again we use the brctl command to do that job.

   brctl addif <bridge> <interface>

   Example: brctl addif xenbr0 vif4.0