SUSE Conversations


Assign a Dedicated Network Card or PCI Device to a Xen Virtual Machine



By: jeshort

December 7, 2007 6:16 pm

Reads:673

Comments:2

Rating:0

Authors: Jim Short and Ron Terry

Description:

In this exercise you will “hide” a PCI device (network card in this instance) from the VM Server (Dom0) and assign it directly to a paravirtual Xen VM.

Steps:

  • Rebuild initrd to include the required pciback module
  • Hide the Network Card (NIC) from the Dom0 using a technique called “late binding”
  • Expose this device to a DomU
  • Make the configuration persistent

Instructions/Notes:

  • Fully apply all updates to the SLES 10 SP1 server
  • Install the NIC and verify that it is visible to the Dom0
  • Perform this exercise on the VM Server (Host Machine) booted to the XEN kernel
  • Login as ROOT to perform these operations
  • These instructions apply to any PCI device
  • This can currently only be done on a paravirtual Xen VM
  • This feature in not officially supported in SLES 10 SP1

Part I: Rebuild the initrd to include the pciback module

  1. In a text editor open the following file to be edited:
    /etc/sysconfig/kernel
  2. Edit the line that begins with INITRD_MODULES by adding the following to the end of the list (inside of the ” “) saving the file when complete:
    pciback

    The line will look something like the following (possibly with other drivers):

    INITRD_MODULES="piix mptapt ide-generic processor thermal fan reiserfs edd pciback"
    
    
  3. Rebuild the initrd by entering the following command on the command line:
    mkinitrd

    Note: You may ignore any errors relating to the modules.dep not being updated.

  4. Reboot to ensure the kernel module gets loaded and Xen is running.
  5. Ensure the pciback kernel module is loaded by entering the following at the command line:
    lsmod | grep pciback
    

Part II: Manually unbind/bind a PCI device in Dom0

  1. Determine the PCI ID of the NIC you will be assigning to the DomU:
    lspci

    or

    lspci | grep -i "Ethernet"
    

    You will something like the following:

    3f:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5755 Gigabit Ethernet PCI Express (rev 02)
    
    

    In this instance the NIC card PCI ID = 0000:3f:00.0

    Note: You will be using the long PCI ID throughout this exercise (i.e. 0000:3f:00.0)

  2. A symbolic link with the same name as the PCI ID will exist in the sys file system under the directory of the driver to which it is bound. To find the driver the PCI device is currently bound to enter the following at the command line:
    find /sys/bus/pci/drivers -name <Your PCI ID>
    

    You should see something similar to the following:

    /sys/bus/pci/drivers/tg3/0000:3f:00.0
    

    Note: <driver dir> = the directory after /sys/bus/pci/drivers . tg3 in this example) If the above command did not return anything do steps 3 and 4 and then repeat step 2. If it is successful proceed to step 5.

  3. If the find command does not “find” anything, try temporarily configuring the NIC using YaST. First enter the following command to determine what your networking script is:
    grep ^\(network-script /etc/xen/xend-config.sxp
    

    Your network script is the text following (network-script …) (i.e network-bridge or network-multinet)

  4. To avoid the error that occurs when YaST is loaded from Dom0 when Xen bridges are active, stop all VMs and deactivate the bridge(s) by issuing the following at the command line:
    /etc/xen/scripts/<your network script> stop
    

    After finishing with YaST, re-activate the Xen bridge(s):

    /etc/xen/scripts/<your network script> start
    

    Try step 2 again.

  5. Unbind the PCI device from its current driver by entering the following at the command line:
    echo -n "<Your PCI ID>" > "/sys/bus/pci/drivers/<driver dir>/unbind"
    
    
  6. Verify that the PCI device have been unbound by entering the following at the command line:
    ls -l /sys/bus/pci/drivers/<driver dir>
    

    You should no longer see a symbolic link with the PCI ID in this directory.

  7. Bind the PCI device to the pciback driver by entering the following at the command line:
    echo -n "<Your PCI ID>" > /sys/bus/pci/drivers/pciback/new_slot
    echo -n "<Your PCI ID>" > /sys/bus/pci/drivers/pciback/bind
    
    
  8. Verify that the device is now bound to the pciback driver by entering the following at the command line:
    ls -l /sys/bus/pci/drivers/pciback
    

    You should now see a symbolic link with the name of PCI ID in this directory.

  9. Repeat the previous steps for every NIC you wish to manually hide.

Part III: Connect the NIC to a DomU manually

  1. Boot the VM with the PCI device connected by entering the following at the command line
    xm create pci=Your PCI ID /etc/xen/vm/<vm config file>
    
  2. When the VM is finished booting log in as the root user
  3. See that the PCI device is now visible to the VM by entering the following at the command line in the VM:
    lspci
    

    You should see the pci device listed in the output:

    00:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5755 Gigabit Ethernet PCI Express (rev 02)
    
    
  4. Launch the YaST Network Card module to configure the network interface:
    YaST->Network Devices->Network card
    
  5. Select the Network Card from the list and then select Edit.
  6. Configure the card as desired and save changes.
  7. Test using ping, etc.

At this stage in the configuration, you can re-boot the DomU and the NIC configuration will still work. However, if you reboot Dom0 the NIC mapping will be lost. You need to complete the next steps to make it permanent.

Part IV: Configure pciback to hide the NIC at boot time

  1. In a text editor open the following file to be edited:
    /etc/modprobe.conf.local
  2. To insure that the pciback modules binds the NIC when it loads, add a new line at the bottom of the file as follows:
    options pciback hide=(<Your PCI ID>)

    options pciback hide=(<Your PCI ID>)
    
    Example:
    #
    # please add local extensions to this file
    #
    options pciback hide=(0000:3f:00.0)
    
    

    Note: To hide multiple PCI devices add:

    options pciback hide=(0000:00:1b.0 0000:00:13.0)
    
  3. To rebuild the initrd to include the updated modprobe.conf.local file enter the following at the command line:
    mkinitrd
    

    Note: Any time you edit the modprobe.conf file you must rebuild the initrd.

  4. Reboot the machine to test the new configuration.
  5. To verify that the change worked find the driver to which the NIC is currently bound by entering the following at the command line:
    find /sys/bus/pci/drivers -name <Your PCI ID>
    
  6. You should now see that a symbolic link with the name of the PCI ID exists in the /sys/bus/pci/drivers/pciback directory.

Part V: Configure a DomU to connect to a NIC persistently

  1. Using a text editor open the Virtual Machine’s configuration file (Do NOT use the .xml file) found in the /etc/xen/vm directory and add the following line to the end of the file:
    pci=['Your PCI ID']
    

    Note: To connect to multiple PCI devices the configuration change would look like:

    pci=['0000:00:1b.0','0000:00:13.0']
    
  2. If your VM is a managed VM use virt-manager to delete the current entry for this machine or from the command line enter.
    xm delete <vm name>
    
  3. If you wish your register your VM as a managed VM enter the following command:
    xm new /etc/xen/vm/<vm config file>
    
  4. If you wish to launch your VM using the traditional configuration file instead enter the following at the command line:
    xm create /etc/xen/vm/<vm config file>
    

List of useful commands for this exercise:

  • lsmod | grep pciback
  • gedit <filename to edit>
  • lspci
  • find /sys/bus/pci/drivers -name <Your PCI ID>
  • grep ^\(network-script /etc/xen/xend-config.sxp
  • /etc/xen/scripts/network-bridge stop/start/status
  • ls -l /sys/bus/pci/drivers/<driver dir>
  • ls -l /sys/bus/pci/drivers/pciback
  • virt-manager
  • xm create /etc/xen/vm/<vm config file>
  • xm new /etc/xen/vm/<vm config file>
  • xm delete <vm name>

This article is just one example of the lab exercises covered in the XEN Novell Advanced Technical Training courses. For more information on ATT courses see the ATT Home Page.

VN:F [1.9.22_1171]
Rating: 0.0/5 (0 votes cast)

Tags: , ,
Categories: SUSE Linux Enterprise Server, Technical Solutions

Disclaimer: As with everything else at SUSE Conversations, this content is definitely not supported by SUSE (so don't even think of calling Support if you try something and it blows up).  It was contributed by a community member and is published "as is." It seems to have worked for at least one person, and might work for you. But please be sure to test, test, test before you do anything drastic with it.

2 Comments

  1. By:coyote_de

    At least in SLES11

    options pciback hide=(0000:00:1b.0 0000:00:13.0)
    

    in /etc/modprobe.conf.local will not work.
    The correct syntax for hiding more than one PCI-Device (in SLES 11) seems to be

    options pciback hide=(0000:00:1b.0)(0000:00:13.0)
    
  2. By:ebelcher

    Using this excellent article, I added a SAS card in a blade server to a virtual machine running on it, exposing the RAID array and a Tape Library. I had to add smp_utils using YAST before I could see the devices, but it has worked very well.

    Environment is SLES11 SP1 64 bit for Dom0 and DomU

Comment

RSS