Solving Network Woes After Cloning A SLES 10 Virtual Machine From Template On VMware Virtual Infrastructure / ESX 3.5 Server
So here I am reading all the latest tech magazines and boards watching how virtualization has quickly become the latest fad in the IT sector. Although IBM has been successfully implementing virtual machines since the 1960s, it has only recently become popular and advantageous to utilize virtualization on Intel/AMD based servers with the advent of VMware’s ESX server.
The company I currently work for decided to jump in the fad-mobile and plug VMware’s Virtual Infrastructure 3 virtualization suite into our infrastructure. With the deployment of VMware came many new concepts and techniques…not to mention learning curves…that I as an engineer have had to learn. I’ve noticed there are many tricks, configuration methods and nuances to running this suite.
Standing a Linux virtual server up is a no-brainer. The virtual machine does not know it is virtual and thus your configuration is straightforward just as if you had racked a 1U pizza box and installed the OS via CD-ROM or the network. In VMware when you want to deploy multiple servers you can clone virtual machines from a template instead of building one box individually at a time. Where things got interesting for me was when I built a SLES 10 SP1 template that had been hardened for security and customized for our environment.
I found that when I would clone a new Linux virtual machine it would lose network connectivity. I noticed that two NIC cards had appeared and neither one would establish link. I scoured the Internet and found a few helpful posts, but nothing that specifically addressed my problem. So to help anyone out there who has felt the same pain and had the same woes, I drafted up a mini-HOWTO for correcting the NIC issue when cloning a Linux server on VMware ESX server. So let’s get to it…
The environment I was working in included VMware Virtual Infrastructure 3 (VI), ESX Server version 3.5 and Intel Xeon based hardware. The specific Linux distribution being cloned was SuSE Linux Enterprise Server SP10.
Confirm MAC Address
The first step in setting up your Linux clone is to confirm the network settings for the new virtual machine (vm). VMware creates unique MAC addresses for each vm and requires that you assign the NIC card to a network connection. These MAC addresses will start with a 00:50. Confirm both that you have a unique MAC address for your clone as well as assigning it the appropriate network connection name. There are two ways you can do this.
Method 1) From the shell on the ESX server: look inside the vmx file associated with your new vm. Not sure where your vmx is located? From the prompt issue:
#find / -type f -name
Once it has returned the results, ‘cd’ into that directory and either cat or less the vmx file. You will see a section that addresses ethernet0. Confirm that ethernet0.generatedAddress MAC value is not the same from your template and that ethernet0.networkName has the correct value (see figure 1 for an example screenshot).
Figure 1 – VMX ethernet0 configuration.
Method 2) From the VI client select your virtual machine and edit the settings. Double-check the section MAC Address and the label in Network Connection (see the red dots in figure 2).
Figure 2 – Confirm in VI Client the MAC address and Network Connection.
The Nitty Gritty
Now that we have confirmed the MAC address and the network label we can get to the actual clean up process to re-establish our clone’s network connectivity. Follow these steps in order and you will be up in no time.
Figure 3 shows that the initial assignment of your network going to eth1. This is due to the fact the template is created with eth0 still containing the ip address assigned during the original Linux server build out. You will notice that PCnet32 is used by VMware as its NIC driver.
Figure 3 – Incorrect assignment of ethernet NIC.
Step 1: Delete the two persistent rules in ‘/etc/udev/rules.d/30-net_persistent_names.rules’.
There are two rules that have been created in this file. You will notice that they are aptly named eth0 and eth1 (see figure 4). If you are like me, you like to understand what you are doing instead of just mindlessly bumbling along and following some HOWTO. Linux used to have a very frustrating and archaic method to handle kernel interaction with devices. It was administratively intensive. After several generations we now have udev. This file-system is responsible for detecting, interacting with and creating nodes in the /dev directory. Each time the server is booted udev takes information passed by sysfs and filters it through rules set up by the administrator in /etc/udev/rules.d. This is how devices that either exist at boot up or are hot pluggable, such as a USB drive, are loaded and named. By deleting these rules and manipulating the NIC settings a bit later is how we will reset our server to have the correct NIC assignment.
Figure 4 – Delete ethernet NIC rules.
Fire up your favorite editor and delete the two entries (entire line) for eth0 and eth1.
Step 2: Start Yast or Yast2 and enter the Network Card configuration module.
You are presented with the current set up. A quick scan shows two NIC cards and if you look more closely you’ll notice that the primary eth0 card is the template server’s MAC and ip address (figure 5).
Figure 5 – Two NICs are installed at the moment.
Delete the first NIC that has the configuration from the template server. Edit the second AMD PCnet card adjusting the ip address, hostname, and default gateway (figures 6 through 9). There are two elements you want to be and check during this process. In the address set up screen confirm the Configuration Name id lists your assigned MAC address as discovered earlier in your vmx file (indicated by the red dot in figure 6). The other location you check the id is under Advanced -> Hardware. Confirm the Configuration Name id listed is the same as well.
Figure 6 – Configure ip address on remaining NIC. Confirm id name.
Figure 7 – Configure hostname, name server etc.
Figure 8 – Configure default gateway.
Figure 9 – Confirm in the hardware that the configuration name is the same MAC.
When you have finished the configuration of your new NIC and click on the “Next” button Yast will write the values out and return you to the main network card configuration overview menu. You will notice that the only card listed is your newly minted eth0. Click “Finish” and let Yast complete the write out.
Step 3: Reboot the virtual machine.
Now some folks may say that you can restart network services using rcnetwork or /etc/init.d/network restart, however the reason I suggest rebooting is two fold. First I’d like to see the kernel load eth0 up correctly during the start up (figure 10) and record it appropriately in dmesg.
Figure 10 – Post configuration reboot looks good. Eth0 has correct assignment now.
The second reason is I want to confirm that udev writes the correct values back into its network persistent rule set cleanly upon booting. After the reboot I can perform some quick status checks to make sure my cloned virtual machine is up and has established network connectivity. Note the red dot in figure 10. If you do not see the hardware assignment on the first line when you issue ifstatus eth0 you have an issue.
Figure 11 – Post configuration connectivity tests. Positive ping and status responses.
If you less out /etc/udev/rules.d/30-net_persistent_names.rules you will see only one entry labeled eth0 with the correct MAC address.
That’s it! Your Linux virtual machine cloned from your Linux base template is good to go. If you have any questions please feel free to contact me and I will be happy to help in any way I can. Linux is clearly the best operating system out there. Adding in the functionality of virtualization and VMware yields greater and if I might say a sweeter functionality. Combining the power of Linux and ESX (hey, it’s a modified Linux environment anyways) gives you as an administrator true high availability and disaster recovery capabilities never seen before. That’s one powerful penguin.