I have always tried to preach the importance of standardization. Having all of your servers set to a standard build reduces administration, aids in troubleshooting and increases the efficiency of global changes. It also reduces the amount of time new employees need to learn your server configurations.
Creating a Standard Server Template in VMware greatly reduces the time needed to deploy new servers and eliminates variations in servers caused by multiple engineers building from a guide. It also defines your standard of options and security settings defined by your organization’s various entities. I have my templates built so when I need to deploy a new server, all I have to configure is the NIC(s), change the hostname and make one small modification to the sudoers file. Reboot, server is ready for service.
The purpose of this article is to guide you through the steps required to build a new virtual template using SLES11 SP1 for deploying new Linux servers in VMware.
Once completed, you can have your VMware administrators to restrict access to modify the template, thus maintaining its integrity. Grant modification access to a select few.
To prepare for creating a Standard Server Template, there are a few things to consider.
- Disk size for the server base. Although more devices can be added later, a basic server installation should be considered as this will be, “the standard”
- What users need to be included. Does everyone one need to be included? Keep in mind that it is bad form to allow developers access to a production server. If they are included in the template, they will be on every server that is created from this template.
- What applications does your organization use that needs to be on every server? Monitoring, Backup, Administration tools. If it is something that needs a particular configuration, perhaps the package can be added to the template and then installed and configured post deployment.
- Multiple Data Centers. If you have multiple data centers you might want to consider cloning this template to those locations and then making modifications that are needed for that data center, like NTP servers.
This guide is separated into the following sections, most of which must be followed in their listed order. Some are guidelines and can be modified to reflect your organization’s own policies, but should be included in the template.
- Preparation of Template in the Virtual Console (VIC)
- Installation of OS – Initial
- LVM Configuration
- Installation Pre-Boot
- Installation Post-Boot
- VMware Tools Installation
- User Administration
- Additional Packages
- Final Cleanup
- Conversion to Template
Preparation of Template in the Virtual Console (VIC)
Copy your SLES 11 SP1 ISO image to a Datastore on the ESX server. We will use it to create our template.
Login to the VIC and select the site for this template’s creation.
Using the Inventory pick list, select Virtual Machines and Templates.
Right click the container you wish this template to reside and select Create New Virtual Machine.
Select Custom and click Next
Enter the name of the new template similarly, as shown. Select the location for the new template. Click Next.
Select the location for the template. Making sure that this location is accessible to any clusters and Virtual Hosts at this site. Click Next.
Since we have to build the template as a VM guest first, then convert it, we need to place it on a virtual host so we can load everything. Select a host to build out template on. Click Next.
Select the Datastore that this template will reside. Noting that enough space must be available for our template, which will be 50GB. Click Next.
Operating System used in this guide is Linux and the Version is SLES 11 32-bit. If you use SLES 64-bit, then adjust your selection accordingly. Click Next.
Select the number of CPUs and click Next.
Select the amount of RAM, remembering that you can add more to a particular deployment later, if needed. Click Next.
Leave the NICs to 1 and make sure Connect at Power On is checked. If you have a need for more, it is better to add them to a deployment than waste resources on the VM Host. Click Next.
Leave the I/O Adapters to default or change if your needs are different. Click Next.
Since we are creating a new server template, check Create a new virtual disk, and click Next.
I specify 50GB for the disk and Store with the virtual machine. This gives me plenty of room for the base OS and because I use Linux Volume Management (LVM) for my partitions, I can add more space to these partitions as needed and without interrupting service. Storing with the VM allows me to clone this template as whole without worrying if a particular cluster has access to any particular Datastore. Click Next.
I don’t perform any advanced options, so I Leave the Advanced Options as default and click Next.
On the summary screen, look over your choices and go Back if you need to make changes. When you are satisfied with your selections, click Finish. The new virtual machine will be created. Now let’s modify the new VM before we power it up.
Once the new VM is created, select it and click Edit Virtual Machine Settings on the right side pane.
I remove the Floppy Drive from my templates. Select CD/DVD Drive 1, Select Browse to Image where the SLES11SP1 ISO is located when you copied it in the beginning.
Ensure that Connect at Power On is checked, we’ll need to boot from it.
When you are done, click OK and the changes will be made.
Installation of OS – Initial
Now that we have created the new virtual machine and configured the settings, we are ready to install SLES 11 SP1. If you have not logged into the VIC and selected the new template VM, please do so before continuing.
Power on the new VM and quickly open a console. This is done because the bootable DVD, by default, will attempt to boot from hard disk. Using the arrow keys select Installation and press Enter.
The system will boot and the YaST installation software will load.
Agree to the license by checking the box in the lower middle left, and click Next.
This is the Media Integrity Check screen to ensure that your ISO image is intact. If you have used this image or DVD before, you can simply ignore this and click Next.
YaST is now probing your hardware. Just wait until it is finished.
Since this is a new VM template, select New Installation and click Next.
Select Time Zone. Select your time zone according to the location of your template. Since this will be a template for virtual servers, uncheck Hardware Clock Set to UTC and ensure the Date and Time are correct. Don’t worry about being a few minutes off, when we configure NTP later, it will correct it. When you have completed these items, click Next.
This screen tells YaST what type of Server Scenario. Our virtual host will be fully virtualized so we don’t need any special configurations. To SLES, it’s should appear that this is a physical server. Select Physical Machine and click Next.
YaST is gathering it’s base configuration and when it’s complete, we can make our custom changes.
Using this section of the installation, we can customize all parts of what we want installed and how. Click the Expert tab at the top.
First, we’ll configure our disk devices by clicking on the section, Partitioning.
SLES presents a very basic partitioning recommendation. My standard for a basic Linux server uses Linux Volume Management (LVM) for its partitioning scheme. LVM allows for dynamic growth of volumes and easier administration. Plus, creating multiple partitions allows for corralling of applications that might potentially crash the server in the event of a runaway process filling the disk. Earlier this year, I wrote an article on LVM partitioning from the command line, which is helpful should you need to add more Volume Groups and/or Logical Volumes. Here, we will use YaST’s Interface.
We will go step by step in creating the LVM partitioning scheme used.
If you haven’t selected Partitioning, do so now.
Select Custom Partitioning and click Next.
This is the first, informational screen and we can see our 50GB device. Expand the + next to Hard Disks to view the /dev/sda device.
Select the “sda” and you can see that there are no partitions present or proposed. First we’ll need to add a boot partition. Click Add on the bottom.
There are two partitions that can technically be configured using LVM. /boot and SWAP. It is recommended that these partitions be non-LVM. We’ll create them first and then configure LVM.
Select Primary Partition and click Next
Select Custom Size and enter the size for your boot partition. Here we’re specified 300 MB for the size. Click Next.
Ensure that Format Partition is selected and you can either specify your own File System or use the default of Ext3. Change the Mount Point to /boot and click Finish.
You will see the new proposed partition. I say proposed, because until we actually start the installation, nothing is written to disk. If you make a mistake, remove it and re-add it. Very flexible.
Now let’s add the SWAP partition. Click Add and select Primary. Click Next.
Select Custom Size and enter the size for the SWAP partition. There are many recommendations and arguments as to SWAP size. Some say twice the RAM, but SWAP is ineffective over 8 GB and is a waste of disk space. I have found that a max of 4GB is more than efficient and sometimes 2 SWAPs can be used in systems that need it. Luckily, a second can be added later, if needed. For this template, we are creating one SWAP and it’s size is 4GB. When completed, click Next.
Under File System, select Swap and you’ll notice that the mount point changes to swap also. Click Finish.
Now you see the two physical partitions. Now we must create one large partition for LVM. Click Add.
We want to use the remainder of the disk for LVM, so select Maximum Size and click Next.
We do not want to format this partition. We’ll do that when we create the logical volumes later. Click Do not format partition. Under File System ID, select 0x8E Linux LVM and make sure Do Not Mount Partition is checked. Click Finish.
We can see the partitions. Now we have to configure LVM. We’ll start by creating a Volume Group. Click on the Volume Manager, under System View.
There are no LVM partitions created yet. Click on Add Volume Group
In the field under Volume Group Name, enter a name for your Volume Group. This name will be associated with all Logical Volumes created within it. In this guide, we’ll use “root”. Leave the Physical Extent Size to the 4 MB default. This refers to Logical Volumes larger than 256 GB and if you do, it probably shouldn’t be in your template. Which means you would add the additional space to your deployed server, add a new Volume Group and change the Physical Extend size for that Group only.
In the frame for Available Physical Volumes, highlight the LVM partition we created and click the Add button in the center of the screen. When complete, click Finish.
You can see the root volume group listed under Volume Manager.
We will be creating six logical volumes. Although for this guide, I will only walk you through the creation of the / (root) partition. The remaining logical volumes are created using the same steps.
They are listed here for reference for this guide. Feel free to adjust sizes as your needs may vary. These have shown successful in my experiences.
/ (root) 10 GB
/opt 5 GB
/var 5 GB
/var/log 5 GB
/tmp 5 GB
/home 6 GB
Select “root” and we will begin adding Logical Volumes. Click Add.
I label out logical volumes with a “vg-“ prefix, so when we run a df or a mount command, we’ll know that they are LVM and not physical.
For the root partition, enter “vg-root” for the name and click Next.
Select Manual Size and enter 10 GB for the size. Leave striping set to 1 since we only have one disk device. Striping can occur over multiple disks, but remember that in a VM environment, the physical disk devices are probably already configured in a RAID or SAN type configuration. Click Next.
Ensure that Format Partition is selected and the File System is Ext3 or your choice. Select “/” (root) for the mount point. Click Finish.
Add any remaining Logical Volumes.
We have completed the configuration for our partitioning scheme. If you are satisfied with your settings, click Accept at the lower right. Remember that this is not written to disk, until we are ready to Install on the next screen.
You can see our changes on the main Installation Settings screen. Now we need to make a change to the Booting section. Click Booting when ready.
On virtual servers, I always remove the Floppy menu item from the GRUB menu, since I removed it from the Virtual Machine.. Highlight the Floppy line and click the Delete button and confirm. Click Ok to save.
From the Installation Settings screen, click Software.
In this section, we will define which software packages are to be installed in our template system. To make this process easier, click the Details button in the lower center.
Choose what packages you need to be on each and every server you would deploy with this template.
I normally don’t print from a Linux server so I always remove Print Server.
VMware Tools and various other third party packages require a compiler to complete their installation, so I recommend checking the box next to C/C++ Compiler and Tools
If you wish to add any addition utilities that are included in SLES, but not necessarily installed by default, you can click on the Search tab and enter the names and then select them. Here are a few I always like to have on each of my servers.
Findutils-locate – Builds a database of all files on a server and makes it easy to find a file when you can’t remember where it is.
Acct – Linux process accounting. There may be times when you need to know who, what and when.
Sysstat – a collection of system performance tools that be really help when troubleshooting.
Iftop – a utility that can help troubleshoot network traffic issues. I consider it like top, but for the network.
Supportutils – system collection that Novell will ask to be run when contacting Support. I include it here although SLES11 SP1 has it preselected for installation.
We are now done choosing which packages we want in our template. Click Accept.
Accept the license agreement and click Continue to allow YaST to include changes that were made.
From the Installation Settings screen, scroll down and click on Runlevel.
The default is set to 5, but running the GUI on the server creates unneeded overhead and is not recommended at boot up. I’ve always set the runlevel to 3, but then that is my own preference. I can still run X, if I need it, but it is not run by default. Make your selection and click OK.
We have completed the basic configuration for the installation. If you need to make any changes to anything we’ve discussed prior, please make your adjustments now. If you are satisfied with your work, Click Install in the lower right.
This is the final notification. Everything will be committed after clicking Install. Click Install and take a break.
The next couple of screens show examples of the YaST Installation in progress.
Creating partitions and format.
Final prep before initial reboot.
The next section will cover Post-boot installation configuration.
When the server reboots during installation, no intervention is required. YaST will load by default and when complete you will be presented with the following screen.
When entering the root password for your template, remember that when you deploy each server from this template, that password will need to be used and also changed as one of your deployment steps. Because it is a template, I always choose something easy so when I do a deployment, initial access is simpler and when I’m done, I change it to the company’s standard.
You can assign it to whatever you and your company dictates for server templates.
When you have entered the password, click Next and Yes to the popup complaint window.
Since this server is going to be a template, enter “sles11-sp1-server-template” for the Hostname or something similarly obvious and your applicable domain name for the Domain. Uncheck Change Hostname via DHCP and click Next.
Under Network Configuration, perform the following steps, if applicable. At this time, there have been issues relating to IPv6 with certain applications on a IPv4 network only. Also, if your company is PCI certified, disabling IPv6 can eliminate some security issues that are related on your scans. I disable the internal Firewall because this template applies to internal servers only. I also do not configure the Network Interfaces at this point because patching, updates and registration will need to apply to the deployed servers, not the template. We’ll do that when the server is complete. Click Next.
We want to skip this step, as mentioned above. Select No, Skip This Test and click Next.
This screen should be left to default. One thing to note. Check the entry “Password: [root password]”
If this entry contains “not found” or “not specified”, then click CA Management and set the root password.
Otherwise, click Next.
Although we will add all the local users needed for this template later, YaST requires at least one local non-root user to be created. We will delete this user after the configuration is completed.
Select Local and click Next.
Configure a “dummy” user and set the password to “dummy”. I always name it something obvious, so I know it isn’t a legitimate system account. Click Next and ignore any security popups.
The system will write it’s configurations.
This is the final configuration for the YaST Installation. Accept the defaults and click Next.
The installation of SLES11 is complete. Uncheck Clone This System for AutoYaST. Click Finish.
The server will initialize at the runlevel you chose and present with a login prompt.
Now that the Linux installation is complete, we have to configure all of the settings and such before we are done.
Login as root and run the following command to configure the NIC.
sles11-sp1-server-template:~ # yast lan
Using the Tab key or Alt-key combination, we want to Edit the unconfigured NIC.
Tab to Statically assigned IP Address and hit the Space bar to select. Then Tab and enter the IP Address, Subnet Mask. Leave the Hostname as the currently assigned value. Tab to Next and press Enter.
Select the Routing tab at the top and enter the Default Gateway. Tab to OK and press Enter. The system will write the settings and start the network daemon.
Ping the Default Gateway to ensure connectivity.
Now you can leave the console and SSH to the server as root for now. This will make configuring the server easier for copy and paste operations later.
VMware Tools Installation
From the VIC, right-click the template server, select Install/Upgrade VMware Tools.
Login to the server, either through the console or SSH
First we need to mount the device that contains the VMware tools rpm.
sles11-sp1-server-template:~ # mount /dev/sr0 /media/cdrom mount: block device /dev/sr0 is write protected, mounting read only sles11-sp1-server-template:~ #
Copy the file VMwareTools-3.5.0-xxxxxx.i386.rpm to /tmp/
Unmount the cdrom device. Using umount /media/cdrom
Install the rpm.
sles11-sp1-server-template:/tmp # rpm -ivh VMwareTools-3.5.0-213532.i386.rpm Preparing... ########################################## [100%] 1:VMwareTools ########################################## [100%] sles11-sp1-server-template:/tmp #
We need to configure vmtools.
sles11-sp1-server-template:/tmp # vmware-config-tools.pl Stopping VMware Tools services in the virtual machine: Guest operating system daemon: done Trying to find a suitable vmmemctl module for your running kernel. None of the pre-built vmmemctl modules for VMware Tools is suitable for your running kernel. Do you want this program to try to build the vmmemctl module for your system (you need to have a C compiler installed on your system)? [yes] Using compiler "/usr/bin/gcc". Use environment variable CC to override. What is the location of the directory of C header files that match your running kernel? [/lib/modules/126.96.36.199-0.7-pae/build/include] Extracting the sources of the vmmemctl module. Building the vmmemctl module. Using 2.6.x kernel build system. make: Entering directory `/tmp/vmware-config0/vmmemctl-only' ... The configuration of VMware Tools 3.5.0 build-213532 for Linux for this running kernel completed successfully. You must restart your X session before any mouse or graphics changes take effect. You can now run VMware Tools by invoking the following command: "/usr/bin/vmware-toolbox" during an X server session. If you wish to configure any experimental features, please run the following command: "vmware-config-tools.pl --experimental". Enjoy, --the VMware team sles11-sp1-server-template:/tmp #
That’s it. Now, onward to the next section.. User Administration.
Since this is going to be a new template, we will add all of the normal users now, that way when we spin up a new server from this template, they will already be present with their passwords. We will also configure sudo.
If you do not have a repository of Linux user accounts along with their password hash, I highly recommend it. It is not a security issue because all of these users are non-root and their passwords are already encrypted. We store the hash just as it is in /etc/shadow, which can be read by any user.
I also store the user info as a useradd command so if I happen to get a request to add a user to a system, I already have it and copy and paste the command string on that server and it’s done.
UID’s will also be concurrent on all new servers. Some people prefer to allow the system to generate the UID for the user, usually sequential and could be different on each server. If possible, use a unique number for each user that is associated to that user, Employee ID, for example. Each is unique and associated to that person/user. It’s totally up to you.
Password Hash, where to find it. If you have an existing Linux server with the users you wish to add to the template, look in /etc/shadow and copy the hash.
In my useradd string, I define the following parameters:
-u : UID to assign.
-g : initial group
-d : path to home directory
-m : create home if it does not exist
-c : comment – I put the user’s name and department.
-s : default shell – bash
-p : Password hash in single quotes.
sles11-sp1-server-template:~ # useradd -u 30091 -g users -d /home/mfaris -m -c "Michael Faris – Cool Guy -" -s /bin/bash mfaris -p '$2a$10$Uj0yN/QD7ek2dxwiYbzqJqdGBuu' sles11-sp1-server-template:~ #
I can take that string as above and copy it into a text file and use it on any server I need to add that user on. Much, much simpler. You also might want to create a default password hash for the default password you assign a new user so they can initially login and change.
To configure Sudo, edit the /etc/sudoers file to include the commands you allow on your existing servers. Remember to use visudo!
Below is a list of security-type changes I included in my first OES Migration Guide. I’m including them here for ease of reference.
These recommendations are optional and should be used as, at least, a guide to securing your server. Refer to your organization’s security policies regarding hardening your servers.
Tuning Network Kernel Parameters
There are a few parameters that can be applied to the kernel through the proc file system to improve protection of the server.
Modify /etc/sysconfig/sysctl to add these options along with the default configuration options.
accept_source_route="0" -- Disables source routing. tcp_syncookies="1" -- TCP syn flood protection parameter. tcp_max_syn_backlog="4096" -- Additional TCP syn flood protection. rp_filter="1" -- Enables anti-spoofing protection. send_redirects="0" -- Disables the sending of ICMP redirects. accept_redirects="0" -- Disables receipt of ICMP redirects.
Include this warning message for all direct methods of connection to the server.
/etc/motd Add this banner to this file
Below is an example that is used.
Change My Company to your Organization – It’s lengthy, but you know the legal guys..
In addition to setting a banner as above, it should be restricted to version 2 of the protocol only. SSH version 1 has some inherent weaknesses and so should be avoided. Edit this file and make the changes listed. Most settings are fairly self explanatory. No hosts should be automatically trusted through the rhosts types of authentication or even with a machine based certificate as with the RSA variants. Root should not be allowed direct access. For administration, you should connect to the machine as a regular user and then SUDO to root for additional needed rights.
Port 22 Protocol 2 #ListenAddress 0.0.0.0 #ListenAddress :: SyslogFacility AUTH # #LoginGraceTime 600 PermitRootLogin no #StrictModes yes RhostsAuthentication no # Don't read the user's ~/.rhosts and ~/.shosts files IgnoreRhosts yes # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts RhostsRSAAuthentication no # similar for protocol version 2 HostbasedAuthentication no PermitEmptyPasswords no AllowTCPForwarding no
Further Securing Remote Login
In addition to the restrictions made on SSH, we should also further disable remote interactive login for root in case, mistakenly or maliciously, telnet or some other method of tty access was enabled again. Modify the /etc/securetty file. All lines except the TTY1 should be commented out. This is needed for console access. SSH is running its own daemon and is not affected by these settings.
# This file contains the device names of tty lines (one per line, # without leading /dev/) on which root is allowed to login. # tty1 #tty2 #tty3 #tty4 #tty5 #tty6
Now this file should be protected by executing the following:
chown root:root /etc/securetty chmod 400 /etc/securetty
this makes it so that only root can read the file and nobody can write to it, even root, until root chmod’s the file with more permissions again.
Modification to /etc/inittab
/etc/inittab has several settings in it that should be hardened. Disable Ctrl-Alt-Delete from shutting down the server, edit the default run level, protect the server even in Single User mode, and disable extra console login daemons (Ctrl-Alt-Fx) to further protect console access.
# The default runlevel is defined here id:3:initdefault: # First script to be executed, if not booting in emergency (-b) mode si::bootwait:/etc/init.d/boot # what to do in single-user mode ls:S:wait:/etc/init.d/rc S ~~:S:wait:/sbin/sulogin # what to do when CTRL-ALT-DEL is pressed. Comment to disable. #ca::ctrlaltdel:/sbin/shutdown -r -t 4 now
The “3″ in the id:3:initdefault line designates that the default run level is level 3 which does not load the GUI. The GUI can be loaded as necessary with the “startx” command but should not remain loaded or load by default on the server.
The line beginning with “~~:S” is the command for what to do in single user mode. (i.e. typing “single” as a boot parameter in grub — which now requires password access anyway). Change the “respawn” command to “wait.” This will prompt for the root password before continuing.
The “ca::ctrlaltdel:/sbin/shutdown –r –t4 now” line is the command to execute when Ctrl-Alt-Delete is pressed. This should be commented out as shown to disable this functionality and prevent someone with physical access from shutting down the machine without a valid login.
Xwindows – GUI protections
Although X-windows is not loading by default on the server, this could be changed easily by an administrator and it is available to load manually by changing run levels or typing “startx” at the console prompt. Therefore, implement the following extra safeguards:
Remote machines should not be able to get an X terminal login window. Edit the following lines in /etc/X11/xdm/Xaccess to prepend them with a “!” as shown.
!* #NO host can get a login window !* CHOOSER BROADCAST #NO indirect host can get a chooser
Disable listening on port 6000
This prevents the X system from listening for X events from remote machines. Local X access at the console is not affected. Edit the config file /etc/X11/xdm/Xservers as shown below adding the “-nolisten tcp” switch to this line.
:0 local /usr/X11R6/bin/X :0 vt07 -nolisten tcp
Restrict cron and at
Cron and at daemons run processes on the system as root so access to them as well as the crontab command and files so that malicious code can’t be “scheduled.” The binaries are also world executeable and SUID to root so they can be dangerous. Restrict access to them with the following steps.
1. Create cron.allow and at.allow files
These files will restrict access to cron to only the users listed in the files. All others will be denied. The only user in the list should be root. These files don’t exist by default so you can create them with the echo command as follows. Delete any deny files. (/var/spool/cron/deny)
# echo root > /etc/cron.allow # echo root > /etc/at.allow
2. Modify permissions on cron/at related files
Since all cron and at files are read and written to by processes that are SUID root, normal users on the system will not ever need to have direct access to the files so they should be secured to prevent tampering.
# chown -R root:root /etc/cron* /var/spool/cron # chmod -R go-rwx /etc/cron* /var/spool/cron
Add any additional packages that would normally run on EVERY server and that does not need an individual configuration.
Once you have applied all of the steps listed above, it is then time to complete a couple of items and we can convert our server to a template. I recommend this be done from the console.
Remove the Network Card
Run YaST with the lan option and highlight the network card and tab to Delete. Tab to OK and the config will be saved.
Edit the file /etc/udev/rules.d/70-persistent-net.rules and remove all entries at the bottom.
Delete the “dummy” user that was created during the installation.
Shutdown the server using either “shutdown –h now” or “init 0”
Conversion to Template
From the VIC, right-click the server and choose Convert to Template.
If there are no errors, then your new template will be complete and ready for deployment.
Creating a template will take a bit of effort, about the same amount of time as digging out the SLES11 DVD and doing everything by hand. Of course, there was no real hardware to assemble, rack or cable. Multiple trips to the Data Center to power up, add/remove DVD’s, etc. Plus once you’ve got it all configured, locked down, and installed, all you’ve got to do is deploy a new server, give it a name and address and you’re ready to rock and roll.
Personally, I have configured more than 15 new servers in one day with all the extras and signed off on them by using a template very similar to the one described here.