Amazon image management improved
Just about a year ago at SUSECon ’14 in Orlando I presented a session about the Public Cloud, more to come at SUSECon ’15 in Amsterdam in just a few weeks, of course . During the session last year I spoke about images for the different frameworks and how to build them with KIWI. Of course once an image is built it needs to be uploaded to the framework and used, or it is just a file sitting on your hard drive taking up space and collecting dust.
The process of creating an EBS backed AMI takes a number of steps and is not necessarily something one wants to repeat manually many times over. This process is now nicely wrapped up in a new utility called ec2uploadimg. Along with ec2deprecateimg and ec2publishimg we have 3 new utilities that are available in the SUSE Linux Enterprise 12 Public Cloud module delivered with the python-ec2uploadimg, python-ec2deprecateimg, and python-ec2publishimg packages, respectively. Lets take a look at ways these new utilities can help with image management.
All three utilities use a common configuration file, by default ~/.ec2utils.conf, in INI format. The code is developed on GitHub in the SUSE Public Cloud team Enceladus project. An example configuration file is part of the GitHub repository. The configuration may contain 2 different types of sections, account sections and region sections. One can configure as many accounts and regions as desired. The example configuration file has information for each region in Amazon EC2 pre-populated and the region sections from the example configuration file can be used verbatim. Account sections start with the “account-” prefix and region sections start with the “region-” prefix. For example the following section configures an account named “servers”.
Using this account to access EC2 would be accomplished by specifying the “–account servers” command line option on any of the utilities. An account section configures the EC2 API access key and secret access key with the “access_key_id” and “secret_access_key” options, respectively. If you are uncomfortable keeping these keys in an un-encrypted file on your system you do not have to add the entries to the account section of the configuration file. Both values can be specified on the command line for any of the utilities with the “–access-id” and “–secret-key” command line options. The “ssh_key_name” option in the account section specifies the name of the ssh key as it is known in Amazon EC2. The “ssh_private_key” option specifies the path to the private SSH key. The SSH key information is only needed for image uploading.
A region section is specified as follows, using the configuration for the ap-southeast-2 region as an example:
ami = ami-3d6cfc07
instance_type = m1.small
aki_i386 = aki-cd62fff7
aki_x86_64 = aki-c362fff9
g2_aki_x86_64 = aki-8faec3b5
user = ec2-user
# Allow a region to overwrite the account key
# ssk_key_name =
# ssh_private_key =
Rather obviously the section name specifies the Amazon EC2 region for which the following information is to be used. The “–regions” command line option on any of the utilities specifies the region(s) on which the utility should operate. The information provided by each region section is currently only needed by the image upload utility. The “ami” option specifies the AMI-id of an image to launch to function as a helper to create a new EBS backed AMI from the raw image you will already have at hand. This can be pretty much any available image, but a bit more on this later when we cover the upload utility in more detail. The “instance_type” option specifies the instance type to use for our helper instance. The “aki_i386”, “aki_x86_64”, and “g2_aki_x86_64” options specify the Amazon Kernel Image IDs for para virtual (PV) images for x86, x86_64, and x86_64 with GRUB2 images, respectively. These are only needed if you are creating PV images. The “user” option specifies the user name to be used to log into the running helper instance. Last but not least the “ssh_key_name” and “ssh_private_key” options can be used to overwrite the information provided in the account section. Thus you can have different ssh keys for each region.
With the configuration file covered lets take a look at the individual utilities. Each utility will print command line help with the “–help” command line option and the man pages provide more detailed information.
The ec2publishimg command allows you to control the visibility of an image you own from your account. You can make an image public, such that it can be found in the Amazon EC2 general catalog, share an image with one or more specific accounts, or make an image private.
ec2publishimg –account servers –image-name-frag test-server-3 –regions sa-east-1 –share-with 123456789
The command above would share an image that contains “test-server-3” in the image name in the “servers” account in the “sa-east-1” region with the Amazon EC2 account “123456789” in that region. Sharing with multiple accounts is possible by specifying a comma separated list as the value for the “–share-with” option or by running the command once per account with which the image is to be shared. Making the image public is achieved by using the “all” keyword as the value for “–shared-with” and making the image private is accomplished by using the “none” keyword. The utility offers multiple ways to select the image to be shared, “–image-id” allows you to specify the AMI-id of the image to share, “–image-name” is used to find an image that matches the given name exactly and “–image-name-match” allows you to specify a regular expression to match the image name. If the “–regions” option is not specified the utility will work on all connected regions.
A while ago we introduced the image life-cycle policy . This utility implements the tagging of images as described by the life-cycle policy. While these tags are currently not visible to users outside the account that owns the image the tags can still help you to manage your images. The utility will create the tags “Deprecated on”, with the value of the date on which the utility was run in the format YYYYMMDD, “Removal date” with the value of deprecation date plus the specified deprecation period in the same format, and “Replacement image” with the value of the AMI-id of the replacement image. The deprecation period is 6 month by default and is specified with the “–deprecation-period” command line option, the value specified is interpreted as months.
ec2deprecateimg –account servers –image-name-match v15 –image-virt-type hvm –replacement-name exampleimage_v16
The command above would tag all HVM images in all connected regions in the account “servers” that contain “v15” in the image name. The deprecation period would be 6 months and the value of the “Replacement image” tag would be set to the AMI-id of the image with the name “exampleimage_v16” in the region that is being processed. As with the ec2publishimg utility multiple selection options exist for the image to be tagged and the replacement image. For operating in a specific region or multiple selected regions use the “–regions” command line option.
Creating an EBS backed AMI involves a number of steps and the use of a helper instance. The utility expects a raw image file in a known compression format or a tarball that contains a raw image file (indicated by .raw in the filename) as an argument. Basically file formats produced by KIWI running in the openSUSE Build Service, SUSE Studio, or locally on your machine.
ec2uploadimg –account servers –backing-store ssd –description “my test server” –grub2 –machine x86_64 –name test_server_2 –sriov-support –virt-type hvm –verbose –regions eu-central-1 testServer.raw.xz
This command will create a new HVM EBS backed AMI in the “eu-central-1” region using SSD as the backing store for the image. The AMI will be named “test_server_2”. The image uses GRUB2 as the bootloader (–grub2) and supports SRIOV (–sriov-support). By default the command is quiet and “–verbose” will provide information about the progress of the image creation process.
The ec2uploadimg utility has another trick up it’s sleeve, it can also be used to create SUSE Linux Enterprise Server on demand images. Once you create an on demand image with the upload utility the image has the necessary data to hook into the SUSE operated update infrastructure in Amazon EC2. In order to connect to the update infrastructure the following packages, available from the SLE 12 Public Cloud Module and the SUSE Linux Enterprise Server 11 Public Cloud Channel need to be installed in the image:
Additionally the “guestregister” service must be enabled to run at boot time. Once an image has been enabled to connect to the SUSE operated update infrastructure by installing the above packages all that is left to do is to create an AMI that is an on demand image. For this to work the helper AMI launched for the image creation MUST be a SUSE Linux Enterprise Server on demand of matching distribution, i.e. a SLES 11 SP4 helper instance to create a SLES 11 SP4 on demand image. Obtaining the AMI-ids of the currently active images can be accomplished via the EC2 web console or a browser session using the following URL:
more about the susepubliccloudinfo service in another blog. Anyway, with the AMI-id in hand you can modify the config file or provide it on the command line. Lets say we have a SUSE Linux Enterprise Server 12 based image intended for region us-west-2 that is supposed to be an on demand image to take advantage of the SUSE operated update infrastructure.
ec2uploadimg –account servers –backing-store ssd –description “my SLE on demand mail server” –grub2 –machine x86_64 –name mail_server –sriov-support –virt-type hvm –verbose –use-root-swap –ec2-ami ami-d7450be7 –regions us-west-2 mail_server_image.raw.xz
Rather than modify the configuration file I opted to use the “–ec2-ami” command line option to specify the AMI-id of the helper image to launch. All other values will be read from the configuration file. The “–use-root-swap” option tells the utility to swap the root device of the running instance with my new image and thus transfer all the metadata to my new image, therefore turning it into an on demand image. For information about on demand images as compared to BYOS (Bring Your Own Subscription) images see my previous blog. The creation of on demand images is also useful for those looking to offer integrated solutions with SUSE Linux Enterprise in the Amazon Marketplace.
Hopefully the new utilities will be useful to help you manage your Amazon EC2 images. For those using SLES 11 SP3 or SP4 the packages are available in the openSUSE Build Service in the Cloud:Tools project.