SUSE Image Life Cycle for Public Cloud Deployments

Update:

Superseded by: SUSE Public Cloud Image Life-cycle

Over the last few weeks we have worked with Amazon Web Services (AWS), Google, Hewlett Packard, and Microsoft Azure teams to define a policy that allows us to treat all cloud frameworks equally from the perspective of image maintenance. Some cloud-specific notes follow at the end.

In the event of severe security vulnerabilities disclosed through the Common Vulnerabilities and Exposures process (CVE), the SUSE public cloud development team will refresh on-demand base images to ensure that new instances launch in a patched state. This is in addition to the updates available through the SUSE maintained update infrastructure. Severe vulnerabilities do not always garner great publicity, but some of them do: remember Shellshock, Poodle, and Heartbleed? However, we do not only consider the big attention grabbers for an image refresh. For example, just before Christmas 2014 a local privilege escalation issue was disclosed and fixed. The on-demand base images in all cloud frameworks where SUSE manages the images were refreshed at the time. Information about the effects of CVEs can always be found on our dedicated web page.

As we refresh images we create an ever increasing list of available images, and it is not necessarily easy to identify the latest image. We have tried to improve your ability to find the latest image by using a consistent naming scheme containing a date string in the format “vYYYYMMDD” in the image name. This has been in place for a while and has helped us from a maintenance point of view as well. The addition to the naming helps spot the latest image; however, it does not address the problem of the accumulation of older images. Therefore, it was time to put in place a practice that tackles this backlog. This is what we consider the life cycle management of images.

For image life cycle management we need to not only consider the image refresh cycle due to security updates and general refresh, but also the life cycle of the distribution. The overlay of these two creates the life cycle policy for SUSE Linux Enterprise images in the public cloud.

The standard life cycle of SUSE Linux Enterprise calls for end-to-updates and patches for a given release 6 months after a new Service Pack (SP) is released. For example, SUSE Linux Enterprise 11 Service Pack 3 was released on July 1st, 2013 and thus SUSE Linux Enterprise 11 Service Pack 2 stopped receiving updates on January 1st, 2014. For more details about the SUSE Linux Enterprise life cycle please see the documentation. At present there is no LTSS (Long Term Service Pack Support) offering available from any of public cloud vendors. Therefore, it is important that a running instance be upgraded to the next Service Pack in a timely fashion. Thus, having SUSE Linux Enterprise Server 11 SP2 images ready for launch past January 1, 2014 really is misleading to our users as it may provide the impression that the repositories for this code stream are actively maintained, when they are not.

Moving forward to a new service pack has become reasonably easy since the introduction of a supported in place upgrade mechanism in SUSE Linux Enterprise 11 SP1. The following outlines the general process. The detailed process for each cloud framework is provided in each cloud section.

1.) zypper up
2.) cd /etc/zypp/repos.d
3.) for i in *SP[n-1]*; do newname=${i/SP[n-1]/SP[n]}; cp $i $newname; done
4.) sed -i s/SP[n-1]/SP[n]/ *SP[n]*.repo
5.) sed -i s/enabled=0/enabled=1/ *SP[n]*.repo
6.) zypper refresh
7.) zypper in zypper
8.) zypper dup

where [n] indicates the target service pack number of the update. In detail, after SP4 is released the following process will work from inside a running SP3 instance:

1.) zypper up
2.) cd /etc/zypp/repos.d
3.) for i in *SP3*; do newname=${i/SP3/SP4}; cp $i $newname; done
4.) sed -i s/SP3/SP4/ *SP4*.repo
5.) sed -i s/enabled=0/enabled=1/ *SP4*.repo
6.) zypper refresh
7.) zypper in zypper
8.) zypper dup

With the life cycle for the distribution in mind we have devised the following policy for the image life cycle:

Only one version of a given distribution image is current. Images will be marked as deprecated when a new image is published with the intent of replacing a given image. A deprecated image will be removed 6 months after it has been marked as deprecated but no later than the end of the support life cycle for a given release.

Details about the deprecation are outlined below for each public cloud framework.

Lets run through some examples to clarify how this works. On December 24, 2014, updated images for SUSE Linux Enterprise 11 SP3 were published. On January 27, 2015, new images for SUSE Linux Enterprise Server 11 SP3 were published to address the so-called “Ghost” vulnerability (CVE2015-0235). Thus, the images published in December became deprecated and will be removed on July 27, 2015.

This was the simple example, now for a bit more complication let us look at a transition in the distribution. If, hypothetically, a new service pack is released on July 1, 2015, images for this new Service Pack should be published on the same day. But, lets say that for whatever reason the images do not get published until July 17, 2015. At this point the images of the previous Service Pack become deprecated, that’s the easy part. These deprecated images however, will not be removed on January 17, 2016; they will be removed on January 1, 2016, as that is the end of the life cycle of the prior Service Pack.

We realize that chasing after images is not a fun thing to do and you would definitely not do so on your off days. Changing any scripts or templates that have to have image names or image IDs hard coded in them is not fun either. However, consider the issues that come along when not making these changes and relying on outdated images when new instances are launched. First, you might run into the situation where you have to apply the upgrade procedure mentioned above from the start. That’s no fun if you just launched a hundred instances. Or, you might run into a situation where launching instances from an older image leaves your front door wide open. Neither of these scenarios is pleasant. With the 6-month deprecation period we hope to provide a reasonable time frame for any changes that need to be made to your scripts and templates. Having the published date indicator in the image name should make it reasonably easy to pick out new images in an automated way; this could also be used to automatically update your scripts and templates. This will require some work, but it is well worth it.

One important thing to note, while this is the first time the issue surrounding old images is being formalized, the issues are not new. Previously, this was just not addressed. With this official approach we set clear expectations and provide information you can count on as well as coordinate with in your own planning.

Before we take a look at the specific cloud framework details there is a perk for those running SUSE Linux Enterprise 12 images, you are done reading. The lifecycle policy described above applies to both distributions, SLES 11 and SLES 12, but none of the processes described in the following sections apply to SUSE Linux Enterprise 12 images. With that, lets dive into the cloud framework specific details.

AWS

Of all cloud frameworks where SUSE Linux Enterprise can be found, and SUSE maintains the images, AWS has the longest track record. Thus, in Amazon Elastic Compute Cloud (EC2) we also have the largest back log of now formally deprecated images.

It took us a long time to publish SLES 11 SP3 images after SLES 11 SP3 was released. Therefore, at the time we made the decision to keep SLES 11 SP2 images public for a while to ensure people had enough time to migrate. Many things happened, and “for a while” has turned into a very long time. The SLES 11 SP2 images are still public. Obviously making these images private without warning is a bit unfair, and therefore these will be set to private in 6 months as well. The SLES 11 SP2 images as well as older SLES 11 SP3 images still access an old implementation of the update infrastructure that we would very much like to get rid of.

With the deprecation policy now formalized the clock on all old images has started ticking.

Amazon EC2 facilitates the deprecation process by supporting the use of tags to indicate the status of an image. Images that get deprecated are marked with the following 3 tags:

Deprecated on

– A date string of the form “YYYYMMDD”

Removal date

– A date string of the form “YYYYMMDD”

Replacement image

– The image ID and name of the image that is considered the replacement for this deprecated image.

The applied image tags are not visible through the web UI from the instance details panel, nor are they part of the output from the “aws ec2 describe-images” command. A programmatic approach using python-boto does not reveal the tags either. Therefore, at present, the pertinent deprecation information added to the images cannot be retrieved. The latest images are always available from the quick-launch wizard and those are the current images. Programmatically you can search for the images with the largest “vYYYYMMDD” value to identify the current images, all others are deprecated.

If you are still running instances based on SUSE Linux Enterprise 11 SP2 you want to follow the upgrade procedure below to move to SUSE Linux Enterprise 11 SP3. Run ‘cat /etc/SuSE-release | grep PATCHLEVEL’ to determine whether your instance is a SP2 or SP3 instance. For SP2 instances you need to upgrade to SP3.

1.) zypper up
2.) cd /etc/zypp/repos.d
3.) for i in *SP1*; do newname=${i/SP1/SP3}; cp $i $newname; done
4.) sed -i s/SP1/SP3/ *SP3*.repo
5.) sed -i s/enabled=0/enabled=1/ *SP3*.repo
6.) zypper refresh
7.) zypper in zypper
8.) zypper dup

We break from the general procedure to avoid complications with the naming of repositories. The names between SP1 and SP3 match, but do not match for SP2. Also note that should you still be running instances based on SLES 11 SP1 you need to first upgrade to SLES 11 SP2. For SP2 the name for the “Pool” repository transitioned to “Core” and then in SP3 it became “Pool” again. Due to the way the repositories were set up in the old update infrastructure you will get some warnings during the “zypper refresh” step in the procedure above. Ignore these warnings by entering “i.” The repositories from the old infrastructure will be replaced in the next step when you switch your instance to the new update infrastructure.

-> wget http://54.197.240.216/instanceInfraUpgrade.noarch.rpm
Then as root
# zypper –non-interactive in instanceInfraUpgrade.noarch.rpm
# instanceInfraUpgrade
# rm instanceInfraUpgrade.noarch.rpm
This procedure will migrate your instance to the newer, HA update infrastructure that was put in place at the end of September 2014.

If you are running an instance based on SLES 11 SP3 that has been around for a while run:

grep smt-ec2 /etc/hosts

If this returns empty you also need to migrate your instance to the new update infrastructure. With the formalization of the deprecation policy the old update infrastructure is deprecated and it will be decomissioned in mid November 2015.

The process of migrating to the new update infrastructure will change an instance to use cloud-init, away from the deprecated SUSE-only implementation of instance initialization. Compared to new images, the configuration (/etc/cloud/cloud.cfg) is different in that root remains the default user (ec2-user is the default user for new images), the hostname and hostkeys are preserved.

For those running 32-bit instances I do have some really bad news, sorry. As all the hurdles of running 32-bit applications on 64-bit kernels have been cleared for some time a decision was made not to provide 32-bit support with the new update infrastructure. There is no advantage to running a 32-bit instance over running a 64-bit instance. Thus, come mid-November when the legacy update infrastructure will be decommissioned, you will no longer be able to get updates for 32-bit instances.

We have not refreshed 32-bit images for a while and they have all been marked as deprecated.

Google

Google Compute Engine supports a formalized deprecation mechanism which we have been using from the get go. Thus, there is nothing really new to report. Older images have been marked as deprecated since we started having SUSE Linux Enterprise images available in Google Compute Engine and we have followed the six-month removal practice, although this was not previously advertised. With the SUSE Linux Enterprise presence post dating the life cycle for SLES 11 SP2 there is also no need to upgrade any running instances.

However, within GCE we also had a version upgrade of the update infrastructure. The initial implementation of the update infrastructure will be decommissioned in mid-November as well. If you have instances that have been around for a while, you want to check whether or not you need to update those instances to point to a newer version of the update infrastructure.

1.) rpm -qa | grep regionServiceClientConfig

If the version of the package is less than version 2.1.1 us the procedure below to switch your instance to the updated update infrastructure.

1.) wget http://108.59.80.221/instanceInfraUpgrade.noarch.rpm
Then as root
# zypper –non-interactive in instanceInfraUpgrade.noarch.rpm
# instanceInfraUpgrade
# rm instanceInfraUpgrade.noarch.rpm

Hewlett-Packard

HP Helion Public Cloud is based on OpenStack and it also has a formal deprecation process through the tooling and framework. We use this process when images get refreshed. Therefore, there is nothing special to consider. SUSE Linux Enterprise Server was made available in HP Helion Public Cloud right around the time when the distribution moved to the SP3 release. Thus, no SP2 images were ever released in HP Helion Public Cloud and no instance upgrade procedure is needed.

However, the update infrastructure in HP Helion Public Cloud has changed and if you are still running instances started from images with the “Partner Image” designator in the name, you will want to update those to migrate to newer update servers by using the following procedure:

1.) wget http://15.126.238.135/instanceInfraUpgrade.noarch.rpm
Then as root
# zypper –non-interactive in instanceInfraUpgrade.noarch.rpm
# instanceInfraUpgrade
# rm instanceInfraUpgrade.noarch.rpm

Microsoft

The Microsoft Azure framework has no formal way of marking images deprecated and does not provide a tagging mechanism. Thus, we have no way of indicating which images are deprecated. We have to count on our users to remember the policy; every image except for the latest one is deprecated and will go away 6 months after the successor image has been released.

As far as long running instances are concerned, we have also updated our update infrastructure, and you should use the upgrade process already documented here.

As a quick reference:

-> wget http://104.45.147.201/instanceInfraUpgrade.noarch.rpm
Then as root
# zypper –non-interactive in instanceInfraUpgrade.noarch.rpm
# instanceInfraUpgrade
# rm instanceInfraUpgrade.noarch.rpm

The old update server will be decommissioned on June 30th, 2015. Should you still run instances in Microsoft Azure that are based on a SLES 11 SP2 image you will need to follow the distribution upgrade process outlined in the AWS section above. The procedure for Microsoft Azure is identical.

Final thoughts

We appreciate that you make SUSE Linux Enterprise Server the base image of choice for your public cloud deployments and hope that we do not create too much work on your end with the formalization of the image life cycle. We believe that the timely publishing of images with known severe vulnerabilities fixed is important. Our goal is to provide the best experience possible. From our point of view, having a reasonably simple process you can count on that addresses the issues surrounding old images is part of the experience, and therefore is important.