Search results: “cloud-regionsrv-client”

Our update infrastructure that provides updates for on-demand instances has been running with more or less no changes for more than 5 years and has shown great reliability over this period of time. Over time new requirements have arisen and some bugs have been worked around due to some fundamental limitations of the implementation. The time has come for a change and a wholesale upgrade to something new and shiny.

What’s happening with our update infrastructure?

Let me start with the things that stay the same. As with the previous incarnation there is an HA setup, this has been improved and more on this later. Updates will remain region local for maximum performance.

The major improvements being implemented with the new update infrastructure are:

1. Consolidation of update servers to serve all products from the same set of servers
2. Improved HA setup
3. Support of traffic routing through the DC

We worked really hard to avoid any hiccups, but didn’t quite manage, more on this later. For running instances there is no service disruption during the transition, “zypper up” works.

What are the practical implications for you?

1. Consolidation of update servers to serve all products from the same set of servers

In the previous implementation we had to run update servers on a per product basis. Which resulted in interesting setup challenges in network constructs that by default denied egress to the Internet at large. As a user one had to either have a subnet that had a routing table that allowed access to all update servers for all products or one had to segregate subnets and run instances with SLES For SAP in one subnet and instances with SLES in another subnet to get to a minimum set of IP addresses to allow egress to. As we begin to roll out SLES For HPC as a product one would have had to have yet another subnet or open egress to the outside world to more IP addresses. With the new update server technology this problem is resolved! Meaning all update servers are capable of serving updates for all products and do proper authentication such that registration does not cross the channels between the various products. Practically this means there is/will be 1 less IP address that has to be white-listed for egress. The new update infrastructure consists of at least 3 update servers per region, which leads me to the next topic.

2. Improved HA setup

While we have not experienced a total outage of the update infrastructure as a whole or in any given region, there was always the nagging monkey on our back that we only had 2 systems in an HA configuration, i.e. minimal redundancy. With the new update servers, we have 3 servers in the HA configuration providing greater resiliency. The servers are spread across availability zones wherever possible to isolate us from zone based issues. With the consolidation of all products onto one set of servers we reduce the total number of systems we operate – yay for us; and at the same time improve our resilience. Less is more.

3. Support of traffic routing through the DC

With the new update infrastructure, we are in a position to eventually allow traffic that flows from your Public Cloud Network construct through your data center and then back to our update infrastructure. This type of traffic flow is not supportable with the previous technology and we know this is a concern for many of our users. Supporting this data flow comes with a data verification caveat that has a side effect in AWS and I’ll get to this in a minute. Due to this side effect we will not support the Cloud -> DC -> Cloud data flow immediately.

Update 2019-12-16

The cut-over date and transition period has been set. Details can be found in Step 2 Toward Enhanced Update Infrastructure Access

End Update 2019-12-16

The “Grace Period”/Delay Of Cloud -> DC -> Cloud Data-flow

In order to support the Cloud -> DC -> Cloud traffic flow we needed a reliable way to look for the marker that makes SLES and SLES For SAP on-demand instances exactly that, on-demand instances. This implies that we need to look for this marker every time a system comes to visit the update servers. This checking process has two components, one server side and one client side (your instances). The client-side changes are in version 8.1.2 or later of the cloud-regionsrv-client package, and the server side changes are in the new update infrastructure implementation. I know what you are thinking, both are available lets go. Well that’s where the caveat comes into play.

In AWS EC2 a condition existed where it was possible to accidentally lose the marker that identifies an instance as a SUSE Linux Enterprise Server on-demand instance. If we would enable the Cloud -> DC -> Cloud traffic flow immediately all those instances that are in this condition would lose access to the update infrastructure immediately – Not Good. Therefore, there is a transition period, exact dates are to be determined, that will allow those that lost the marker to re-initiate their instances to get the marker back. Another blog will follow on this topic soon. Once the end of the transition period has been reached there will be an announcement specific to the Cloud -> DC -> Cloud traffic flow.

Update 2019-12-16

The cut-over date and transition period has been set. Details can be found in Step 2 Toward Enhanced Update Infrastructure Access. Also, while the code in version 8.1.2, as stated above, in cloud-regionsrv-client provides the necessary bits additional enhancements have been made to address package update concerns and repository duplication we saw after updates to the package along certain paths. Therefore it is recommended to pull the latest version of the package.

zypper up cloud-regionsrv-client

End Update 2019-12-16

A Hiccup and a Caveat related to the transition

As indicated earlier we didn’t quite manage to avoid all potential hiccups. There is a registration issue with SUSE Linux Enterprise Server For SAP instances created from the AWS Marketplace images with a date stamp prior to 20181212. Marketplace images with a date stamp prior to 20181212 have a bug. This bug is immaterial in the previous incarnation of the update infrastructure but rears it’s ugly head with the new update infrastructure. The bug has been fixed for a while, but fixed images never made their way into the Marketplace. The images that are currently on their way to the AWS Marketplace address this issue and also contain fixes for SACK and MDS. We are working very closely with AWS to get these out into the Marketplace as quickly as possible.

The good news is that despite the automatic registration failing there is a pretty easy fix.

First check whether or not your instance is affected

zypper lr

if this doesn’t produce any repositories and you launched a SLES For SAP instance from AWS Marketplace run the following commands as root

cd /etc/products.d
rm baseproduct
ln -s ln -s SLES_SAP.prod baseproduct
systemctl start guestregister.service

After this “zypper lr” is expected to list the repositories as you would expect it if all were as it is supposed to be.

Before moving on, a quick explanation of the bug and what just happened. With SUSE Linux Enterprise 15 inter module dependencies are supported. This means one module may depend on another. Naturally dependencies require ordering. Therefore, modules have to be registered in the expected order. The new update infrastructure enforces the proper module registration order while the old update infrastructure basically accepted registration in any order and thus other weird issues could arise. The bug in the images with date stamps prior to 20181212 is that the so called “baseproduct”, indicated by the “baseproduct” link is pointing to the incorrect product definition file and this breaks the registration order. The above set of commands fixes the problem and allows registration to take place as expected.

Once the new images are in the AWS Marketplace the issue will simply go away.

An expected but previously not communicated side effect of the update infrastructure update is that it is no longer possible to register newly created SLES 11 SPx instances. The new update infrastructure servers do not support the SLES 11 registration protocol. Following our life-cycle and the general life cycle of the product the SUSE Linux Enterprise Server 11 release series reached the end of general support on March 31st, 2019. This implies that on-demand images for SUSE Linux Enterprise Server 11 (any service pack) are no longer supported to be launched and over time all images will disappear. LTSS is available for BYOS instances or via SUSE Manager for on-demand instances.

The new update infrastructure servers do have the SLES 11 repositories and therefore no apparent service interruption to running instances is occurring. However, the SLES 11 repositories no longer receive any updates and therefore the connection to the update infrastructure as such does not deliver anything new anymore. It is really time to upgrade to a newer version of SLES. For the transition to SLES 12 we have a documented, unfortunately, tedious process. This is expected to work for the most part but you have to be careful and you have to know what you are doing. Major distribution upgrade gets easier from SLES 12 SP4 to SLES 15. The new process is fully supported, while the SLES 11 to SLES 12 is a more “do at your own risk” process. By the end of this year, 2019, the SLES 11 repositories will disappear from the update infrastructure.

The transition to the new update infrastructure is in full swing. Therefore the natural questions are, how does one know if a particular region has already been transitioned, and when is this going to be done. The answer to the first question can be obtained using “pint”

pint $PROVIDER servers

produces a list of all the update servers we run in AWS, Azure, and GCE. If a region has 3 entries then that region has been switched to the new update infrastructure. For example:

pint amazon servers
….
<server ip=”52.66.49.238″ name=”smt-ec2.susecloud.net” region=”ap-south-1″ type=”smt-sles”/>
<server ip=”52.66.45.16″ name=”smt-ec2.susecloud.net” region=”ap-south-1″ type=”smt-sles”/>
<server ip=”52.66.51.63″ name=”smt-ec2.susecloud.net” region=”ap-south-1″ type=”smt-sles”/>
….

There are 3 servers all designated as “smt-sles” and therefore the new update infrastructure is up and running in the “ap-south-1” region.

….
<server ip=”54.223.131.108″ name=”smt-ec2.susecloud.net” region=”cn-north-1″ type=”smt-sles”/>
<server ip=”54.223.140.138″ name=”smt-ec2.susecloud.net” region=”cn-north-1″ type=”smt-sles”/>
….

There are only 2 servers in the “cn-north-1” region designated as “smt-sles” and therefore this region has not yet transitioned. Similar for other frameworks

pint google servers
….
<server ip=”34.65.167.82″ name=”smt-gce.susecloud.net” region=”europe-west6″ type=”smt-sles”/>
<server ip=”34.65.120.183″ name=”smt-gce.susecloud.net” region=”europe-west6″ type=”smt-sles”/>
<server ip=”34.65.187.174″ name=”smt-gce.susecloud.net” region=”europe-west6″ type=”smt-sles”/>
….

pint microsoft servers
….
<server ip=”102.133.128.124″ name=”smt-azure.susecloud.net” region=”southafricanorth” type=”smt-sles”/>
<server ip=”102.133.128.67″ name=”smt-azure.susecloud.net” region=”southafricanorth” type=”smt-sles”/>
<server ip=”102.133.129.51″ name=”smt-azure.susecloud.net” region=”southafricanorth” type=”smt-sles”/>
…

You will eventually also see a change in the “type” designation in the data. With the split infrastructure we had to introduce the “smt-sles” and “smt-sap” designation. These will go away and type will simply be “smt“.

In Summary

We almost pulled it off. We are going through the switch with no downtime in update server availability, but we stumbled over an issue that is not completely in our control. The SLES 11 caveat should have been pre-announced. We apologize for the inconvenience caused.

Release Candidate Phase

SUSE Linux Enterprise 15 Service Pack 1 is entering the Release Candidate Phase since we are closer and closer to the official release date (so called First Customer Shipment) scheduled for mid-June 2019.

SUSE Engineering Team has a special mindset regarding the Release Candidate phase, because we are targeting to have FCS quality with those milestones. We have the constraints of considering RC milestones like FCS releases, in term of bug fixing and providing security fixes, which means that we are focusing on stability but there is also less room for intrusive changes.

Of course exception happens, and developing, stabilizing, building and testing an entire Operating System and products around it is a complex task. Nevertheless with the help of our talented engineers, powerful tool like Open Build Service and openQA AND the ability for external parties (you!) to participate early and during the development phase, we will be able to deliver the best SUSE Linux Enterprise version yet!

Notes for RC 1

Packages Added

• buildah
• cni

Packages Changes

• Tons of security and bug fixes,
• The usual bug fixes for the Linux Kernel and YaST
• amavisd-new updated to 2.11.1
• ceph updated to 14.1.0-559-gf1a72cff25
• cloud-netconfig-azure and cloud-netconfig-ec2 updated to 0.9
• cloud-regionsrv-client updated to 8.1.3
• containerd updated to v1.2.2
• docker updated to 18.09.1-ce.
• google-compute-engine updated to 20190124
• java-1_8_0-openjdk updated to jdk8u201 (icedtea 3.11.0)
• mariadb updated to 10.2.22 GA
• openvswitch updated to 2.11.0
• rmt-server updated to 1.2.2
• supportutils updated to 3.1.1
• samba updated to 4.9.4

 

As always, we highly recommend to check our Release Notes and our changelogs files for a complete overview of the changes in this new version.

SLE-BETA Public Web Page

Please visit https://www.suse.com/betaprogram/sle-beta for more information like how to download, how to report issues, how to start a discussion with SLE Engineering and more.

 

Another exciting news, we have also released SUSE Enterprise Storage 6 Beta 11, which is based on SUSE Linux Enterprise Server 15 Service Pack 1 RC 1!

SUSE Enterprise Storage 6 Major Themes

• Built on the latest
  • upstream Ceph release: Nautilus. Targeted for upstream availability in February 2019
  • Updated to run on SUSE Linux Enterprise Server 15 SP1 (Beta)
• Manageability
  • Ceph Dashboard (Replacing openATTIC)
    • Check out this openATTIC and http://docs.ceph.com/docs/master/mgr/dashboard/ for more information!
    • Ceph Dashboard localized
    • Ceph Dashboard Single Sign-On (SAMLv2)
  • Automatic metric reporting phase 1
  • CephFS directory quotas
• Interoperability
  • IPv6
  • TECH-PREVIEW: RDMA back-end
• Availability
  • Sync to external cloud via S3
  • CephFS snapshots
  • TECH-PREVIEW: Asynchronous file replication
• Efficienty
  • QoS for RDB
  • Background operation QoS

 

SUSE Enterprise Storage 6 Beta Webinar

If you have missed the SUSE Enterprise Storage 6 Beta Webinar, you can now watch it on Brighttalk.

The agenda was:
– Welcome
– SES 6 Schedule
– Feature Overview
– Demo Installation
– Question & Answers

Notes for Beta 11

Updated Packages

• ceph 14.1.0-559-gf1a72cff25
• ceph-iscsi 3.0+1552304123.g67b0d30
• deepsea 0.9.15
• openssl-1_0_0
• samba  4.9.4
• ses-manual_en

 

Removed Packages

• golang-github-prometheus-node_exporter
• python-netaddr
• python-oauthlib
• python-PyJWT
• python-requests-oauthlib
• python-rsa
• salt (but now using the one provided by SLES)

 

As always, we highly recommend to check our Release Notes and our changelogs files for a complete overview of the changes in this new version.

STORAGE-BETA Public Web Page

Please visit https://www.suse.com/betaprogram/storage-beta for more information like how to download, how to report issues, how to start a discussion with SLE Engineering and more.

 

Have fun beta testing,

Your SUSE Linux Enterprise and SUSE Enterprise Storage Team.

Use cases exist where it may not be desirable to start a deployment “from scratch” with a newly deployed on-demand instance from a SUSE released SUSE Linux Enterprise Server image. However, if you are on the fence and your primary concern is uncertainty about achieving an equivalent system when starting from scratch then Machinery may be able to help.  With Machinery you can also compare two systems and thus moving from point A to point B, where point B is an equivalent system is not as painful as it might appear.

In any event, if the “equivalent system” route is not for you and you want an on-demand image, there is a solution. The background processes in such cases may include the conversion of an existing Virtual Machine (VM) or the transformation of a physical system into a VM.

For the rest of this article I will assume that you have arrived at a suitable “disk.raw” file, using information in the Google Documentation , that could be booted in GCE. The second underlying assumption is that the OS in the image is of a SUSE Linux Enterprise Server version that is considered supported in the Public Cloud.

While SLES 11 SP4 is still supported, i.e. SUSE releases and maintains a SLES 11 SP4 image in GCE, the described processes are not suitable for SLES 11 SP4 as not all necessary packages are available for download. For SUSE Linux Enterprise Server For SAP Applications you want to have at least a version based on 12 SP2.

You can go about creating your on-demand images in a couple of ways, both are more or less involved depending on your point of view. In one approach the preparation of the disk.raw file is minimal and no access to the SUSE Customer Center (SCC) is required, in the other approach there is a bit more prep work to the disk.raw file and access to SCC is required, but less work is required once the image is uploaded to GCE.

Some background

The fundamental difference between a Bring Your Own Subscription (BYOS) and an on-demand image is a metadata entry associated with an image that references license information. This license information triggers the platform (GCE framework) to track an instance per usage hour in a way that the SUSE update infrastructure in GCE can verify the instance and thus grant access to the repositories.

There are a couple of ways to add a license to an image. One is to use the API which takes the URI for the license you want to attach, and the other is to use the “gcloud” command line tool.

Here is an example of a command line to attach the sles-12 license to an image:

gcloud compute images create YOUR_IMAGE –family FAMILY \

–source-uri gs://BUCKET/path/image.tar.gz \

–licenses=”https://www.googleapis.com/compute/v1/projects/suse-cloud/global/licenses/sles-12″

The key here is the URI for the license. All the SUSE licenses for SLES are in the suse-cloud project and licenses for SLES For SAP are in the suse-sap-cloud project. The license names for SLES are sles-12 and sles-15 when SUSE Linux Enterprise Server version 15 is released later this year. For SLES For SAP the names are sles-sap-12 and sles-sap-15 when SLES 15 For SAP gets released.

With this the URI for SLES 15 For SAP would be

https://www.googleapis.com/compute/v1/projects/suse-sap-cloud/global/licenses/sles-sap-15

Once an image is created with the license information attached it can be verified by the SUSE update infrastructure as a SLES or SLES For SAP instance and access to the update repositories will be granted. But the image needs a “little” more than just have the license attached.

Option 1: The lesser image prep approach

With the information above you already know how to create an on-demand image. The next step is to get the benefits of the on-demand image, i.e. registration with the SUSE operated update infrastructure in GCE. Once you have your custom image tarball uploaded and you have created an image with the license attached start and instance and login. You will need root access.

In this approach we spent no time to prepare the image (disk.raw) to be an on-demand source for on-demand instances. Therefore we need to fiddle with the running instance and then create a new image-from it to avoid having to do the same fiddling for every instance started from the original image.

First you want o gather information from pint about the update infrastructure in the region you are running. You can install pint from source or from the openSUSE Build Service, python3 is needed.

Lets say you are running in us-east1 and thus you’d run the following command:

pint google servers –region us-east1

and that will produce the following output:

<?xml version=’1.0′ encoding=’UTF-8′?>
<servers>
<server ip=”104.196.61.109″ name=”smt-gce.susecloud.net” region=”us-east1″ type=”smt-sles”/>
<server ip=”104.196.26.155″ name=”smt-gce.susecloud.net” region=”us-east1″ type=”smt-sles”/>
<server ip=”104.196.220.87″ name=”smt-gce.susecloud.net” region=”us-east1″ type=”smt-sap”/>
<server ip=”35.185.34.152″ name=”smt-gce.susecloud.net” region=”us-east1″ type=”smt-sap”/>
</servers>

If you are running SLES you will want to point your instance to one of the two servers with ‘type=”smt-sles”‘ and if it is SLES For SAP, stating the obvious, you want to point to one of the servers with ‘type=”smt-sap”‘.

The next step is to get the certificate from the SMT server. The update infrastructure uses self signed certificates and you need to install the cert into the chain of trust on your instance. First lets add the host to your hosts file, as the hosts are not resolvable via DNS. Assuming I have a SLES instance and continuing under the assumption we are in us-east1 the following example will do the trick:

echo “# Added by SMT registration do not remove, retain comment as well” >> /etc/hosts
echo “104.196.61.109 smt-gce.susecloud.net smt-gce” >> /etc/hosts

The IP address above is based on your choice from the pint output. The comment is needed for later when we tie everything together to be fully automagic.

With the host entry added:

wget http://smt-gce.susecloud.net/smt.crt

The next step is to add the cert to the trust chain.

cp smt.crt /usr/share/pki/trust/anchors
update-ca-certificates

Next you need the gcemetadata tool. You can install this from source or from OBS.

Once gcemetadata is installed and operational, test simply by running ‘gcemetadata‘, you need to collect the proper metadata to make authentication with the update infrastructure work. this happens as follows:

gcemetadata –query instance –identity http://smt-gce.susecloud.net –identity-format full –xml >& instance_md.xml

With this you are ready to register your system.

SUSEConnect –url https://smt-gce.susecloud.net –instance-data instance_md.xml

Next you need to add the Public Cloud Module repository explicitly as follows:

SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-public-cloud/12/x86_64 –instance-data instance_md.xml

The next steps are to make the instance look more like an instance started from a SUSE released image and include registration automation.

On SLES 12:

zypper in google-compute-engine-init cloud-regionsrv-client cloud-regionsrv-client-plugin-gce regionServiceClientConfigGCE regionServiceCertsGCE

On SLES 12 For SAP:

zypper in google-compute-engine-init cloud-regionsrv-client cloud-regionsrv-client-plugin-gce regionServiceClientConfigSAPGCE regionServiceCertsSAPGCE

If you want other SLES 12 module repositories you need to register each of them:

SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-adv-systems-management/12/x86_64 –instance-data instance_md.xml
SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-containers/12/x86_64 –instance-data instance_md.xml
SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-hpc/12/x86_64 –instance-data instance_md.xml
SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-legacy/12/x86_64 –instance-data instance_md.xml
SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-toolchain/12/x86_64 –instance-data instance_md.xml
SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-web-scripting/12/x86_64 –instance-data instance_md.xml

This works on SLES 12 and SLES 12 For SAP.

The next step is to enable the necessary services.

On SLES 12 and SLES 12 For SAP:

systemctl enable google-accounts-daemon.service
systemctl enable google-clock-skew-daemon.service
systemctl enable google-instance-setup.service
systemctl enable google-ip-forwarding-daemon.service
systemctl enable google-shutdown-scripts.service
systemctl enable google-startup-scripts.service
systemctl enable guestregister.service
systemctl enable haveged.service

If you want root disk resize to work you need to install the growpart-rootgrow package. At the time of this writing the package has not been released in the Public Cloud Module, thus you can get the build from OBS.

Once installed enable the service the package provides

systemctl enable rootgrow.service

With that we are almost there. Now it is time to clean up and then create a new image from the running instance.

rm -rf /etc/zypp/{repos.d,credentials.d,services.d}/*
rm -rf /etc/SUSEConnect

The last step is to create a new image from the running instance, please refer to the Google Cloud Platform documentation for details.

Option 2: More prep less fiddling once uploaded

Lets assume that you have a running SLES system that you want to convert into an on-demand image for GCE. The system is already registered against SCC and you have access to all the packages. In this process you just install the necessary packages, clean up, and then create disk.raw and the tarball. It is reasonably similar to the previous approach and therefore I will be a little less explicit.

As in the example above you need the Public Cloud Module enabled on your running system but the command is a bit shorter since the system is already registered against SCC.

SUSEConnect -p sle-module-public-cloud/12/x86_64

Then install the packages:

zypper in google-compute-engine-init cloud-regionsrv-client cloud-regionsrv-client-plugin-gce regionServiceClientConfigGCE regionServiceCertsGCE

On SLES 12 For SAP:

zypper in google-compute-engine-init cloud-regionsrv-client cloud-regionsrv-client-plugin-gce regionServiceClientConfigSAPGCE regionServiceCertsSAPGCE

If you want all the SLES 12 module repositories enabled by the registration automation you will want to install the following packages:

SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-adv-systems-management/12/x86_64 –instance-data instance_md.xml
SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-containers/12/x86_64 –instance-data instance_md.xml
SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-hpc/12/x86_64 –instance-data instance_md.xml
SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-legacy/12/x86_64 –instance-data instance_md.xml
SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-toolchain/12/x86_64 –instance-data instance_md.xml
SUSEConnect –url https://smt-gce.susecloud.net -p sle-module-web-scripting/12/x86_64 –instance-data instance_md.xml

The registration of these repos installs the -release packages for each module which triggers the automatic set up for these repos in GCE.

This works on SLES 12 and SLES 12 For SAP.

Enable the services

systemctl enable google-accounts-daemon.service
systemctl enable google-clock-skew-daemon.service
systemctl enable google-instance-setup.service
systemctl enable google-ip-forwarding-daemon.service
systemctl enable google-shutdown-scripts.service
systemctl enable google-startup-scripts.service
systemctl enable guestregister.service
systemctl enable haveged.service

As in the previous section if you want root disk resize to work you need growpart-rootgrow.

If your running system is a VM and you have any special initialization code that runs on boot you need to disable that. Next is the clean up part:

rm -rf /etc/zypp/{repos.d,credentials.d,services.d}/*
rm -rf /etc/SUSEConnect

Once this step is complete create the disk.raw file by which ever process you prefer. Once you have the disk.raw file create the tarball:

tar –format gnu -cSzf my_custom_image.tar.gz disk.raw

Create an image from the tarball, don’t forget to attach the license, and you should be off to the races.

Last but not least you will want to login to the SUSE Customer Center and remove the system you just moved from your registrations since it is now an on-demand instance.

The proceses above will upon instance start up configure an instance to meet the GCE “default” settings, i.e. ssh login and user creation. If this is not desired do not enable the google-accounts-daemon.service.

Thanks to Google and Velostrata for feedback on the draft and for testing.

Starting Friday March 25, 2016 Azure will enter a 2 day rolling maintenance period that might effect the availability of updates, i.e. “zypper up” may fail, for SUSE Linux Enterprise Server 12 and 11 (and all service packs) on demand running instances and may effect the automatic registration of newly launched on demand instances to the SUSE operated update infrastructure.

During the maintenance window it is expected that systems (our update servers) are shut down no longer than 15 minutes. Our update servers are placed in each region in a fashion that they should not all be shut down at the same time. However, we have no control over this process.

If you have version 6.4.4 or later of the cloud-regionsrv-client package installed,

rpm -qa cloud-regionsrv-client

your instance will automatically find an available update server in the region, if there is one.

The maintenance window schedule is as follows:

• Australia East
  • 23:00 Friday, Mar 25–11:00 Saturday, Mar 26 (UTC+11)
  • 12:00 Friday, Mar 25–00:00 Saturday, Mar 26
• Japan West
  • 23:00 Friday, Mar 25–11:00 Saturday, Mar 26 (UTC+9)
  • 14:00 Friday, Mar 25–02:00 Saturday, Mar 26
• East Asia
  • 23:00 Friday, Mar 25–11:00 Saturday, Mar 26 (UTC+8)
  • 15:00 Friday, Mar 25–03:00 Saturday, Mar 26
• West Europe
  • 23:00 Friday, Mar 25–11:00 Saturday, Mar 26 (UTC+1)
  • 22:00 Friday, Mar 25–10:00 Saturday, Mar 26
• West US
  • 23:00 Friday, Mar 25–11:00 Saturday, Mar 26 (UTC-7)
  • 06:00 Saturday, Mar 26–18:00 Saturday, Mar 26
• Central US
  • 23:00 Friday, Mar 25–11:00 Saturday, Mar 26 (UTC-5)
  • 04:00 Saturday, Mar 26–16:00 Saturday, Mar 26
• South Central US
  • 23:00 Friday, Mar 25–11:00 Saturday, Mar 26 (UTC-5)
  • 04:00 Saturday, Mar 26–16:00 Saturday, Mar 26
• South India
  • 23:00 Friday, Mar 25–11:00 Saturday, Mar 26 (UTC+5:30)
  • 17:30 Friday, Mar 25–05:30 Saturday, Mar 26
• Australia Southeast
  • 23:00 Saturday, Mar 26–11:00 Sunday, Mar 27 (UTC+11)
  • 12:00 Saturday, Mar 26–00:00 Sunday, Mar 27
• Japan East
  • 23:00 Saturday, Mar 26–11:00 Sunday, Mar 27 (UTC+9)
  • 14:00 Saturday, Mar 26–02:00 Sunday, Mar 27
• South East Asia
  • 23:00 Saturday, Mar 26–11:00 Sunday, Mar 27 (UTC+8)
  • 15:00 Saturday, Mar 26–03:00 Sunday, Mar 27
• North Europe
  • 23:00 Saturday, Mar 26–11:00 Sunday, Mar 27 (UTC)
  • 23:00 Saturday, Mar 26–11:00 Sunday, Mar 27
• East US 2
  • 23:00 Saturday, Mar 26–11:00 Sunday, Mar 27 (UTC-4)
  • 03:00 Sunday, Mar 27–15:00 Sunday, Mar 27
• North Central US
  • 23:00 Saturday, Mar 26–11:00 Sunday, Mar 27 (UTC-5)
  • 04:00 Sunday, Mar 27–16:00 Sunday, Mar 27
• East US
  • 23:00 Saturday, Mar 26–11:00 Sunday, Mar 27 (UTC-4)
  • 03:00 Sunday, Mar 27–15:00 Sunday, Mar 27
• Central India
  • 23:00 Saturday, Mar 26–11:00 Sunday, Mar 27 (UTC+5:30)
  • 17:30 Saturday, Mar 26–05:30 Sunday, Mar 27
• West India
  • 23:00 Saturday, Mar 26–11:00 Sunday, Mar 27 (UTC+5:30)
  • 17:30 Saturday, Mar 26–05:30 Sunday, Mar 27
• Brazil South
  • 23:00 Saturday, Mar 26–11:00 Sunday, Mar 27 (UTC-3)
  • 02:00 Sunday, Mar 27–14:00 Sunday, Mar 27

All our servers are monitored and we will verify they return to proper functionality as soon as possible after the reboot event. This does not effect the functionality of your running instances as a whole, it will only potentially effect the capability to install new packages or retrieve updates for installed packages.

On December 15, 2015 was First Customer Ship (FCS) date for SUSE Linux Enterprise Server 12 SP1 and images were published in AWS EC2, Google Compute Engine and Microsoft Azure. With this release the transition phase for SLES 12 images/instances has started. One of the features of SLES 12 SP1 is that it provides an upgrade path from SUSE Linux Enterprise Server 11 SP4, which was not available in SLES 12. While the SUSE Linux Enterprise Server 11 release stream still has a lot of life left (until March 31st 2019), waiting that long for an upgrade to SLES 12 SP1 will accumulate a longer string of upgrades that will need to be stacked and performed later. By the way, information about SUSE product support dates are found here.

SUSE Linux Enterprise Server 12

As the 6 month clock for SLES 12 is ticking we’ll start with the upgrade process for SUSE Linux Enterprise Server 12. Note that this does not yet apply to HPC instances running in Microsoft Azure. Due to changes to the host configuration in Microsoft Azure for HPC other changes at he image level and a separate upgrade process will be documented when the time comes and all the parts fit together.

First double check the version of SLES you are running in your instance as follows:

zypper products -i | grep Server

This will produce a one line output as follows for a SLES 12 system:

i

|

@System

|

SLES

|

SUSE Linux Enterprise Server 12

|

12-0

|

x86_64

|

Yes

Once you have confirmed that you are on SLES 12 the upgrade path to SLES 12 SP1 is as follows:

1.) zypper up

2.) cd /etc/zypp/repos.d

3.) for i in *SLES12*;do newname=${i/12/12-SP1}; mv $i $newname; done

4.) for i in *SDK12*;do newname=${i/12/12-SP1}; mv $i $newname; done

5.) sed -i s/12/12-SP1/ *SP1*.repo

6.) zypper refresh

7.) zypper in zypper

This will produce a request to resolve a package requirement dependency issue indicated by a message starting as follows:

Problem: zypper-1.12.23-1.3.x86_64 requires libzypp.so.1519()(64bit), but this requirement cannot be provided

There is nothing to worry about the package that provides the necessary library was renamed and the proper (libyui-ncurses-pkg7) will automatically be pulled in. Therefore the solution is to select “Solution: 1”

Solution 1: deinstallation of libyui-ncurses-pkg6-2.46.1-3.4.x86_64

by entering “1” at the prompt. Then enter “y” at the next prompt

The final step is to do the distribution upgrade.

8.) zypper dup

Now if you run the ‘zypper products -i | grep Server‘ you will see that you have successfully upgraded your running instance to SLES 12 SP1, all that’s left to do is reboot the instance at your earliest convenience.

If you have the default motd (message of the day file, /etc/motd) you will be greeted with a message that announces the system as being SLES 12 the next time you log in, thus you might want to change this to avoid any confusion. A simple

9.) sed -i s/Server\ 12/Server\ 12-SP1/ /etc/motd

will do the trick.

If all your running instances are SLES 12 based you are done and can skip the rest of this blog, unless of course you are curious what comes along with a major version upgrade.

SUSE Linux Enterprise Server 11 SPx

Update 2019-12-16

This process will stop working after May 31, 2020, see Step 2 Toward Enhanced Update Infrastructure Access for details. Also note the process is quite old and things have drifted since the original post. We have made an effort to keep this working until May 31, 2020.

End update 2019-12-16

If way back when you started your cloud endeavor you started on SUSE Linux Enterprise Server 11 SP3 or even earlier and followed the upgrade procedures described in the Life Cycle Blog and/or the SLES 11 SP4 announcement blog then you should be running SUSE Linux Enterprise Server 11 SP4 and the following instructions apply just as if your instance originated from a SLES 11 SP4 image. If your ‘zypper products -i | grep Server‘ does not produce a line that contains “SUSE Linux Enterprise Server 11 SP4” please upgrade to SLES 11 SP4 first as outlined in the previous blogs before moving to SLES 12 SP1.

As this is a major distribution upgrade things are a bit more complicated than for a service pack upgrade. Therefore this also requires a disclaimer, of course.

First you should backup your data, just in case something goes terribly wrong. Further, there are many packages you may have installed on your running system in addition to the packages that are part of the base image. These packages may have an effect on the way the dependency solver charts its course through the package changes between the two distributions. Thus, you want to evaluate the solutions the solver proposes and take appropriate action. The solution steps shown during the procedure apply to the migration of a SLES 11 SP4 system as released by SUSE with no additional packages installed. It is not feasible for us to test all possible combinations of package installations. In AWS where some may still run 32-bit systems, please note it is not possible to migrate from a 32-bit system to a 64-bit system and, as announced in the Life Cycle blog 32-bit images are no longer maintained. There are also changes to applications, MariaDB is shipped with SLES 12 while MySQL was shipped with SLES 11 and thus appropriate actions need to be taken. PostgreSQL has been updated to a newer version and manual intervention is required for DB migration as well. The Upgrade Preparation Guide contains helpful information for the DB migrations, sections 14.1.5 and 14.1.6.

Many things have changed since the first release of SLES 11. YaST has been completely rewritten in Ruby, Networking is now handled by wicked, system registration uses a different protocol, as well as upgrades to libraries, kernel etc. The changes around YasT, Networking, system registration, and system initialization need to be handled more or less manually. For a more complete guide of the feature changes please consult the SLES 12 SP1 Release Notes . The following step by step guide is expected to render a fully functional SLES 12 SP1 system at the end.

We start out as usual by first updating the running instance to the latest state of the art:

1.) zypper up

The repository structure also has undergone major changes and thus rather than trying to bend existing configurations into shape it is much easier to simply get new pre-configured repositories

2.) cd /etc/zypp/repos.d

3.) rm *.repo

The pre-configured repositories are different for the different cloud frameworks and are in different locations. Pull the repository configuration as follows:

for AWS EC2:

4.) wget http://54.197.240.216/sles12sp1-ondemand-repos.tar.bz2
4 a.) sha1sum sles12sp1-ondemand-repos.tar.bz2
ba144d80107d265135b38b78d30175493c9ca63b sles12sp1-ondemand-repos.tar.bz2
for GCE:

4.) wget http://108.59.80.221/sles12sp1-ondemand-repos.tar.bz2
4 a.) sha1sum sles12sp1-ondemand-repos.tar.bz2
6243e12096b1c0782883a624c31261f5a4ce54ab sles12sp1-ondemand-repos.tar.bz2

for Microsoft Azure:

4.) wget http://52.147.176.11/sles12sp1-ondemand-repos.tar.bz2
4 a.) sha1sum sles12sp1-ondemand-repos.tar.bz2
716dec1db99590e2f275d5caca90f2edb2c556ea sles12sp1-ondemand-repos.tar.bz2

Unpack the repos and dispose of the tarball

5.) tar -xjf sles12sp1-ondemand-repos.tar.bz2
6.) rm sles12sp1-ondemand-repos.tar.bz2

With the new repositories configured we need to refresh the data for zypper

7.) zypper refresh

On AWS EC2 PV instances a new repository will be added that strictly speaking is not necessary. You can choose to add the repo by trusting the key, enter “a” at the prompt, or not accept the repo by entering “r” at the prompt. If you do not want the repository simply remove the file at your convenience  with

rm /etc/zypp/repos.d/nVidia-Driver-SLE12.repo

after you entered “r“.

Now it is time to deal with the changes that would otherwise cause trouble if we would just update zypper and then upgrade the system. First lets get rid of YaST completely, we’ll install the new modules again later.

8.) zypper rm yast2 yast2-core yast2-libyui

Another big change for SLES 12 was the move to systemd as the initialization system, and this is where we will start with the upgrade process.

9.) zypper in systemd

This install request requires manual interaction in helping zypper cope with the package and package dependency changes. For each transaction that requires manual interaction zypper displays a message of the form:

Problem: * PACKAGE_NAME requires ….

The “*” is a placeholder and may be empty depending on the condition zypper encounters. For example a message may be

Problem: cloud-regionsrv-client-6.4.3-26.1.noarch requires systemd, but this requirement cannot be provided

or

Problem: solvable libffi4-5.2……

The tables below list the package name zypper will show in the messages in the left hand column and shows the solution to be selected in the middle column. The right hand column mentions the action and package name of the first entry in the solution proposed by zypper. The solution path is different on the various cloud platforms as the packages installed in the SLES 11 SP4 image published are different. Simply type the number at the prompt and hit “Enter”.

for AWS EC2:

systemd-210	1	deinstallation of sysvinit
gvfs-backends-1.4.3	2	deinstallation of gvfs-backends
cloud-regionsrv-client-6.4.0 *	1	deinstallation of cloud-regionsrv-client
apache2-mod_php53-5.3.17	3	deinstallation of syslog-ng
libsnmp15-5.4.2.1	2	deinstallation of libsnmp15
perl-satsolver-0.44.5	2	deinstallation of perl-satsolver
blt-2.4z	2	deinstallation of OpenIPMI
python-m2crypto-0.21.1	2	deinstallation of scout
python-ply-3.6	4	deinstallation of rpm-python
libffi4-5.2.1	**	deinstallation of command-not-found

* Only for HVM instances
** Solution 2 on PV instances, Solution 3 on HVM instances

for GCE:

systemd-210	1	deinstallation of sysvinit
cloud-regionsrv-client-plugin-gce-1.0.0	2	deinstallation of cloud-regionsrv-client-plugin-gce
gvfs-backends-1.4.3	2	deinstallation of gvfs-backends
libblocxx6-2.1.0.342	3	deinstallation of syslog-ng
libsnmp15-5.4.2.1	2	deinstallation of libsnmp15
perl-satsolver-0.44.5	2	deinstallation of perl-satsolver
python-satsolver-0.44.5	2	deinstallation of python-satsolver
python-crcmod-1.7	2	deinstallation of scout
python-m2crypto-0.21.1	4	deinstallation of rpm-python
python-python-gflags-2.0	4	deinstallation of command-not-found

for Microsoft Azure:

systemd-210	1	deinstallation of sysvinit
gvfs-backends-1.4.3	2	deinstallation of gvfs-backends
cloud-regionsrv-client-6.4.0	1	deinstallation of cloud-regionsrv-client
libblocxx6-2.1.0.342	3	deinstallation of syslog-ng
libsnmp15-5.4.2.1	2	deinstallation of libsnmp15
perl-satsolver-0.44.5	2	deinstallation of perl-satsolver
limal-ca-mgm-perl-1.5.24	2	deinstallation of OpenIPMI
python-m2crypto-0.21.1	1	deinstallation of scout
libffi4-5.2.1+r226025	2	deinstallation of rpm-python

After all the questions are answered it is time to let zypper do its magic and switch the system over from SysVinit to systemd. This update will also bring with it the new version of zypper and a few other goodies, including wicked for network management.

With systemd installed we are one step closer to a SLES 12 SP1 system.

Another change in SLES 12 is the way zypper handles credentials and thus we need to make some changes to handle this and preserve our repositories.

10.) cd /etc/zypp/credentials.d
11.) cp NCCcredentials SCCcredentials

for AWS EC2:

12.) mv NCCcredentials SMT-http_smt-ec2_susecloud_net

for GCE:

12.) mv NCCcredentials SMT-http_smt-gce_susecloud_net

for Microsoft Azure:

12.) mv NCCcredentials SMT-http_smt-azure_susecloud_net

Now we need to modify the repository configuration to pick up the new credentials file.

13.) cd /etc/zypp/repos.d

for AWS EC2:

14.) sed -i s/NCCcredentials/SMT-http_smt-ec2_susecloud_net/ *.repo

for GCE:

14.) sed -i s/NCCcredentials/SMT-http_smt-gce_susecloud_net/ *.repo

for Microsoft Azure:

14.) sed -i s/NCCcredentials/SMT-http_smt-azure_susecloud_net/ *.repo

Almost there. Next we need to refresh the repository data and with it re-import all the signing keys.

15.) zypper refresh

For each repository zypper will ask to confirm if the signing key should be imported. Answer each question with “yes“, sorry this involves a lot of typing.

Now we need one of the packages back that was removed earlier.

16.) zypper in --replacefiles cloud-regionsrv-client

Select solution “1“. In this install we added “–replacefiles” option to instruct zypper to accept the new packages even if potential file conflicts may arise.

For the next zypper operation do not worry about the failure of the refresh of the “cloud_update” service at the beginning of the output of the zypper command . It is caused by a mismatch in the PYTHON_PATH as we are currently in a system state somewhere in between SLES 11 SP4 and SLES 12 SP1. Once the process is complete and the migration to SLES 12 SP1 is done this will correct itself.

on GCE We need an extra package:

16.a) zypper in cloud-regionsrv-client-plugin-gce

Finally we are ready for the big switch over and the full system upgrade.

17.) zypper dup --replacefiles

Once you answer “y“, zypper will do its magic and you have some time for a beverage. The performance of the upgrade process heavily depends on whether your instance is SSD backed or runs on a spinning media backend.

With the system upgrade complete we need to perform a few more steps to make certain everything is consistent. For the following installations zypper will produce the “Installation has completed with error.” message at the end of each install step. This is caused by the intermediate system state we find ourselves in. At this point there is nothing to worry about.

Start out by installing YaST and the YaST modules that we removed in the early part of this migration process:

18.) zypper in --replacefiles yast2-slp-server yast2-country yast2-sudo yast2-trans-stats yast2-squid yast2-nfs-client yast2-iscsi-client yast2-nfs-common yast2-dns-server yast2-pam yast2-update yast2-storage yast2-network yast2-slp yast2-nis-server yast2-transfer yast2-sysconfig yast2-packager yast2-audit-laf yast2-http-server yast2-dbus-server yast2-installation yast2-add-on yast2-ruby-bindings yast2-hardware-detection yast2 yast2-support yast2-services-manager yast2-ntp-client yast2-iscsi-lio-server yast2-core yast2-online-update yast2-bootloader yast2-tune yast2-pkg-bindings yast2-ycp-ui-bindings yast2-security yast2-users yast2-samba-server yast2-ftp-server yast2-trans-en_US yast2-xml yast2-proxy yast2-firewall yast2-online-update-frontend yast2-docker yast2-samba-client yast2-mail yast2-isns yast2-dhcp-server yast2-schema autoyast2-installation yast2-perl-bindings yast2-tftp-server yast2-printer yast2-ca-management yast2-ldap yast2-nis-client yast2-inetd yast2-theme-SLE yast2-country-data yast2-nfs-server yast2-kdump

At some point the command-not-found utility was removed, lets add this back as well:

19.) zypper in command-not-found

There are also some packages that fall between the cracks in this migration and a few packages that are useful but did not yet exist when the SLES 11 SP4 images were released, lets get those installed next

20.) zypper in cracklib-dict-small dhcp dhcp-client fonts-config grub2-x86_64-xen mozilla-nss-certs patterns-sles-Minimal sle-module-adv-systems-management-release sle-module-containers-release sle-module-legacy-release sle-module-public-cloud-release sle-module-web-scripting-release

For AWS EC2 we developed some tools that help with image management and you may install those if you are interested.

20 a.) zypper in python-ec2deprecateimg python-ec2publishimg python-ec2uploadimg python-ec2utilsbase

For Microsoft Azure we have started work on command line tools  a while back and a package that provides a good set of functionality, including the Azure Files feature is available

20 a.) zypper in python-azurectl

As the init system has changed services need to be re-enabled. The following lists the minimum services to be able to get back into your system after reboot.

21.) systemctl enable guestregister.service
systemctl enable haveged.service
systemctl enable irqbalance.service
systemctl enable iscsi.service
systemctl enable iscsid.socket
systemctl enable lvm2-lvmetad.socket
systemctl enable ntpd.service
systemctl enable purge-kernels.service
systemctl enable rsyslog.service
systemctl enable sshd.service
systemctl enable wicked.service

In addition there are some framework specific services that need to be re-enabled.

for AWS EC2:
22.) systemctl enable cloud-init-local.service
23.) systemctl enable cloud-init.service
24.) systemctl enable cloud-config.service
25.) systemctl enable cloud-final.service

for GCE:
22.) systemctl enable google.service
23.) systemctl enable google-accounts-manager.service
24.) systemctl enable google-address-manager.service
25.) systemctl enable google-startup-scripts.service

for Microsoft Azure:
22.) systemctl enable hv_fcopy_daemon.service
23.) systemctl enable hv_kvp_daemon.service
24.) systemctl enable hv_vss_daemon.service
25.) systemctl enable waagent.service

If you had other services enabled, such as Apache you want to enable those services at this time as well.

With the system initialization changed to systemd the information in inittab is no longer useful, lets clear out the file.

26.) echo "" > /etc/inittab

The implementation of ssh has gained some new ciphers and thus we want to generate any keys that might be missing.

27.) /usr/sbin/sshd-gen-keys-start

There were also some changes in PAM (Pluggable Authentication Module) and fiddling with the existing configuration files would be rather cumbersome. Thus a tarball with the configuration files needs to be downloaded and installed. If you have custom files please copy those to a “safe” place and restore them after the next few operations:

28.) cd /etc/pam.d
29.) rm *

for AWS EC2:

30.) wget http://54.197.240.216/pam_sles12_sp1_config.tar.bz2

for GCE:

30.) wget http://108.59.80.221/pam_sles12_sp1_config.tar.bz2

for Microsoft Azure:

30.) wget http://23.101.123.131/pam_sles12_sp1_config.tar.bz2

The checksum is framework indipendent in this case:
sha1sum pam_sles12_sp1_config.tar.bz2
6cd0fde2be66b7e996c571bec1b823ba0eb8d9a0 pam_sles12_sp1_config.tar.bz2

31.) tar -xjf pam_sles12_sp1_config.tar.bz2
32.) rm pam_sles12_sp1_config.tar.bz2

If you had any custom configuration files now is the time to restore them. Note that there were some significant changes in PAM and thus you should review your configuration files for compatibility. If you had modifications to existing “standard” files then you want to merge those modifications rather than clobber the files that were just installed.

The default syslog handling has also changed from syslog-ng in the SLES 11 release series to rsyslog in SLES 12. If you are dependent on syslog-ng for your setup you will need to install the syslog-ng package (zypper in syslog-ng) which will remove the installed rsyslog package.

As with the SLES 12 to SLES 12 SP1 migration described at the beginning you will probably want to change the motd file ‘/etc/motd‘ to avoid version confusion for later logins.

33.) sed -i s/11/12/ /etc/motd
34.) sed -i s/SP4/SP1/ /etc/motd

If you are not upgrading an AWS EC2 ParaVirtual (PV) instance your system is now ready for reboot. If you started with a ParaVirtual instance (PV) some other changes need to be made to be able to boot after shutdown. If you are not certain if you are on a PV instance you can run

35.) ec2metadata | grep aki

If the return is empty you are running an HVM instance and you are good to go, i.e. reboot and run on your freshly migrated SLES 12 SP1 system. If the return is not empty you have a few more steps to complete. The additional steps are necessary to handle the bootloader change that occured between SLES 11 and SLES 12. In SLES 12 the bootloader is GRUB2, while in SLES 11 the bootloader is GRUB. This change requires that a new boot kernel (aki) is associated with the instance.

First we want to stop the running instance that was just migrated to SLES 12 SP1

36.) aws ec2 stop-instances --region THE_REGION_YOU_ARE_WORKING_IN --instance-ids THE_INSTANCE_ID_OF_THE_UPGRADED_INSTANCE

Once the image is stopped, monitor with:

37.) aws ec2 describe-instances --region THE_REGION_YOU_ARE_WORKING_IN --instance-ids THE_INSTANCE_ID_OF_THE_UPGRADED_INSTANCE

you may proceed. Unfortunately there is no simple unique state identifier and thus you need to look through the returned JSON structure for the “State” element and then look at the “Name” key to determine the state of the instance. When it has the value “Stopped” you can proceed with the next steps.

The following table contains the aki-ID for each region that needs to be associated with the stopped instance.

ap-northeast-1	aki-f1ad9bf0
ap-southeast-1	aki-ca755498
ap-southeast-2	aki-8faec3b5
cn-north-1	aki-9c4ad8a5
eu-central-1	aki-e23f09ff
eu-west-1	aki-fce8478b
sa-east-1	aki-b99024a4
us-east-1	aki-f4bc469c
us-gov-west-1	aki-07026424
us-west-1	aki-f9786dbc
us-west-2	aki-5f125d6f

Note that the new region, ap-northeast-2, is not listed in the table. The new region does not support PV instances.

38.) aws ec2 modify-instance-attribute --region THE_REGION_YOU_ARE_WORKING_IN --instance-id THE_INSTANCE_ID_OF_THE_UPGRADED_INSTANCE --kernel VALUE_FROM_TABLE_ABOVE

If you now re-execute the “describe-instances” command from step 37 you will notice that the aki ID in the output has changed. With the instance associated with an aki that can handle GRUB2 you can now start the instance.

39.) aws ec2 start-instances --region THE_REGION_YOU_ARE_WORKING_IN --instance-ids THE_INSTANCE_ID_OF_THE_UPGRADED_INSTANCE

After all this what did we end up with? You now have a fully functional SLES 12 SP1 instance running. One of the differences to a SLES 12 SP1 instance started from scratch is that the upgraded instance contains more system packages. This is due to the fact that our released SLES 11 images contained more packages compared to the SLES 12 images. The reduced number of packages is based on the thought that it is super fast and easy to install any package that one might need. Additionally many users connect instances to a configuration management system and the “check if something exists” is almost time equivalent to “install whatever is missing”, especially with the SUSE supported region local update infrastructure. If you want a detailed comparison between the systems you can use machinery to compare to systems. If the systems are not on the same network the guide Comparing 2 Systems Using Machinery may be useful.

Update 2019-12-12

As the final step and related to the update infrastructure upgrade you want to re-register your instances

40.) registercloudguest --force-new

End update

With this we hope you will enjoy the improvements made in SUSE Linux Enterprise Server 12 SP1 and take advantage of the flexibility that the Module Architecture provides. Also if you want to share information don’t forget the SUSE Forums.

Let’s get on the way back train again for a minute or two, and before I get started let me include a disclaimer; The following includes semi-technical marketing information as a prelude to discussing technical solutions. I will try and keep the marketing speak to a minimum.

Anyway, back to the way back train. Way back when SUSE was part of Novell, and SUSE Studio was born along with it came the so called SUSE Appliance program. The basic idea of the appliance program is pretty simple; as an ISV or systems integrator you sign a contract with SUSE to distribute SUSE Linux Enterprise along with one or more applications in a ready to integrate and ready to run solution package. The solution package may come in a number of form factors, virtual images, integrated with hardware, or self installing appliances on optical media or a USB stick. When the whole thing started the public cloud was already in existence but not many people did really pay attention to it.

Fast forward 5 or 6 six years and the world is a very different place. The appliance program is now called SUSE Embedded and foremost, the Public Cloud, i.e. Amazon Web Service, Google Compute Engine, and Microsoft Azure are the talk of the town. As we all know, when it comes to software and anything related to IT, stuff tends to live more or less forever. Thus, appliances/integrated systems/embedded in the form of self installing images, ready to run solutions as virtual images as well as hardware appliance have their place and will continue to have a place going forward. However, what is really taking off as far as market growth is concerned is the Public Cloud, IaaS (Infrastructure as a service), as well as PaaS (Platform as a Service), and SaaS (Software as a service) along with the Marketplaces offered by the Public Cloud Service Providers (CSP) . In the appliance model one concern has always been how to handle the OS that is bundled with the application(s) into the appliance. What to do with security updates and how to manage the implied responsibility for the OS by the one providing the appliance. In the solution oriented way offered by the super market approach of Amazon and other CSPs many of these concerns go away thus reducing a business/support problem to a purely technical problem. Technical problems are often much easier to solve and this is really what the rest of this blog is all about. Before we get there however, lets take a little closer look at the support and update issue outside the public cloud to help set the stage for what is to come.

As an embedded systems vendor there is an implied responsibility for the OS that gets shipped along with the appliance. In the light of hartbleed, shellshock, and other issues this may appear as a daunting task, even if you get backed by SUSE as the distribution vendor. On the SUSE side we tried to make this reasonabaly easy with the SUSE Lifecycle Management Server. But this solution was not really sufficiently universal and it expected the provider to get out of their comfort zone to make updates available for the underlying OS via a web infrastructure. For many the operation of such a service is way outside their core business and way outside their focus area, no matter how big of a check the end customer is wiling to write. In the Public Cloud, and I will focus on Amazon Web Services from here on forward these concerns are eliminated.

With the semi-technical background behind us lets take a closer look at how it works. First lets address the “how does stuff get updated” question. SUSE maintains an update infrasturcture within Amazon EC2, details about how this works are included in this presentation from SUSECon 14, that any on demand image in the Amazon EC2 infrastructure has access to. Thus as a solution provider the question about how to update the underlying OS for security fixes is resolved. “All” that needs to be done is to offer a solution in Amazon Marketplace that is build on top of a SUSE on demand image. On demand SUSE Linux Enterprise Server images automatically connect to the SUSE maintained update infrastructure and thus have access to security and bug fixes. Such solutions can be offered in the Amazon Marketplace or the general Amazon EC2 catalog. There are 3 ways to get there:

1.) build your image with KIWI,
2.) build your image with SUSE Studio
3.) create a new image from a running customized instance.

1.) Build your image with KIWI

KIWI is an image build system that allows you to build images in numerous form factors. A setup guide for using kiwi on a SUSE Linux enterprise Server 12 system can be found here  and for SUSE Linux Enterprise Server 11 here . The examples provided by the kiwi-doc package are a great place to start. The Cloud:Images project in the openSUSE build service contains images built with openSUSE as sub-projects for a number of public clouds. The build set up for openSUSE and SLES is pretty much the same with the key difference being the repository configuration file in the kiwi image description. Building images for Amazon EC2 requires the following type definition in the kiwi configuration file:

<type image=”vmx” filesystem=”ext4″ boot=”vmxboot/suse-SLES12″ bootloader=”grub2″ kernelcmdline=”console=ttyS0,115200n8 multipath=off net.ifnames=0 NON_PERSISTENT_DEVICE_NAMES=1″ boottimeout=”1″ installprovidefailsafe=”false” firmware=”ec2hvm”>

In this case we are building a SLES 12 image that is targeted for an HVM image. Amazon prefers that new images use HVM and are SSD backed. The other piece that belongs into the kiwi configuration file that makes things work is the Xen kernel module which must also be included in the initrd that kiwi builds. This happens with the following specification in the <packages> section:

<package name=”xen-kmp-default” bootinclude=”true”/>

You also need to specify the bootloader xen package:

<package name=”grub2-x86_64-xen” bootinclude=”true”/>

The openSUSE example in the openSUSE build service is a relatively minimalistic setup and thus you can use it to build up from there. For images that are destined for the Marketplace Amazon requires that the default user is a user other than root. This is accomplished via the cloud-init config file that you will want to create and place into the kiwi overlay tree. There are also interesting tidbits to be found in the config.sh file. Most of the settings should apply to your image build.

In order to resolve the update problems of appliances discussed earlier you want to include the following packages in your build:

<package name=”cloud-regionsrv-client”/>
<package name=”regionServiceClientConfigEC2″/>
<package name=”regionsrv-certs”/>
<package name=”regionServiceCertsEC2″/>

These packages contain the necessary functionality to hook into the SUSE operated update infrastructure in Amazon EC2. You will need to enable the registration service which is accomplished by adding

suseInsertService guestregister

to the config.sh file. Your image must also contain at least the release package for SUSE Linux Enterprise Server. On demand images published by SUSE contain the release packages of all modules to allow customers access to all the packages in SLE 12 that are considered part of SLES. This looks as follows in the kiwi configuration file:

<package name=”sle-module-adv-systems-management-release”/>
<package name=”sle-module-adv-systems-management-release-POOL”/>
<package name=”sle-module-legacy-release”/>
<package name=”sle-module-legacy-release-POOL”/>
<package name=”sle-module-public-cloud-release”/>
<package name=”sle-module-public-cloud-release-POOL”/>
<package name=”sle-module-toolchain-release”/>
<package name=”sle-module-toolchain-release-POOL”/>
<package name=”sle-module-web-scripting-release”/>
<package name=”sle-module-web-scripting-release-POOL”/>
<package name=”sle-sdk-release”/>
<package name=”sle-sdk-release-POOL”/>
<package name=”sles-release”/>
<package name=”sles-release-POOL”/>

You can pick and choose which modules you want to enable in your appliance. We strongly recommend to at least enable the public cloud module in addition to the base product.

If you already have access to the SUSE Linux Enterprise repositories (remember access for development purposes if free through Partner Net) you probably want to set up a local SMT server to decrease you kiwi image build times. Optionally you can build your image on a SLES instance running in Amazon and pull the updates from the SMT server that SUSE provides to update on demand images. With this you are well on your way to build a SLES image that can include your application(s) that you want to offer in Amazon Marketplace. Once the image is built you want to upload it as an on demand image. This process is automated with the ec2uploadimg utility. As the base image to help with the upload you want to use a SLES 12 on demand image for the region where you want to offer your solution. Find the latest SLES 12 image with pint using the following command.

pint amazon images –region THE_REGION_OF_INTEREST –active –filter ‘name~sles-12’ | grep -v byos

Use the ami-id of the entry in the configuration file for ec2uploadimg and then use the –root-swap option. Once your image is uploaded you have an on demand image that can connect to the SUSE update infrastructure in Amazon EC2. Once tested and ready to publish you can copy the image into different regions using the

aws ec2 copy-image

command. The magic that turns the image into an on demand image will come along in the copy image process. With the image uploaded to your account you can publish it in the Amazon Marketplace and/or in the general catalog. For publishing to the general catalog you can utilize the ec2publishimg utility.

This is pretty much all that needs to be done to build an image with kiwi and publish it in the Amazon Marketplace. You do not need to have a contract with SUSE or any kind of formal business relationship. You do need a Marketplace agreement with Amazon. Thus all the steps that may have been a hindrance in the integrated system approach outside of the public cloud have been resolved.

2.) Build your images with SUSE Studio

SUSE Studio is a web application that uses KIWI in the backend to build images. As such it has all the magic, spare the user configuration for cloud-init, built in. You can provide the cloud.cfg file as an overlay file in SUSE Studio and build your image. When using the upload functionality built into SUSE Studio for Amazon you automatically get an on demand image that is ready to tie into the SUSE operated update infrastructure in Amazon EC2. The caveat is that when you build an image with SUSE Studio it will be a PV image that is backed by standard, i.e. spinning disks, storage. Thus this image will launch a bit slower than an SSD backed image. With SUSE Studio you can get your image built in 15 minutes and be up and running in Amazon with full access to update repositories in no time.

3.) Create a new image from a running customized instance

Start out by running a SUSE Linux Enterprise on demand image. You can find the latest image using pint as shown previously or you can launch it from the quick launcher. Once the instance is running login as ec2-user. You can use “sudo -i” to become super user and install software as you see fit. SLES packages are available from the configured repositories.

If you want the default user to be a different user than “ec2-user” modify the cloud-init config file (cloud.cfg in /etc/cloud). For a customized message of the day replace the /etc/motd file. After you have customized the running instance to your liking it is time for some clean up before you create a new image. You need to follow these steps to ensure the user of your Marketplace image has a first use experience.

1.) Reset the repository registration
rm /etc/zypp/repos.d/*
rm /etc/zypp/credentials.d/*
rm /etc/zypp/services.d/*
rm /var/log/cloudregister

2.) Reset the instance configuration
rm -rf /var/lib/cloud/*

3.) Reset the region association
Remove the comment and the entry that refer to smt from /etc/hosts

4.) remove the ec2-user
from /etc/passwd
from /etc/shadow
sed -i ‘s/ec2-user//’ /etc/group
rm -rf /home/ec2-user/

With the reset complete, stop the instance (shutdown -h now) or from the AWS web console

Now you are ready to create a new image

aws –profile PROFILE_NAME ec2 create-image –instance-id ID_OF_STOPPED_INSTANCE –name NAME_FOR_YOUR_IMAGE –description DESCRIPTION_FOR_YOUR_IMAGE –block-device-mappings ‘[{“DeviceName”:”/dev/sda1″,”Ebs”:{“VolumeSize”:10,”VolumeType”:”gp2″}}]’

Note the image creation command above is for HVM images, the preferred image type. For a PV image the DeviceName value needs to be set to “/dev/sda“.

The new image is ready for the Amazon Marketplace and can be copied to any region. It is an on demand image and will register with the SUSE maintained update infrastructure.