Public Cloud Infrastructure Update

1. Consolidation of update servers to serve all products from the same set of servers
2. Improved HA setup
3. Support of traffic routing through the datacenter

• There will be 3 update servers per region spread across multiple AZ's
• There is now a marker in instances which will identify it to update servers so that Cloud-Datacenter-Cloud dataflow is supported
• There will be SLES11 repos but none of them will receive any updates since general support has ended for SLES11
• VM's must be upgraded to the package "cloud-regionsrv-client" version 9.0.0 or greater
  • zypper up cloud-regionsrv-client
• IP address access restriction checks of old infrastructure will be disabled no later than end of June, 2020. After this Cloud-Datacenter-Cloud access is open for all regions. This also enables the "bring your own IP" feature in AWS

1. Instance is patched up-to-date
2. Instance image date stamp of 20200526 or greater for anything SLE 12 base in AWS and Azure

Situation 1: In AWS, there was a way to create private images that would drop the identifier in the metadata that makes an instance from this image a PAYG instance. These instances will not meet requirements to have access to the new update infrastructure. A migration to a new instance will be required to get access to update infrastructure.

Situation 2: In AWS, the marker could be lost if an older image was used. For SLES for SAP images with a date stamp prior to 20181212, there was a bug. This bug was caused by the baseproduct pointing to the wrong product definition file. If a zypper lr is run and no repositories are listed, the fix is:

cd /etc/products.d
rm baseproduct
ln -s SLES_SAP.prod baseproduct
systemctl start guestregister.service

Situation 3: In GCE, to access update infrastructure where traffic is routed to the datacenter, all instances must have access to instance metadata for identity tokens. This token is used to validate the identity of the request to SUSE's update infrastructure. Instances must have an associated service account in order to generate this token. This is the default behavior when an instance is launched. However, it is possible to create instances without a service account and therefore without the identity token. For such instances, you will need to stop the instance and add a service account to enable access to SLES update infrastructure.

curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=test

If the metadata returned is not a token and instead a message indicating the following, then instance is affected by this situation and remediation will need to be performed:

{"error":"invalid_request","error_description":"Service account not enabled on this instance"}

To remediate, add a service acount to the instance with no scopes replacing <instance>, <zone> and <service_account>:

gcloud compute instances stop <instance> --zone <zone>
gcloud compute instances set-service-account <instance> --zone <zone> --service account <service_account> --no-scopes
gcloud compute instances start <instance> --zone <zone>

To perform a bulk check for multiple instances, run the following gcloud command to list all instances in the project without a service account:

gcloud compute instances list --filter "serviceAccounts.email = ''"

Situation 4: In Azure, AWS, or GCE, access to repositories went away or are no longer present. This happens because access to the update servers was denied. Update the required packages manually.

All commands executed as root.

The example commands below require the substitution of one or two placeholders:
<arch> is either aarch64 if running ARM based instances or x86_64 if running x86_64 instances.
<slebase> is the base version of the SLES release running, either SLE12 or SLE15. 

For AWS instances:
wget --no-check-certificate https://52.15.49.139/late_instance_offline_update_ec2_<arch>_<slebase>.tar.gz
sha1sum late_instance_offline_update_ec2_<arch>_<slebase>.tar.gz
tar -xf late_instance_offline_update_ec2_<arch>_<slebase>.tar.gz
cd <arch>
zypper --no-refresh --no-remote --non-interactive in *.rpm

For Azure:
wget --no-check-certificate https://52.188.224.179/late_instance_offline_update_azure_<slebase>.tar.gz
sha1sum late_instance_offline_update_azure_<slebase>.tar.gz
tar -xf late_instance_offline_update_azure_<slebase>.tar.gz
cd x86_64
zypper --no-refresh --no-remote --non-interactive in *.rpm

For GCE:
wget --no-check-certificate https://104.196.61.109/late_instance_offline_update_gce_<slebase>.tar.gz
sha1sum late_instance_offline_update_gce_<slebase>.tar.gz
tar -xf late_instance_offline_update_gce_<slebase>.tar.gz
cd x86_64
zypper --no-refresh --no-remote --non-interactive in *.rpm

Here are the checksums for the tarballs:

AWS x86_64 SLE12: 1912c23c9140e406690160a9bd003a81fc79c14e late_instance_offline_update_ec2_x86_64_SLE12.tar.gz
AWS aarch64 SLE15: 250be20f24a941ef27fdb9727a3a4d34fc525ea3 late_instance_offline_update_ec2_aarch64_SLE15.tar.gz
AWS x86_64 SLE15: d8906af58483f9107146590cb604e1f32629a749 late_instance_offline_update_ec2_x86_64_SLE15.tar.gz
Azure SLE12: 7ec151e96455e629b128e03de073365874ded6b3 late_instance_offline_update_azure_SLE12.tar.gz
Azure SLE15: a64556df71a487c8adf93adca10b06c236030275 late_instance_offline_update_azure_SLE15.tar.gz
GCE SLE12: 497e2c455696db1ae8e3450a07ac8c74f1674fe5 late_instance_offline_update_gce_SLE12.tar.gz
GCE SLE15: b12ef58ff7575aa5314d1cb393a326c08681174b late_instance_offline_update_gce_SLE15.tar.gz

The tarballs contain the latest released packages of the code that connects an instance to the update infrastructure and the related configuration plus the dependencies.

It is not required to re-register the instance. Once the packages are installed, access to the update infrastructure should be restored by checking the following:

zypper ref

This list of situations will be updated if anything new is reported and verified.  If issues are still occurring after working through these resolutions, work through respective cloud service provider support services.