SUSE Cloud Application Platform Air gapped installation

Containers has become first choice and ask from customers and Kubernetes is the first choice for container orchestration. Cloud native applications are being built. SUSE Cloud Application Platform is a modern application delivery platform used to bring an advance cloud native developer experience to Kubernetes. SUSE has containerized Cloud foundry.

Container images being downloaded on Kubernetes master and worker nodes when we deploy SUSE Cloud Application Platform. Source of these container images can be SUSE registry site which is registry.suse.com or can be a local registry in the same network of Kubernetes master and worker nodes.

Kubernetes master and worker nodes should not have internet access If we talk about a deployment in production environment and this will require us to setup repository and registry server in local network. This method is called airgapped method and we will see how to setup airgapped environment in this blog. A big thank you to Derek So who helped me in my setup.

# command to be executed

## line commented in a file

Environment Setup:

I have used NFS storage in this setup for sharing data between external RMT and internal RMT but an external drive can also be used. CaaSP v4 and CAPv1.5 images will be used.

Extrmt : Internet facing rmt server(it has access to internet)

No internet access on any other server but default route should be there for SUSE CaasP installation

Intrmt : internal rmt server

mgmt : kubernates management node

master : kubernetes master node

worker1 : kubernetes worker node

registry : registry pointing to intrmt(cname record in DNS)

charts : charts pointing to intrmt(cname record in DNS)

Install and configure RMT on extrmt server by following below link, Include package hub module, CaasP module. add aliases for registry.example.com, charts.example.com & intrmt in certificates section while configuring RMT. Please follow procedure at below link.

https://documentation.suse.com/sles/15-SP1/single-html/SLES-rmt/index.html#sec-rmt-installation-zypper

Once RMT is configured sync and mirror repositories which are required.

extrmt#rmt-cli sync

extrmt#for i in `cat`; do rmt-cli product enable $i; done

1772

1790

1867

1780

1863

1809

1923

1919

1763

1871

extrmt#rmt-cli mirror

Install and configure NFS

extrmt#mount /dev/sdb1 /mnt/storage (NFS share for data sharing with intrmt)

extrmt#zypper in nfs-kernel-server yast2-nfs-server

extrmt#yast2 nfs-server (export /mnt/storage directory)

Export RMT data and repos on NFS share.

extrmt#rmt-cli export data /mnt/storage

extrmt#rmt-cli export settings /mnt/storage

extrmt #rmt-cli export repos /mnt/storage

We will download all images required to setup SUSE CaaSP and CAP in below section on extrmt server. extrmt #zypper in docker helm-mirror skopeo

extrmt # cat /etc/sysconfig/docker

DOCKER_OPTS=”–insecure-registry registry.example.com”

extrmt #systemctl enable –now docker.service

extrmt #systemctl start docker.service

extrmt #CAASP_IMAGE_LIST_URL=https://documentation.suse.com/external-tree/en-us/suse-caasp/4/skuba-cluster-images.txt

extrmt #LOCAL_DIR=/mnt/storage/suse

extrmt #LOCAL_CHART_URL=https://charts.example.com

extrmt #LOCAL_REGISTRY_URL=https://registry.example.com:5000

extrmt #export CAASP_IMAGE_LIST_URL

extrmt #export LOCAL_DIR

extrmt #export LOCAL_CHART_URL

extrmt #export LOCAL_REGISTRY_URL

extrmt #mkdir -p $LOCAL_DIR

extrmt #curl  $CAASP_IMAGE_LIST_URL -o $LOCAL_DIR/caasp-image-list.txt

extrmt #awk ‘{print $NF}’ $LOCAL_DIR/caasp-image-list.txt | cut -c 7- | sed ‘/^$/d’ | sort -u > extrmt $LOCAL_DIR/caasp-image-download.txt

extrmt #mkdir –p $LOCAL_DIR/skopeodata/registry.suse.com/caasp/v4/

extrmt #for img in `cat $LOCAL_DIR/caasp-image-download.txt`

do

skopeo copy docker://$img dir:$LOCAL_DIR/skopeodata/registry.suse.com/caasp/v4/$img

done

extrmt #vi $LOCAL_DIR/cap-image-download

registry.suse.com/cap/uaa-configgin-helper:7ef898a83f98f20b3340e88760e99e30c60081c0

registry.suse.com/cap/uaa-mysql-proxy:797ce1924c85379bf1c83e830955d528597cc832

registry.suse.com/cap/uaa-mysql:903a2ac9b66484f32137b9029b63ce845695635f

registry.suse.com/cap/uaa-post-deployment-setup:18a5bef2ffe2f8dc43b47f0fc1c34266c49623cf

registry.suse.com/cap/uaa-secret-generation:0653dd4863841f90cb585d36907afdfb7c1369bf

registry.suse.com/cap/uaa-uaa:9de4f84fcaf0672b7488cc0c77342ed955c87e2e

registry.suse.com/cap/recipe-downloader:0.30.0

registry.suse.com/cap/recipe-executor:0.31.0

registry.suse.com/cap/recipe-uploader:0.28.0

registry.suse.com/cap/scf-adapter:c306b00f317984c17c7a16e7895e664152754725

registry.suse.com/cap/scf-api-group:240bba58ea8c6816e874d0df72bec5e6a7288bd8

registry.suse.com/cap/scf-autoscaler-actors:76a2de51e3236e9bfbfb4037c839a6f966d4ec51

registry.suse.com/cap/scf-autoscaler-api:6dda56e79f76a07d755acdf5f0787a5f5bb5e0de

registry.suse.com/cap/scf-autoscaler-metrics:bed854fffa70100254372ec6e860e13ea9131934

registry.suse.com/cap/scf-autoscaler-postgres:87654c0891279508582d31167ba3552739507b5a

registry.suse.com/cap/scf-bits:98c80aec4ca157e70004a3e22b08a53fe529e9b8

registry.suse.com/cap/scf-blobstore:e9843533b9b50c8732c26a1f019c32c0e5d1d026

registry.suse.com/cap/scf-cc-clock:5979bc5c9f23499889b870eee3a473dadf32c458

registry.suse.com/cap/scf-cc-uploader:0da2a1e77e892ad37f09c512e34f3f08ec28379d

registry.suse.com/cap/scf-cc-worker:311734ce2f5a8b3c7d3661233f4ca8f34ec2e399

registry.suse.com/cap/scf-cf-usb-group:43fd573f368c9298867830eae61503182198005d

registry.suse.com/cap/scf-configgin-helper:f9d9c2c3d04ad90d1609b7b31b5548979c54b17f

registry.suse.com/cap/scf-configure-eirini:16243b323f8a90221d32c9fd1e40dea521dc2487

registry.suse.com/cap/scf-credhub-user:22b8e050b92b3b9f044166460e83a358d3bb34af

registry.suse.com/cap/scf-diego-api:89981b0a374f50ff4c97558f2c4ea2abf325da8d

registry.suse.com/cap/scf-diego-brain:d5c5ea22e4535230579c6b25d7b709fa60dc1e4b

registry.suse.com/cap/scf-diego-cell:735bec6bc52438a4b73b670ad4a4c2959b82ae69

registry.suse.com/cap/scf-diego-ssh:b5d6f4e69a3d51a6f17618caba4e3352cd6fca7c

registry.suse.com/cap/scf-doppler:da14867a9ba02904e7986db10b6fb70ad1b6eab5

registry.suse.com/cap/scf-eirini-persi:8158bc02c4ed40d622fc33ae506ca1ce4ed8ca3f

registry.suse.com/cap/scf-eirini-ssh:6ca2c078eeb69357b9d5dc901577ebf96e7c994d

registry.suse.com/cap/scf-eirini:155759633d709645f18b2e3cdced9183d0f0169f

registry.suse.com/cap/scf-locket:f075cf11061939b5582b2f6cd41defd902f8f8da

registry.suse.com/cap/scf-log-api:964528e7a74b1cc28bd9b934d30ef28f0fcc1ced

registry.suse.com/cap/scf-loggregator-agent:98a13dd53091f3cfc60126204d00aa5033ea7e1f

registry.suse.com/cap/scf-mysql-proxy:3e3d60899c81a9736cb4f36db2d73a8c5a8cc4b7

registry.suse.com/cap/scf-mysql:d7e00908f08c51769925d5fbaee039ea9282705f

registry.suse.com/cap/scf-nats:ea3de18b750ca894e7062f7b060a22ee4f019c0e

registry.suse.com/cap/scf-nfs-broker:fdbf98742036fd626f96fa27539b90bf43845fe3

registry.suse.com/cap/scf-post-deployment-setup:4d437fbe87dae722cfc235f75cc4f7774b335330

registry.suse.com/cap/scf-router:4218c48f670db8b16ae19690d1dcf2c44c4312ba

registry.suse.com/cap/scf-routing-api:099258688e494ba27d13333f98e557bcf90cab5e

registry.suse.com/cap/scf-secret-generation:9d16624f9a6e8131119e3efbf6ff555f14822ddf

registry.suse.com/cap/scf-syslog-scheduler:d2aee32c52df5a9a93f34f5e6f729f31b03be7da

registry.suse.com/cap/scf-tcp-router:821f7f863c989a53ee00d65936360be2c2c05dc7

registry.suse.com/cap/scf-uaa:c2c5e587774a14e04c429288d10b9a84a42d48c4

registry.suse.com/cap/stratos-console:2.7.0-35f5964bd-cap

registry.suse.com/cap/stratos-jetstream:2.7.0-35f5964bd-cap

registry.suse.com/cap/stratos-mariadb:2.7.0-35f5964bd-cap

registry.suse.com/cap/stratos-postflight-job:2.7.0-35f5964bd-cap

registry.suse.com/cap/stratos-metrics-cf-exporter:1.1.2-85daaa2-cap

registry.suse.com/cap/stratos-metrics-firehose-exporter:1.1.2-85daaa2-cap

registry.suse.com/cap/stratos-metrics-firehose-init:1.1.2-85daaa2-cap

registry.suse.com/cap/stratos-metrics-nginx:1.1.2-85daaa2-cap

registry.suse.com/cap/stratos-metrics-configmap-reload:1.1.2-85daaa2-cap

registry.suse.com/cap/stratos-metrics-init-chown-data:1.1.2-85daaa2-cap

registry.suse.com/cap/stratos-metrics-kube-state-metrics:1.1.2-85daaa2-cap

registry.suse.com/cap/stratos-metrics-node-exporter:1.1.2-85daaa2-cap

registry.suse.com/cap/stratos-metrics-prometheus:1.1.2-85daaa2-cap

extrmt #for img in `cat $LOCAL_DIR/cap-image-download`

do

skopeo copy docker://$img dir:$LOCAL_DIR/skopeodata/registry.suse.com/cap/$img

done

extrmt #helm-mirror –new-root-url $LOCAL_CHART_URL https://kubernetes-charts.suse.com $LOCAL_DIR/suse-charts

extrmt #skopeo copy docker://registry.suse.com/caasp/v4/helm-tiller:2.16.1 dir:/mnt/storage/suse/skopeodata/registry.suse.com/caasp/v4

We have our extrmt server ready with all repositories and container images required to install SUSE CaaSP and CAP.

We will copy there repositories and container images to intrmt server so it can be used in SUSE CaaSP cluster. We will start configuring intrmt server in below section.

Install and configure RMT on intrmt server. We need to use RMT certificates from extrmt server so we will copy them on intrmt server.

intrmt#zypper in rmt-server

intrmt #scp extrmt:/etc/rmt/ssl/* /etc/rmt/ssl/

intrmt #yast2 rmt

intrmt #systemctl stop rmt-server-sync.timer

intrmt #systemctl disable rmt-server-sync.timer

intrmt #mount extrmt:/mnt/storage /mnt/storage

intrmt #rmt-cli import data /mnt/storage

intrmt #rmt-cli import repos /mnt/storage

intrmt #for i in `cat`; do  rmt-cli products enable $i; done

1772

1790

1867

1780

1863

1809

1923

1919

1763

1871

intrmt #zypper in docker

intrmt #zypper in -t pattern SUSE-CaaSP-Management

intrmt #SUSEConnect –product PackageHub/15.1/x86_64

intrmt #zypper install docker-distribution-registry

We need to create a virtual host server for helm charts. We will use nginx used by RMT for this purpose.

intrmt #vi /etc/nginx/vhosts.d/charts-server-https.conf

server {  listen 443   ssl;  server_name  charts.example.com;   access_log  /var/log/nginx/charts_https_access.log;  error_log   /var/log/nginx/charts_https_error.log;  root        /srv/www/;   ssl_certificate     /etc/rmt/ssl/rmt-server.crt;  ssl_certificate_key /etc/rmt/ssl/rmt-server.key;  ssl_protocols       TLSv1.2 TLSv1.3;   location /charts {    autoindex on;  }}

intrmt #systemctl restart nginx

We will configure registry server in below section. Our registry server is listening to 5000 port on intrmt server.

intrmt #systemctl enable –now docker.service

intrmt #mkdir /var/lib/registry

intrmt #cat /etc/sysconfig/docker

DOCKER_OPTS=”–insecure-registry registry.example.com”

intrmt #vi /etc/registry/config.yml

version: 0.1

log:

fields:

service: registry

storage:

cache:

blobdescriptor: inmemory

filesystem:

rootdirectory: /var/lib/registry

http:

addr: 0.0.0.0:5000

headers:

X-Content-Type-Options: [nosniff]

tls:

certificate: /etc/rmt/ssl/rmt-server.crt

key: /etc/rmt/ssl/rmt-server.key

health:

storagedriver:

enabled: true

interval: 10s

threshold: 3

intrmt #systemctl start registry

intrmt #systemctl enable registry

We will now sync all SUSE CaaSP and CAP container images from extrmt server to our registry server created in above section.

intrmt #zypper in skopeo

intrmt #skopeo sync dir:/mnt/storage/suse/skopeodata/registry.suse.com docker://registry.example.com:5000

intrmt #cd /mnt/storage/suse/suse-charts

intrmt #mkdir /srv/www/charts

intrmt #rsync -avP * /srv/www/charts/

intrmt #chown -R nginx:nginx /srv/www/charts

intrmt #chmod -R 555 /srv/www/charts/

So we have our internal RMT server and registry server ready now and we can use them to install SLES and then SUSE CaaSP and CAP. We need to perform below tasks on all SUSE CaaSP cluster machines(master, worker and management nodes).

#scp extrmt:/etc/rmt/ssl/rmt-ca.crt /etc/pki/trust/anchors/ #update-ca-certificates

#vi /etc/containers/registries.conf

[[registry]]

location = “registry.suse.com”

mirror = [{ location = “intrmt.example.com:5000”}]

## Optional: if the registry is not secure this can be set

## insecure = true

Once everything setup follow below link to setup SUSE CaaSP cluster.

https://documentation.suse.com/suse-caasp/4.1/single-html/caasp-deployment/index.html#deployment_bare_metal

Once SUSE CaaSP cluster is up and running follow below link to setup SUSE CAP.

https://documentation.suse.com/suse-cap/1.5/single-html/cap-guides/index.html#cha-cap-depl-caasp