Last time…

This is a series of articles dealing with SUSE AI, please take a look at the first article here.

In the last article, we explored OpenWebUI and SUSE Security, a great tool for monitoring the communications of your entire K8s cluster.

Installing SUSE Observability

Before starting

Please make sure you have the following requirements:

• A resolvable DNS entry for your Observability main UI ingress, like “suse-observability.mydomain”
• A resolvable DNS entry for your Open Telemetry collector ingress, like “suse-observability-otlp.mydomain”
• To avoid errors about docker sockets, please be aware that there should not be a file/directory with the name /var/run/docker.sock in the host OS of each node, or else the node-agent will get confused and not use the correct RKE2/containerd socket.
• The promtail pod WILL FAIL due to the amount of inotify probes on some host OSes. To avoid this, configure a higher number of inotify probes on sysctl:

# vi /etc/sysctl.d/50-inotify-promtail.conf
fs.inotify.max_user_instances = 512
# sysctl -p
# sysctl fs.inotify.max_user_instances
fs.inotify.max_user_instances = 512

Installation

Add the repository:

# helm repo add suse-observability  https://charts.rancher.com/server-charts/prime/suse-observability
# helm repo update

Create the values files:

# export VALUES_DIR=.
# helm template --set license='<SUSE Observability License key>' --set  baseUrl='<FQDN for your Observability main UI>' --set sizing.profile='10-nonha' suse-observability-values  suse-observability/suse-observability-values --output-dir $VALUES_DIR

In this case, we’re using the smallest profile for a standalone Observability server in a non-HA configuration. See the Observability docs for more options.

Run the installation:

# helm upgrade   --install   --namespace suse-observability   --values $VALUES_DIR/suse-observability-values/templates/baseConfig_values.yaml   --values $VALUES_DIR/suse-observability-values/templates/sizing_values.yaml suse-observability suse-observability/suse-observability

Creating the Ingresses for SUSE Observability with valid certificates

Create the issuer for cert-manager:

# cat issuer.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: letsencrypt-prod
spec:
 acme:
   # The ACME server URL
   server: https://acme-v02.api.letsencrypt.org/directory
   # Email address used for ACME registration
   email: <YOUR EMAIL ADDRESS>
   # Name of a secret used to store the ACME account private key
   privateKeySecretRef:
     name: letsencrypt-prod
   # Enable the HTTP-01 challenge provider
   solvers:
     - http01:
         ingress:
           ingressClassName: nginx

Create the Issuer object:

# kubectl apply -n suse-observability -f issuer.yaml  
issuer.cert-manager.io/letsencrypt-prod created

Verify that it’s been created correctly:

# kubectl describe issuer letsencrypt-prod -n suse-observability

You should see these values under the “Status:” section:

Reason:                ACMEAccountRegistered
   Status:                True
   Type:                  Ready

Check if the certificate has been successfully issued:

# kubectl get certificate -n suse-observability
NAME                       READY   SECRET                     AGE
observability-tls-secret   True    observability-tls-secret   13s

If it’s not “True”, check the issuer pod logs.

Creating the Ingresses:

First, let’s create the main UI ingress. Notice the annotations which are necessary for automatically creating the certificates and allowing larger POST sizes.

# cat observability-ui.yaml  
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
   cert-manager.io/issuer: letsencrypt-prod
   nginx.ingress.kubernetes.io/proxy-body-size: 50m
 name: observability-ui
 namespace: suse-observability
spec:
 ingressClassName: nginx
 rules:
   - host: <FQDN to the Observability UI>
     http:
       paths:
         - backend:
             service:
               name: suse-observability-router
               port:
                 number: 8080
           path: /
           pathType: Prefix
 tls:
   - hosts:
       - <FQDN to the Observability UI>
     secretName: observability-tls-secret

# kubectl apply -n suse-observability -f observability-ui.yaml

Now, let’s create the Open Telemetry collector ingress. Also notice the annotations necessary for it to create certificates and allow the protocols used.

# cat opentelemetry-collector.yaml  
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
   cert-manager.io/issuer: letsencrypt-prod
   nginx.ingress.kubernetes.io/backend-protocol: GRPC
   nginx.ingress.kubernetes.io/proxy-body-size: 50m
 name: opentelemetry-collector
 namespace: suse-observability
spec:
 ingressClassName: nginx
 rules:
   - host: <FQDN to the Open Telemetry collector>
     http:
       paths:
         - backend:
             service:
               name: suse-observability-otel-collector
               port:
                 number: 4317
           path: /
           pathType: Prefix
 tls:
   - hosts:
       - <FQDN to the Open Telemetry collector>
     secretName: observability-tls-secret

# kubectl apply -n suse-observability -f opentelemetry-collector.yaml

If everything deploys correctly, you should now be able to access the FQDN to your SUSE Observability UI.

Installing the Observability Agent

First, get the administrator password from  suse-observability-values/templates/baseConfig_values.yaml. It’s mentioned in the last comment.

Access the FQDN of your SUSE Observability ingress, and authenticate with “admin” and the password from the last step.

Open the menu at the top left corner, select “StackPacks”, then select “Kubernetes”. Click on the Install button and follow the instructions provided. There will be a helm command to install it to your cluster with the proper API token and values.

Activating the Rancher UI Observability extension

First, install the CLI for StackState if you haven’t already. To do this, go to the WebUI for SUSE Observability and select “CLI” from the left menu. The command with the API token will be created for you:

# curl -o- https://dl.stackstate.com/stackstate-cli/install.sh | STS_URL="https://<FQDN to the Observability UI>" STS_API_TOKEN="<API TOKEN>" bash
Trying to install StackState CLI to /usr/local/bin
Installing: https://dl.stackstate.com/stackstate-cli/v3.0.5/stackstate-cli-3.0.5.linux-x86_64.tar.gz
✅ Connection verified to https://<FQDN to the Observability UI> (Platform version: 2.3.1)                                   
✅ Saved context: 'default'
Success! Type sts to get started!

Now, create the Service Token for the extension:

# sts service-token create --name rancher-prime-observability --roles stackstate-k8s-troubleshooter
✅ Service token created: svctok-<LONG RANDOM STRING>                                                                         

Go to the Extensions icon in Rancher UI, add the Observability extension, then reload.

Fill in the URL for the Observability, and the service token created above.

Go to the main cluster dashboard and you’ll see the new section for Observability.

What’s Next?

In the next article, we’ll be adding the GPU and LLM metrics to our SUSE Observability installation, and we’ll also explore a bit of the WebUI. See you then!