Native Kubernetes Monitoring, Part 2: Scaling and Life Cycle Management
This article is a follow up to Native Kubernetes Monitoring, Part One. In this chapter we’ll finish the two remaining demos for the other built-in tools, Probes and Horizontal Pod Autoscaler (HPA).
Prerequisites for the Demo
If you were following previous chapter then your Rancher instance and Kubernetes cluster should be both up and running. If not, please check the article in order to setup the environment.
As a short reminder, we previously mentioned that Kubernetes ships with some built-in tools to monitor the cluster and the many moving parts that form a deployment:
- Kubernetes dashboard: gives an overview of the resources running on your cluster. It also gives a very basic means of deploying and interacting with those resources.
- cAdvisor: is an open source agent that monitors resource usage and analyzes the performance of containers.
- Liveness and Readiness Probes: actively monitor the health of a container.
- Horizontal Pod Autoscaler: increases the number of pods if needed based on information gathered by analyzing different metrics.
We’ve seen the first two in action, so let’s take a look at the remaining tools.
Probes
There are two kind of health checks: liveness and readiness probes.
Readiness probes let Kubernetes know when an app is ready to serve traffic. Kubernetes will only allow a service to send traffic to the pod once the probe passes. If the probe fails, Kubernetes will stop sending traffic to that Pod until it passes again.
These kinds of probes are useful when you have an application which takes some appreciable amount of time to start. The service won’t work until the probe completes successfully even when the process has already started. Be default, Kubernetes will start sending traffic as soon as the process inside the container is started, but with a readiness probe, Kubernetes will wait until the app is fully started before allowing services to route traffic.
Liveness probes let Kubernetes know if an app is alive or not. If it is alive, no action is taken. If the app is dead, Kubernetes will remove the pod and start a new one to replace it. These probes are useful when you have an app that may hang indefinitely and stop serving requests. Because the process is still running, by default, Kubernetes will continue sending requests to the pod. With these probes, Kubernetes will detect that the app is no longer serving requests and will restart the pod.
For both liveness and readiness checks, the following types of probes are available:
- http: The most command type of custom probe. Kubernetes pings a path and if it gets a http response in 200-300 range, it will mark the pod as healthy.
- command: When using this probe, Kubernetes will run a command inside of one of the pod’s containers. If the command returns an exit code of 0, the container will be marked healthy.
- tcp: Kubernetes will try to establish a TCP connection on a specified port. If it’s able to establish the connection, the container is marked healthy.
When configuring probes, the following parameters can be provide:
initialDelaySeconds: the time to wait before sending a readiness/liveness probe when first starting a container. For liveness checks, make sure that the probe will start only after the app is ready or else your app will keep restarting.periodSeconds: how often the probe is performed (default is 10).timeoutSeconds: the number of seconds for a probe to timeout (default is 1).successThreshold: the minimum consecutive successful checks for a probe to be considered successful.failureThreshold: the amount of failed probe attempts before giving up. Giving up on a liveness probe causes Kubernetes to restart the pod. For readiness probes, the pod will be marked as unready.
Demonstrating a Readiness Probe
In this section, we will be playing with a readiness probe, configured using the command check. We will have a deployment of two replicas using the default nginx container. No traffic will be sent to the pods until a file called /tmp/healthy is found within the containers.
First, create a readiness.yaml file with the following contents:
cat readiness.yamlapiVersion: apps/v1
kind: Deployment
metadata:
name: readiness-demo
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx
name: nginx
ports:
- containerPort: 80
readinessProbe:
exec:
command:
- ls
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: lb
spec:
type: LoadBalancer
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: nginxNext, apply the YAML file:
kubectl apply -f readiness.ymlWe will see a deployment and a service being created:
deployment.apps "readiness-demo" created
service "lb" createdThe pods won’t enter the READY state unless the readiness probe passes. In this case, since there is no file called /tmp/healthy, it will be marked as failed, so no traffic will be sent by the Service.
kubectl get deployments
kubectl get podsNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
readiness-demo 2 2 2 0 20s
NAME READY STATUS RESTARTS AGE
readiness-demo-6c48bbb79f-xvgsk 0/1 Running 0 23s
readiness-demo-6c48bbb79f-xvr4x 0/1 Running 0 23sFor a better understanding of what is happening, we will modify the default nginx index page for the two pods. When requested, the first one will show 1 as response and the second one will show 2 as a response.
Replace the specific pod names in the commands below with the ones created by the deployment on your machine:
kubectl exec -it readiness-demo-6c48bbb79f-xvgsk -- bash -c "echo 1 > /usr/share/nginx/html/index.html"
kubectl exec -it readiness-demo-6c48bbb79f-xvr4x -- bash -c "echo 2 > /usr/share/nginx/html/index.html"Let’s create the required file in our first pod so that it transitions into the READY state and can be routed there:
kubectl exec -it readiness-demo-6c48bbb79f-xvgsk -- touch /tmp/healthyThe probe runs every 5 seconds, so we might need to wait a bit before seeing the result:
kubectl get pods
kubectl get podsNAME READY STATUS RESTARTS AGE
readiness-demo-6c48bbb79f-xvgsk 0/1 Running 0 23m
readiness-demo-6c48bbb79f-xvr4x 0/1 Running 0 23m
kubectl get pods
NAME READY STATUS RESTARTS AGE
readiness-demo-6c48bbb79f-xvgsk 1/1 Running 0 23m
readiness-demo-6c48bbb79f-xvr4x 0/1 Running 0 23mAs soon as the state changes we can start hitting the external IP of our load balancer:
curl 35.204.202.158We should see our modified Nginx page, which consists of a single digit identifier:
1Creating the file for the second pod will cause that pod to enter the READY state as well. Traffic will be redirected here too:
kubectl exec -it readiness-demo-6c48bbb79f-xvr4x -- touch /tmp/healthy
kubectl get podsNAME READY STATUS RESTARTS AGE
readiness-demo-6c48bbb79f-xvgsk 1/1 Running 0 25m
readiness-demo-6c48bbb79f-xvr4x 1/1 Running 0 25mAs second pod is now marked as READY, the service will send traffic to both:
curl 35.204.145.38
curl 35.204.145.38The output should indicate that traffic is being split between the two pods:
2
1Demonstrating a Liveness Probe
In this section, we will demo a liveness probe configured with a tcp check. Just as above, we will use a deployment of two replicas using the default nginx container. If port 80 inside the container is not be listening, traffic will not be sent to the container and it will be restarted.
First, let’s take a look at the liveness probe demo file:
cat liveness.yamlapiVersion: apps/v1
kind: Deployment
metadata:
name: liveness-demo
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx
name: nginx
ports:
- containerPort: 80
livenessProbe:
tcpSocket:
port: 80
initialDelaySeconds: 15
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: lb
spec:
type: LoadBalancer
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: nginxWe can apply the YAML with a single command:
kubectl apply -f liveness.yamlAfterwards, we can check the pods and, like above, modify the default Nginx page to respond with a simple 1 or 2.
First, find the names given to the pods by your Nginx deployment:
kubectl get podsNAME READY STATUS RESTARTS AGE
liveness-demo-7bdcdd47d9-l8wj8 1/1 Running 0 2m
liveness-demo-7bdcdd47d9-m825b 1/1 Running 0 2mNext, replace the default index page within each pod with a numerical identifier:
kubectl exec -ti liveness-demo-7bdcdd47d9-l8wj8 -- bash -c "echo 1 > /usr/share/nginx/html/index.html"
kubectl exec -ti liveness-demo-7bdcdd47d9-m825b -- bash -c "echo 2 > /usr/share/nginx/html/index.html"Traffic is already being redirected by the Service, so we can get responses from both pods immediately:
curl 35.204.202.158
curl 35.204.202.158Again, the response should indicate that the traffic is being split between our two pods:
2
1Now we’re ready to stop the Nginx process in the first pod to see the liveness probe in action. As soon as Kubernetes notices that the container is no longer listening on port 80, the pod’s status will change and it will be restarted. We can observe some of the statuses it transitions through until it’s running correctly again.
First, stop the web server process in one of your pods:
kubectl exec -ti liveness-demo-7bdcdd47d9-l8wj8 -- service nginx stopcommand terminated with exit code 137Now, audit the status of your pods as Kubernetes notices the probe failure and takes action to restart the pod:
kubectl get pods
kubectl get pods
kubectl get podsYou will likely see the pod transition through a number of statuses until it becomes healthy again:
NAME READY STATUS RESTARTS AGE
liveness-demo-7bdcdd47d9-l8wj8 0/1 Completed 2 7m
liveness-demo-7bdcdd47d9-m825b 1/1 Running 0 7m
NAME READY STATUS RESTARTS AGE
liveness-demo-7bdcdd47d9-l8wj8 0/1 CrashLoopBackOff 2 7m
liveness-demo-7bdcdd47d9-m825b 1/1 Running 0 7m
NAME READY STATUS RESTARTS AGE
liveness-demo-7bdcdd47d9-l8wj8 1/1 Running 3 8m
liveness-demo-7bdcdd47d9-m825b 1/1 Running 0 8mIf we request the page through our service, we will see the correct response, the modified identifier of “2”, from our second pod. However, the pod that was just created we will return the default Nginx page from the container image:
curl 35.204.202.158
curl 35.204.202.1582
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>This demonstrates that Kubernetes deployed an entirely new pod to replace the failed pod we customized earlier.
Horizontal Pod Autoscaler
The Horizontal Pod Autoscaler, or HPA, is a feature of Kubernetes that enables us to automatically scale the number of pods needed for a deployment, replication controller, or replica set based on observed metrics. In practice, CPU metrics are often the primary trigger, but custom metrics are also possible too.
Each part of the process is automated and based on measured resource usage, so no human intervention is required. The metrics are fetched from APIs like metrics.k8s.io, custom.metrics.k8s.io or external.metrics.k8s.io.
In this example, we will run a demo based on CPU metrics. A useful command that we can use in this scenario is kubectl top pods, which shows CPU and memory usage for pods.
First, let’s create a YAML file that will create a deployment with a single replica:
cat hpa.yamlapiVersion: apps/v1
kind: Deployment
metadata:
name: hpa-demo
spec:
selector:
matchLabels:
app: stress
replicas: 1
template:
metadata:
labels:
app: stress
spec:
containers:
- image: nginx
name: stressApply the deployment by typing:
kubectl apply -f hpa.yamlhorizontalpodautoscaler.autoscaling "hpa-demo" createdThis is a simple deployment with the same Nginx image and a single replica:
kubectl get deploymentNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
hpa-demo 1 1 1 1 38sNext, let’s see how we can implement an autoscaling mechanism. We can list the currently defined autoscalers with kubectl get/describe hpa. To define a new autoscaler, we could use a kubectl create command. However, the easiest way to create an autoscaler is to target an existing deployment, like this:
kubectl autoscale deployment hpa-demo --cpu-percent=50 --min=1 --max=10deployment.apps "hpa-demo" autoscaledThis will create an autoscaler for our hpa-demo deployment that we created earlier with the target CPU utilization set to 50%. The replica number is set here to be between one and ten, so maximum number of pods the autoscaler will create when there is high load is ten.
You can confirm the autoscaler’s configuration by typing:
kubectl get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-demo Deployment/hpa-demo 0%/50% 1 10 1 23sWe can alternatively define this in a YAML format to allow for easier review and change management:
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: hpa-demo
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: hpa-demo
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 50In order to see HPA in action, we need to run a command which creates load on the CPU. There are numerous ways to achieve this, but one very simple example is:
while true; do sleep 1 && date & done
First, let’s check the load on our only pod. As it currently sits idle, there is not much going:
kubectl top podsNAME CPU(cores) MEMORY(bytes)
hpa-demo-7c68555d8b-6hjvj 0m 1MiNow, let’s generate some load on the current pod. As soon as load increases we should see the HPA begin to automatically create some additional pods to handle the increased load. Let the following command run for few seconds before stopping it:
kubectl exec -it hpa-demo-7c68555d8b-6hjvj -- bash -c "while true; do sleep 1 && date & done"Check the current load on the current pod:
kubectl top podsNAME CPU(cores) MEMORY(bytes)
hpa-demo-7c68555d8b-6hjvj 104m 3MiThe HPA kicks in and starts creating extra pods. Kubernetes indicates that the deployment has been automatically scaled and now has three replicas:
kubectl get deployments
kubectl get podsNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
hpa-demo 3 3 3 2 4m
NAME READY STATUS RESTARTS AGE
hpa-demo-7c68555d8b-6hjvj 1/1 Running 0 5m
hpa-demo-7c68555d8b-9b7dn 1/1 Running 0 58s
hpa-demo-7c68555d8b-lt7t2 1/1 Running 0 58sWe can see the details of our HPA and the reason why this has been scaled to three replicas:
kubectl describe hpa hpa-demoName: hpa-demo
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"hpa-demo","namespace":"default"},"spec":{"maxRepli...
CreationTimestamp: Sat, 30 Mar 2019 17:43:50 +0200
Reference: Deployment/hpa-demo
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 104% (104m) / 50%
Min replicas: 1
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 15s horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above targetSince we stopped our load-generating command, if we wait a few minutes, the HPA should notice the decreased load and scale down the number of replicas. Without high load, there is no need for the additional two pods that were created.
Five minutes is the default amount time autoscalers wait before performing a downscale operation in Kubernetes. This limit can be overridden by adjusting the --horizontal-pod-autoscaler-downscale-delay setting, which you can learn more about in the autoscaler documentation.
Once the wait time is over, the pods for the deployment should decrease from the high-load mark:
kubectl get podsNAME READY STATUS RESTARTS AGE
hpa-demo-7c68555d8b-6hjvj 1/1 Running 0 9m
hpa-demo-7c68555d8b-9b7dn 1/1 Running 0 5m
hpa-demo-7c68555d8b-lt7t2 1/1 Running 0 5mThey should return to the baseline number:
kubectl get podsNAME READY STATUS RESTARTS AGE
hpa-demo-7c68555d8b-6hjvj 1/1 Running 0 9mIf you check the description for the HPA again, you should we see the reason for decreasing the number of replicas:
kubectl describe hpa hpa-demoName: hpa-demo
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"hpa-demo","namespace":"default"},"spec":{"maxRepli...
CreationTimestamp: Sat, 30 Mar 2019 17:43:50 +0200
Reference: Deployment/hpa-demo
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 0% (0) / 50%
Min replicas: 1
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 1
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited True TooFewReplicas the desired replica count is increasing faster than the maximum scale rate
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 5m horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 13s horizontal-pod-autoscaler New size: 1; reason: All metrics below targetConclusion
We’ve seen how Kubernetes helps us using the built-in tools to set up monitoring for our cluster. We’ve seen how it works nonstop behind the scenes to keep our apps running, but this doesn’t mean that we shouldn’t be aware of what’s happening.
Gathering all the data from the dashboard and the probes, and having all these container resources exposed by cAdvisor can help us investigate resource limitations and/or capacity planning. Monitoring Kubernetes is vital as it helps us understand the health and performance of a cluster and the applications running on top of it.
Kubernetes Monitoring in Rancher
In Rancher, you can easily monitor and graph everything in your cluster, from nodes to pods to applications. The advanced monitoring tooling, powered by Prometheus, gives you real-time data about the performance of every aspect of your cluster. Watch our online meetup on advanced Kubernetes monitoring in Rancher to see these features demoed and discussed.