Eirini: Mapping Code into Containers

There has been a lot of noise recently about the Project known as Eirini.  I wanted to dig into what this project was in a little more detail.

If you weren’t already aware, its goal is to allow Cloud Foundry to use any scheduler but it’s really for allowing the workloads to run directly inside Kubernetes without needing separately scheduled Diego cells to run on top of.

There are many reason that this is a fantastic change, but the first and foremost is that having a scheduler run inside another scheduler is begging for headaches. It works, but there are odd edge cases that lead to split-brain decisions.

NOTE: There is another project (Quarks) that is working on containerizing the control plane in a way that the entire platform is more portable and requiring significantly less overhead. (As in: you can run Kubernetes, the entire platform, and some work, all on your laptop)

In this blog, I want to trace some code through SUSE Cloud Application Platform to see what actually gets produced by Eirini. To do this, I’m going to run the sample application and with the sample buildpack from my last blog. I believe this will let me highlight different aspects of what’s going on.

The Process

Here is a brief overview of the process that happens when you create an application:

NOTE: This is all hugely simplified and there are many more things going on behind the scenes.

1. The almighty `cf push`. It uploads a zip file the code to the api and kicks off the build
2. The pipeline runs the code through the buildpack and spits out a “droplet” along with some config about how it should be started
3. Eirini takes that droplet and builds it into a container
4. Pushes the container to a private registry
5. Creates the appropriate object definitions for Kubernetes to be able to run the container
6. Tells the router what routes were defined and which containers are up
7. Pipes the logging from each container back to a centralized place stream

It’s important to note that the objects that it creates are StatefulSets. This allows for scaling up and down in a more efficient way as the networking is known ahead of time and routing is done by the cloud foundry router.

The Output

After deploying with cf push, I can see the app running with `cf app custom_test`

agracey@agracey-dev:~/demos/custombuildpack_sample> cf app custom_sample 
Showing health and status for app custom_sample in org suse / space dev as admin... 

name:                custom_sample 
requested state:     started 
isolation segment:   placeholder 
routes:              customsample.cap.susedemos.com 
last uploaded:       Fri 16 Aug 08:59:45 PDT 2019 
stack:               sle15 
buildpacks:          staticfile 0.0.1 
type:           web 
instances:      1/1 
memory usage:   1024M 
     state     since                  cpu    memory    disk      details 
#0   running   2019-08-16T15:59:49Z   0.0%   0 of 1G   0 of 1G

Running `kubectl get pods –neirini` shows me:

agracey@agracey-dev:~> kubectl get pods -neirini 
NAME                                         READY   STATUS    RESTARTS   AGE 
custom-sample-dev-jplx6-0                    1/1     Running   0          3m35s

If I describe this pod, I see: (somewhat redacted to not expose too much about my testbed)

agracey@agracey-dev:~> kubectl describe pod -n eirini custom-sample-dev-jplx6-0  
Name:           custom-sample-dev-jplx6-0  
Namespace:      eirini  
Priority:       0  
Node:           <redacted>  
Start Time:     Fri, 16 Aug 2019 08:59:48 -0700  
Labels:         controller-revision-hash=custom-sample-dev-jplx6-5dcc455d45  
               guid=92542a29-ff4e-47c5-9161-c4c2b5e02656  
               source_type=APP  
               statefulset.kubernetes.io/pod-name=custom-sample-dev-jplx6-0  
               version=7181e53b-5e40-4ad9-ac03-83b2aa5ac17a  
Annotations:    application_id: 92542a29-ff4e-47c5-9161-c4c2b5e02656  
               process_guid: 92542a29-ff4e-47c5-9161-c4c2b5e02656-7181e53b-5e40-4ad9-ac03-83b2aa5ac17a  
Status:         Running  
IP:             10.0.0.169  
Controlled By:  StatefulSet/custom-sample-dev-jplx6  
Containers:  
 opi:  
   Container ID:  docker://77aa1b6f293b5c49b07ab8a085de0b42a27ec8e20ba1f09f6da7a73474f5c37b  
   Image:         registry.cap.susedemos.com/cloudfoundry/ <redacted>  
   Image ID:      docker-pullable://registry.cap.susedemos.com/cloudfoundry/<redacted> 
   Port:          8080/TCP  
   Host Port:     0/TCP  
   Command:  
     dumb-init  
     --  
     /lifecycle/launch  
   State:          Running  
     Started:      Fri, 16 Aug 2019 08:59:56 -0700  
   Ready:          True  
   Restart Count:  0  
   Limits:  
     memory:  1024M  
   Requests:  
     cpu:      120m  
     memory:   1024M  
   Liveness:   tcp-socket :8080 delay=0s timeout=1s period=10s #success=1 #failure=4  
   Readiness:  tcp-socket :8080 delay=0s timeout=1s period=10s #success=1 #failure=1  
   Environment:  
     START_COMMAND:            /home/vcap/deps/0/node/bin/node app.js  
     HTTPS_PROXY:                
     no_proxy:                   
     VCAP_APP_PORT:            8080  
     VCAP_SERVICES:            {}  
     HOME:                     /home/vcap/app  
     CF_INSTANCE_PORTS:        [{"external":8080,"internal":8080}]  
     MEMORY_LIMIT:             1024m  
     HTTP_PROXY:                 
     PORT:                     8080  
     http_proxy:                 
     LANG:                     en_US.UTF-8  
     USER:                     vcap  
     NO_PROXY:                   
     VCAP_APPLICATION:         {"cf_api":"https://api.cap.susedemos.com","limits":{"fds":16384,"mem":1024,"disk":1024},"application_name":"custom_sample","application_uris":["customsample.cap.susedemos.com"],"name":"custom_sample","space_name":"dev"," 
space_id":"b4613b9e-b471-441a-8a43-145f3f9f0a3e","uris":["customsample.cap.susedemos.com"],"application_id":"92542a29-ff4e-47c5-9161-c4c2b5e02656","version":"7181e53b-5e40-4ad9-ac03-83b2aa5ac17a","application_version":"7181e53b-5e40-4ad9-ac03-83b2aa5ac 
17a"}  
     TMPDIR:                   /home/vcap/tmp  
     CF_INSTANCE_ADDR:         0.0.0.0:8080  
     VCAP_APP_HOST:            0.0.0.0  
     https_proxy:                
     PATH:                     /usr/local/bin:/usr/bin:/bin  
     CF_INSTANCE_PORT:         8080  
     POD_NAME:                 custom-sample-dev-jplx6-0 (v1:metadata.name)  
     CF_INSTANCE_IP:            (v1:status.podIP)  
     CF_INSTANCE_INTERNAL_IP:   (v1:status.podIP)  
   Mounts:                     <none>  
Conditions:  
 Type              Status  
 Initialized       True   
 Ready             True   
 ContainersReady   True   
 PodScheduled      True   
Volumes:            <none>  
QoS Class:          Burstable  
Node-Selectors:     <none>  
Tolerations:        node.kubernetes.io/not-ready:NoExecute for 300s  
                   node.kubernetes.io/unreachable:NoExecute for 300s  
Events:  
 Type    Reason     Age    From                                               Message  
 ----    ------     ----   ----                                               -------  
 Normal  Scheduled  4m32s  default-scheduler                                  Successfully assigned eirini/custom-sample-dev-jplx6-0 to <redacted>  
 Normal  Pulling    4m31s  kubelet, <redacted>  pulling image " <redacted> "  
 Normal  Pulled     4m24s  kubelet, <redacted>  Successfully pulled image "<redacted>"  
 Normal  Created    4m24s  kubelet, <redacted>  Created container  
 Normal  Started    4m24s  kubelet, <redacted>  Started container 

There are a few things that we can gleam from this.

• “ControlledBy: StatefulSet/custom-sample-dev-jplx6″ gives us the next object to look at
• We can see all the VCAP settings being passed in
• We can see both a START_COMMAND in the environment and Command in the container section
• /home/vcap/app is the working directory

Interestingly, the START_COMMAND is what was output by the buildpack and Command that’s given to the pod to start with is a layer of indirection. (TODO, asking why)

Let’s do a describe on the StatefulSet now:

agracey@agracey-dev:~> kubectl -neirini describe StatefulSet/custom-sample-dev-jplx6  
Name:               custom-sample-dev-jplx6 
Namespace:          eirini 
CreationTimestamp:  Fri, 16 Aug 2019 08:59:48 -0700 
Selector:           guid=92542a29-ff4e-47c5-9161-c4c2b5e02656,source_type=APP,version=7181e53b-5e40-4ad9-ac03-83b2aa5ac17a 
Labels:             guid=92542a29-ff4e-47c5-9161-c4c2b5e02656 
                    source_type=APP 
                    version=7181e53b-5e40-4ad9-ac03-83b2aa5ac17a 
Annotations:        application_id: 92542a29-ff4e-47c5-9161-c4c2b5e02656 
                    application_name: custom_sample 
                    application_uris: [{"hostname":"customsample.cap.susedemos.com","port":8080}] 
                    last_updated: 1565971171.0 
                    process_guid: 92542a29-ff4e-47c5-9161-c4c2b5e02656-7181e53b-5e40-4ad9-ac03-83b2aa5ac17a 
                    routes: [{"hostname":"customsample.cap.susedemos.com","port":8080}] 
                    space_name: dev 
                    version: 7181e53b-5e40-4ad9-ac03-83b2aa5ac17a 
Replicas:           1 desired | 1 total 
Update Strategy:    RollingUpdate 
  Partition:        824644288072 
Pods Status:        1 Running / 0 Waiting / 0 Succeeded / 0 Failed 
Pod Template: 
  <ALL_THE SAME STUFF FROM ABOVE> 
Events: 
  Type    Reason            Age   From                    Message 
  ----    ------            ----  ----                    ------- 
  Normal  SuccessfulCreate  18m   statefulset-controller  create Pod custom-sample-dev-jplx6-0 in StatefulSet custom-sample-dev-jplx6 successful

Here we can see that we have only one replica being requested. But there’s not much else that we didn’t already know.

Next, lets’ scale up and see what happens.

agracey@agracey-dev:~> cf scale custom_sample -i 5 
Scaling app custom_sample in org suse / space dev as admin... 
OK

Seemed to work, let’s look at our `kubectl get pods-n eirini` again:

agracey@agracey-dev:~> kubectl get pods -neirini             
NAME                   READY   STATUS    RESTARTS   AGE 
custom-sample-dev-jplx6-0                    1/1     Running   0          23m 
custom-sample-dev-jplx6-1                    1/1     Running   0          53s 
custom-sample-dev-jplx6-2                    1/1     Running   0          53s 
custom-sample-dev-jplx6-3                    1/1     Running   0          53s 
custom-sample-dev-jplx6-4                    1/1     Running   0          53s

We can see that there are now 5 replicas. Describing the StatefulSet gives us what we would expect:

<same as before>
Replicas:           5 desired | 5 total 
<same as before>

Now let’s log into one of these pods and see what’s there. We can do this with

kubectl exec -it custom-sample-dev-jplx6-0 -neirini -- /bin/bash

There are three things to look at:

• The working directory is set to /home/vcap/app/
• The script being run on startup is /lifecycle/launch
• The node binary is in /home/vcap/deps/0/node/bin/

Let’s look at /home/vcap/app:

vcap@custom-sample-dev-jplx6-0:/$ ls /home/vcap/app/ 
app.js  function.js  package.json  package-lock.json

This is the same as what we put into the working directory during the buildpack.

There are two scripts inside the /lifecycle/ directory:

vcap@custom-sample-dev-jplx6-0:/$ ls /lifecycle/
launch  launcher

We can see that launch is what’s being run by the podspec above.  If we run this script manually it’ll pick up the existing environment. It complains because there’s a port collision which is expected as the port is being taken by the real process that’s already running.

vcap@custom-sample-dev-jplx6-0:/$ /lifecycle/launch 
ARGS: [/lifecycle/launch] 
Server Listening on  8080 
events.js:177 
      throw er; // Unhandled 'error' event 
      ^ 
 
Error: listen EADDRINUSE: address already in use :::8080 
    at Server.setupListenHandle [as _listen2] (net.js:1228:14) 
    at listenInCluster (net.js:1276:12) 
    at Server.listen (net.js:1364:7) 
    at Object.<anonymous> (/home/vcap/app/app.js:73:4) 
    at Module._compile (internal/modules/cjs/loader.js:776:30) 
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10) 
    at Module.load (internal/modules/cjs/loader.js:643:32) 
    at Function.Module._load (internal/modules/cjs/loader.js:556:12) 
    at Function.Module.runMain (internal/modules/cjs/loader.js:839:10) 
    at internal/main/run_main_module.js:17:11 
Emitted 'error' event at: 
    at emitErrorNT (net.js:1255:8) 
    at processTicksAndRejections (internal/process/task_queues.js:74:11) { 
  code: 'EADDRINUSE', 
  errno: 'EADDRINUSE', 
  syscall: 'listen', 
  address: '::', 
  port: 8080 
}

As we can see, it simply starts the process based on the environment given.

Simplicity

To the user, Eirini isn’t very complicated. This is by design since it needs to be a drop in replacement for existing cloud foundry deployments.

It does make some really important progress towards modernizing and minimizing the platform. It strips away a lot of redundancy and potential pitfalls to give a very powerful system that makes your developers lives much easier by moving complexity to a place where it can be managed appropriately.