Apache Mesos: Mesos + Marathon + Docker = ?

In my previous post, I talked about merging resources from multiple nodes into one using Apache Mesos.

I also talked about the 2 reasons I decided to pick up Mesos:
(1) Google Kubernetes
(2) Docker

In this post, I am going to share the method you can use to deploy Docker container on a Mesos cluster.

* Mesos and Zookeeper have to be up and running. For installation instruction, please refer to my previous post.
** I used CentOS 6.6 as the platform. So, some commands might differ on other platforms.
*** Make sure "docker-io" "(the Docker package) is installed and the daemon is running on all Mesos slave nodes.

(1) Check and update (if needed) the /etc/mesos/zk file on the node you wish to install Marathon.

vi /etc/mesos/zk ** Make sure the IP address of the Zookeeper server is written there

(2) Install Marathon (using the Mesosphere repository created in the previous post) on the node selected above (Master node is recommended for testing purpose and ease of maintenance):

yum install marathon

(3) Make sure Marathon is up and running after the installation. Otherwise, start it using:

initctl start marathon

** Use "ps -ef" command to verify the Marathon process is running with the proper IP addresses (instead of 'localhost') of the zookeeper-server and Master node. If it is not, check the /etc/mesos/zk file again and restart.

(4) Update all the Mesos slave nodes with the following:

echo 'docker,mesos' > /etc/mesos-slave/containerizers echo '5 mins' > /etc/mesos-slave/executor_registration_timeout

(5) Restart all the Mesos slave nodes:

initctl restart mesos-slave

(6) By now, you should be ready to deploy Docker container on the Mesos cluster. To do so, you have to create a JSON file for the Docker container you wish to deploy:

Eg. {    "container": {       "type": "DOCKER",       "docker": {          "image": "192.168.0.210:5000/centos63:httpd",          "network": "HOST"       }    },    "id": "centos63",    "instances": 1,    "cpus": 4,    "mem": 2048,    "uris": [],    "cmd": "/usr/sbin/httpd -DFOREGROUND" }

There are a few things to take note:

(a) For the "image" parameter, you would need to specify an image that is reachable by all of the Mesos slaves, because you would not know for sure which slave or slaves the Master will select to run the container.

** If you are using your own insecured private registry, please make sure you edit the docker "default" file (eg. /etc/sysconfig/docker or /etc/default/docker) to declare the registry as insecured and restart the Docker service (service docker restart):

other_args="--insecure-registry 192.168.0.210:5000"

(b) AFAIK, Mesos (0.21.1) only support 2 network modes now - HOST or BRIDGE. 

(c) The "cmd" parameter works like CMD in Docker.

(7) Once the JSON file is ready, you can submit to Marathon using the POST method:

curl -X POST -H "Content-Type: application/json" http://<marathon host>:8080/v2/apps -d@<JSON filename>

Marathon GUI: After the CURL command and when Mesos is deploying the container

Marathon GUI: The container is successfully deployed and RUNNING

Mesos GUI: Shows one active task running on "mesos4" slave node

Mesos GUI: Clicking on the task shows the details (it's a SANDBOX)

Mesos GUI: STDOUT and STDERR are streamed from the container to the sandbox

On 'mesos4' node, the image is downloaded from the private repo and a container is running

On 'mesos4' node, 'docker inspect <container id>' shows the networking mode is HOST as configured

On 'mesos3' node, a HTTP connection shows HTTPD container is indeed running on 'mesos4' node

Docker: Why delete does not reduce the image size?

If you wonder why your Docker image does not shrink in size after you have deleted some very large files, then you might want to read further for a brief explanation and solution!

In short, the cause of all these drama is the union filesystem (read here). If you work on an image and have performed multiple commits (either through DockerFile or manually), then you might run into this problem later in the stage. The large files might have been committed in one of the layers during the earlier stage.

Eg.

(1) We have this image called "centos63:omnibus731fp8-hm2000" which is about 2.11GB in size.

(2) Next, let us run the image and delete a very large directory in there.

(3) To persist the change, I committed the image and tagged the new image as "centos63:omnibus731fp8-hm2000-rm".

(4) Surprise, surprise! The size is still 2.11GB!?!?

(5) What happened? Let's check it out. Is the large directory still there? Nope!!!

(6) Then why didn't the "rm" work?

You can see that the large directory was most probably (sorry I can't tell for sure because the image was created with a previous very bad habit of manual commits :) added in image "28729cfb27de" that was created very early in the stage. Ever since, there were multiple commits (represented by those multiple different images in the history). 

Another thing to pay attention to is the "rm" command that was persisted in image "ae11f957b955" (latest - highlighted in BLUE box) and the size was only 7 bytes!!! It shows clearly that Docker has only recorded the "rm" command and not the result of it.

That is the "power" of an union filesystem. But sorry, your large directory was still technically in there.

(7) So, what now? How do you trim those extra fat off? The answer is with this awesome tool called docker-squash!

I executed the tool with an option to name the output image "centos63:omnibus731fp8-hm2000-squashed".
You can see how the tool successfully determined all the layers belonging to the image (check the UUID).

(8) From the output of the tool, it seems that it has successfully trimmed down the image.

(9) Let's verify whether the new image "omnibus731fp8-hm2000-squashed" is significantly smaller in size. 

It seems that the tool does deliver after all! :)

I have read that Docker Inc is working on having this feature (shrinking the image size) built-in. So, while waiting for it, you have this tool!