Friday, February 27, 2015

Apache Mesos: When All Becomes One

First of all, for the benefits of those unfamiliar with Apache Mesos, this is the "what is" taken from its official website:

What is Mesos?

A distributed systems kernel

Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments.

Beside the fact that I have always wanted to learn a technology that can merge all the resources available on my multiple machines into one, I am out to learn Apache Mesos for the following 2 reasons (currently):
(1) Google Kubernetes
(2) Docker

Installing Apache Mesos isn't too hard if you follow the instruction available on its official website, but I would like to share an alternative installation method which I found is more straightforward:

* Instructions only suitable for RHEL/CentOS 6 (tested on CentOS 6.6). For other platforms, refer here.
** Run all instructions as 'root' user for simplicity.


(1) On the node or VM image that you would like to designate as the Master and all slave nodes, execute the following command to create the Mesosphere repository:

rpm -Uvh http://repos.mesosphere.io/el/6/noarch/RPMS/mesosphere-el-repo-6-2.noarch.rpm

(2) On the Master and all the slave nodes, install Mesos:

yum -y install mesos

(3) Even though you only plan to have a single Master node, it is advisable to install Zookeeper (just in case you want to expand in the future):

rpm -Uvh http://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm 

yum -y install zookeeper-server

* You can install the Zookeeper server on either the Master node (preferred for ease of maintenance) or any of the slave node.
** You need to have Java installed for Zookeeper to work properly.

(4) On the Master node, initialize Zookeeper:

service zookeeper-server init 

echo 1 | sudo tee -a /var/lib/zookeeper/myid >/dev/null

(5) On the Master node, stop and disable mesos-slave:

initctl stop mesos-slave

cd /etc/init/ 

mv mesos-slave.conf mesos-slave.disable

(6) On all the slave nodes, stop and disable mesos-master:

initctl stop mesos-master

cd /etc/init/ 

mv mesos-master.conf mesos-master.disable

(7) On the Master node, set the IP address:

echo <IP of the Master node> | sudo tee /etc/mesos-master/ip

(8) On the Master node, set the name of the cluster:

echo <cluster name> | sudo tee /etc/mesos-master/cluster 

(9) On the Master and all slave nodes, set the URL of the Zookeeper server:

echo zk://<IP of the Zookeeper server>:2181/mesos | sudo tee /etc/mesos/zk

(10) On all the slave nodes, set their respective IP address:

echo <IP of the Slave node>  | sudo tee /etc/mesos-slave/ip  

(11) On the Master node, restart mesos-master and Zookeeper (if it is installed there):

service zookeeper restart 

initctl restart mesos-master

(12) On all the slave nodes, restart mesos-slave:

initctl restart mesos-slave

(12) Verify that the Master is running and all slaves are registered with it:

http://<IP of the Master node>:5050


When the Master is first initialized
When the first slave joined
When the second slave joined
When the third slave joined
All the slaves

No comments:

Post a Comment