(1) CentOS 7.1.1503 (nodes = hdp1, hdp2 and hdp3)
(2) HDP 2.3.2 (re-using the installed Zookeeper)
(3) Docker v1.8.3
(4) golang 1.4.2
(5) etcd 2.1.1
The official documentation for Kubernetes-Mesos integration can be found here. It uses Google Compute Engine (GCE), but this blog entry will share about deploying Kubernetes-Mesos integration on a local cluster.
Ok, let's begin...
Prerequisites
(1) A working local Mesos cluster
NOTE: To build one, please refer to this.
(2) Install Docker on ALL nodes.
(a) Make sure yum has access to the official Docker repository.
(b) Execute "yum install docker-engine"
(c) Enable docker.service with "systemctl enable docker.service"
(d) Start docker.service with "systemctl start docker.service"
(3) Install "golang" on the node which you wish to install and deploy Kubernetes-Mesos integration.
(a) Execute "yum install golang"
(4) Install "etcd" on a selected node (preferably on the node that host the Kubernetes-Mesos integration for testing purposes).
(a) Execute "yum install etcd"
(b) Amend file "/usr/lib/systemd/system/etcd.service" (see below):
[FROM] ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"[TO] ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --listen-client-urls http://0.0.0.0:4001 --advertise-client-urls http://[node_ip]:4001"WHERE [node_ip] = IP Address of the node (hostname -i)
(c) Reload systemctl daemon with "systemctl daemon-reload".
(d) Enable etcd.service with "systemctl enable etcd.service".
(e) Start etcd.service with "systemctl start etcd.service".
***
Build Kubernetes-Mesos
NOTE: Execute the following on the node selected to host the Kubernetes-Mesos integration.
cd [directory to install kubernetes-mesos] git clone https://github.com/kubernetes/kubernetes cd kubernetes export KUBERNETES_CONTRIB=mesos make
***
Export environment variables
(1) Export the following environment variables:
export KUBERNETES_MASTER_IP=$(hostname -i) export KUBERNETES_MASTER=http://${KUBERNETES_MASTER_IP}:8888 export MESOS_MASTER=[zk://.../mesos] export PATH="[directory to install kubernetes-mesos]/_output/local/go/bin:$PATH"
WHERE
[zk://.../mesos] = URL of the zookeeper nodes (Eg. zk://hdp1:2181,hdp2:2181,hdp3:2181/mesos)
[directory to install kubernetes-mesos] = Directory used to perform "git clone" (see "Build Kubernetes-Mesos" above).
(2) Amend .bash_profile to make the variables permanent.
(3) Remember to source the .bash_profile file after amendment (. ~/.bash_profile).
***
Configure and start Kubernetes-Mesos service
(1) Create a cloud config file
mesos-cloud.conf in the current directory with the following contents:$ cat <<EOF >mesos-cloud.conf
[mesos-cloud]
mesos-master = ${MESOS_MASTER}
EOF
If you have not set ${MESOS_MASTER}, it should be like (example) "zk://hdp1:2181,hdp2:2181,hdp3:2181/mesos".
(2) Create a script to start all the relevant components (API server, controller manager, and scheduler):
km apiserver \ --address=${KUBERNETES_MASTER_IP} \ --etcd-servers=http://${KUBERNETES_MASTER_IP}:4001 \ --service-cluster-ip-range=10.10.10.0/24 \ --port=8888 \ --cloud-provider=mesos \ --cloud-config=mesos-cloud.conf \ --secure-port=0 \ --v=1 >apiserver.log 2>&1 & sleep 3 km controller-manager \ --master=${KUBERNETES_MASTER_IP}:8888 \ --cloud-provider=mesos \ --cloud-config=./mesos-cloud.conf \ --v=1 >controller.log 2>&1 & sleep 3 km scheduler \ --address=${KUBERNETES_MASTER_IP} \ --mesos-master=${MESOS_MASTER} \ --etcd-servers=http://${KUBERNETES_MASTER_IP}:4001 \ --mesos-user=root \ --api-servers=${KUBERNETES_MASTER_IP}:8888 \ --cluster-dns=10.10.10.10 \ --cluster-domain=cluster.local \ --contain-pod-resources=false \ --v=2 >scheduler.log 2>&1 &
NOTE:
Since CentOS uses systemd, you will hit this issue. Hence, you need to add the "--contain-pod-resources=false" to the scheduler (in bold above).
(3) Give execute permission to the script (chmod 700 <script>).
(4) Execute the script.
***
Validate Kubernetes-Mesos services
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
# NOTE: Your service IPs will likely differ
$ kubectl get services
NAME LABELS SELECTOR IP(S) PORT(S)
k8sm-scheduler component=scheduler,provider=k8sm <none> 10.10.10.113 10251/TCP
kubernetes component=apiserver,provider=kubernetes <none> 10.10.10.1 443/TCP
 |
| Kubernetest framework is registered with Mesos |
Let's spin up a pod
(1) Write a JSON pod description to a local file:
$ cat <<EOPOD >nginx.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
EOPOD
$ kubectl create -f ./nginx.yaml
pods/nginx
 |
| Submitted pod through kubectl |
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 14s
 |
| Mesos WebGUI shows active Kubernetes task |
 |
| Mesos WebGUI shows that the Kubernetes task is RUNNING |
 |
| Click through "Sandbox" link of the task to get to the "executor.log" |
 |
| An example of "executor.log" |
 |
| Connected to the node where the container is running |
Getting Kubernetes to work on Mesos can be rather challenging at this point of time.
However, it is possible and hopefully, over time, Kubernetes-Mesos integration can work seamlessly.
Have fun!
