tag:blogger.com,1999:blog-49439346999222595682024-03-05T21:31:23.999+08:00Anything Linux, Big Data, Containerization, Graph Database and CloudI am a technology enthusiast and application support engineer by profession. All posts shared in this blog is based on my best knowledge. The objective of this blog is to share information and experience in hope that it will help save some of you some time while exploring these fun and exciting technologies.Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.comBlogger23125tag:blogger.com,1999:blog-4943934699922259568.post-39734571147214560832017-07-23T15:46:00.000+08:002017-07-24T09:39:21.645+08:00Perfect End-to-End Data Platform for Today (IMHO)Hi Guys,<br />
I am back (like Superman :)!!<br />
<br />
I had been bogged down with work, family and health issues for past 1+ year. But then, I decided that my passion for technology is too great for me to stop sharing :)<br />
<br />
So, here we go again...this time, get ready for more interesting stuff that I learned in the past year.<br />
<br />
***<br />
<br />
Now, let me start by sharing a data platform diagram that I have based my past year's learning on...<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihzPngfZVvUUEO5xM8PMAK6T780OzL9bbs_Ri1uGVBrCYngQJsKLRMGeoUtmnX_o_-Z7HmT-wNMTHFu0eXqMnXgglTewML_WkvhYlYctU1mGJVvX64gFG4dbcH50OrJ5QlrFk47r2kmGo/s1600/dataflow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="578" data-original-width="1600" height="231" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihzPngfZVvUUEO5xM8PMAK6T780OzL9bbs_Ri1uGVBrCYngQJsKLRMGeoUtmnX_o_-Z7HmT-wNMTHFu0eXqMnXgglTewML_WkvhYlYctU1mGJVvX64gFG4dbcH50OrJ5QlrFk47r2kmGo/s640/dataflow.png" width="640" /></a></div>
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">** At the DATA STORAGE layer, I am still contemplating which NoSQL database to select.</span><br />
<br />
<br />
Before you get too excited or disappointed, let me just say that this is a platform that "I" think is perfect (and I am still constantly re-adjusting by adding/removing technologies) for today.<br />
<br />
Each and individual selected technology is the cream of the crop. I do not declare that they are perfect, but then they are good enough to perform the work consistently well.<br />
<br />
Everyone of them plays a specific role from the source to the destination (reporting).<br />
<br />
I am still learning most (if not all) of them, so stay tuned for more exciting sharing!Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-6094822040175039772016-01-26T08:00:00.000+08:002016-01-26T08:52:58.619+08:00Docker: How to start a container on a specific node in a Swarm cluster?If you are using a Docker Swarm cluster and are wondering how you can start a container on a specific node, this post is for you!<br />
<br />
The short answer is to custom label the docker node(daemon):<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">[root@hdp1 ~]# cat /etc/sysconfig/docker
OPTIONS="-H tcp://0.0.0.0:2375 </span><span style="color: red;">--label nodename=n1.manfrix.com</span><span style="color: #333333;">"
[root@hdp1 ~]#
</span></pre>
<b>NOTE</b>: If you want to know more about the "label" option of docker daemon, you can refer <a href="https://docs.docker.com/engine/reference/commandline/daemon/" target="_blank">here</a>.<br />
<br />
Make sure you restart the docker service after the changes:<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">systemctl restart docker
</span></pre>
<br />
Verify that the daemon is now started with a label:<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">[root@hdp1 ~]# ps -ef | grep docker
root 50961 1 0 20:38 ? 00:00:02 /usr/bin/docker daemon -H fd:// -H tcp://0.0.0.0:2375 </span><span style="color: red;">--label nodename=n1.manfrix.com</span></pre>
<br />
Make sure you remember to re-join the nodes to the swarm cluster after you restarted docker:<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">docker run -d swarm join --addr=[IP of the node]:2375 etcd://[IP of etcd host]:[port]/[optional path prefix]
</pre>
<br />
After you have labeled all the nodes (daemons), then you can proceed to test:<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">[root@hdp1 ~]# docker -H tcp://localhost:9999 run -d --name centos-1 -p 80 </span><span style="color: red;">-e constraint:nodename==n3.manfrix.com</span><span style="color: #333333;"> centos /bin/bash
82d42f3052da181ebb876d79e2aeeb68787c17045c625367cced067107f3cb08
[root@hdp1 ~]# docker -H tcp://localhost:9999 run -d --name nginx-1 -p 80 </span><span style="color: red;">-e constraint:nodename==n2.manfrix.com</span><span style="color: #333333;"> nginx
68664b5046b1dc031b015c9241a2f16f1e663f0b384d395d810d36b46f317839
</span></pre>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFoaYdSy1CxhGRzn0YjCq0Ql3hjjn4osTxGnjW63oKQD0Cv3XGgKZ0BUR-HG9CzTs37OvFuEbKU3OUCbKgL_S0yZl-WxKAW_Mgqyzb2TVdG88HS2Blvw38XrxtMGKN6Rwp-EqlORzw6a4/s1600/swarm-label-edit.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFoaYdSy1CxhGRzn0YjCq0Ql3hjjn4osTxGnjW63oKQD0Cv3XGgKZ0BUR-HG9CzTs37OvFuEbKU3OUCbKgL_S0yZl-WxKAW_Mgqyzb2TVdG88HS2Blvw38XrxtMGKN6Rwp-EqlORzw6a4/s1600/swarm-label-edit.png" width="100%" /></a></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
For more information about Swarm node constraints, you can refer <a href="https://docs.docker.com/swarm/scheduler/filter/" target="_blank">here</a>.</div>
<div>
<br /></div>
<div>
<br /></div>
Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-17849169219330882672016-01-15T21:21:00.000+08:002016-01-15T21:54:48.009+08:00Docker: /etc/default/docker or /etc/sysconfig/docker does not work anymore under systemd?If you question why all of the sudden your OPTIONS under /etc/default/docker or /etc/sysconfig/docker no longer work, I can assure you that you are not alone!<br />
<br />
Let's put it simply, it is due to the /usr/lib/systemd/system/docker.service shipped with newer version of Docker (most probably v1.7 and above).<br />
<br />
The file looks like this now =><br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;"><b>[Unit]</b>
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
<b>[Service]</b>
Type=notify
</span><span style="color: red;">ExecStart=/usr/bin/docker daemon -H fd://</span><span style="color: #333333;">
MountFlags=slave<
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
</span><b style="color: #333333;">[Install]</b><span style="color: #333333;">
WantedBy=multi-user.target
</span></pre>
<br />
You can see clearly that it no longer contains any environment variable for you to customize stuff.<br />
<br />
To fix that, we need to create a systemd drop-in file for docker.service.<br />
<br />
(1) Create a directory "/etc/systemd/system/docker.service.d" on <b>ALL</b> servers.<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">mkdir /etc/systemd/system/docker.service.d
</span></pre>
<br />
(2) In the directory, create a file - "local.conf" - using your favorite text editor with the following lines:<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">[Service]
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// $OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$BLOCK_REGISTRY \
$INSECURE_REGISTRY
</span></pre>
<br />
(3) Create or edit the /etc/sysconfig/docker or /etc/sysconfig/docker-storage or /etc/sysconfig/docker-network to specify your customized options.<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">Eg.
<b>[/etc/sysconfig/docker]</b>
OPTIONS="-H tcp://0.0.0.0:2375 --label nodename=node1.mytestmac.com"
</span></pre>
<br />
<br />
(4) Reload systemd.<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">systemctl daemon-reload
</span></pre>
<br />
(5) Verify that docker.service is now aware of its environment files.<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">systemctl show docker | grep -i env
Eg.
[root@hdp1 ~]# systemctl show docker | grep -i env
EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes)
EnvironmentFile=/etc/sysconfig/docker-storage (ignore_errors=yes)
EnvironmentFile=/etc/sysconfig/docker-network (ignore_errors=yes)
[root@hdp1 ~]#
</span></pre>
<br />
(6) Restart docker.<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">systemctl restart docker
</span></pre>
<br />
(7) Verify the changes.<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">ps -ef | grep docker
</span></pre>
<br />
Hope that helps to resolve your headaches! :)Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-33860932125179750642015-11-07T22:36:00.004+08:002015-11-07T22:36:55.516+08:00Docker: How to setup Swarm with etcd?Alright, let me start by sharing some information about my test environment:<br />
(i) 3 nodes (hdp1, hdp2 and hdp3) running CentOS 7.1.1503<br />
(ii) Docker v1.9.0<br />
(iii) etcd 2.1.1 (running only on hdp1) listening for client connection at <b>port 4001</b><br />
<br />
Node "hdp1" will be Swarm Master.<br />
<br />
(1) Firstly, let's reconfigure the Docker daemon running on <b>ALL</b> the nodes by adding "-H tcp://0.0.0.0:2375".<br />
<br />
Eg.<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">vi /usr/lib/systemd/system/docker.service
<b>[Amend the line]</b> ExecStart=/usr/bin/docker daemon -H fd:// <b>-H tcp://0.0.0.0:2375</b>
</pre>
<br />
<br />
(2) You would have to reload systemctl and restart docker on <b>ALL</b> the nodes after the above changes.<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">systemctl daemon-reload
systemctl restart docker
<b>[To verify]</b> ps -ef | grep docker</pre>
<br />
(3) Make sure "etcd" is running. If not, please start it and make sure it's listening on the intended port (4001 in this case).<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">systemctl start etcd
<b>[To verify]</b> netstat -an | grep [port] | grep LISTEN
</pre>
<br />
(4) On the other nodes (non-swarm-master - in this case "hdp2" and "hdp3"), execute the following command to join them to the cluster:<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">docker run -d swarm join --addr=[IP of the node]:2375 etcd://[IP of etcd host]:[port]/[optional path prefix]
</pre>
<br />
<b>WHERE</b> (in this example)<br />
[IP of the node] = IP address of node "hdp2" and "hdp3"<br />
[IP of etcd host] = IP address of node "hdp1" where the only etcd instance is running<br />
[port] = Port that etcd uses to listen to incoming client connection (in this example = 4001)<br />
[optional path prefix] = Path that etcd uses to store data about the registered Swarm nodes<br />
<br />
The final command:<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">docker run -d swarm join --addr=192.168.0.171:2375 etcd://192.168.0.170:4001/swarm
docker run -d swarm join --addr=192.168.0.172:2375 etcd://192.168.0.170:4001/swarm</pre>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcIJljbU3MvjEiyb4hLZWNYCr-HjL9QK284x-DNCsAOOfnLYG8raEkhDa-N46iuZES6D-b8m9C0ZwduEsSrGzQAJfBaCDE2B68gdNEd2k_tiyKRRgBe0UErJXMj6uXNDbW8mw5r3TtPXg/s1600/swarm4.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcIJljbU3MvjEiyb4hLZWNYCr-HjL9QK284x-DNCsAOOfnLYG8raEkhDa-N46iuZES6D-b8m9C0ZwduEsSrGzQAJfBaCDE2B68gdNEd2k_tiyKRRgBe0UErJXMj6uXNDbW8mw5r3TtPXg/s1600/swarm4.png" width="100%" /></a></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGMbGay3vhhIWQrXJuiLLKnyNnru5uU7A86ud_iZOjiUBQMsy9qhyNIkUAy4K3RNnccNqaSV9Kkm2Zmu8bK6DP1cICw-xiLOsFHSzAS7qVbZvgoHoxbaLvxKXN150io4jCrl-4i-d9P_M/s1600/swarm5.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGMbGay3vhhIWQrXJuiLLKnyNnru5uU7A86ud_iZOjiUBQMsy9qhyNIkUAy4K3RNnccNqaSV9Kkm2Zmu8bK6DP1cICw-xiLOsFHSzAS7qVbZvgoHoxbaLvxKXN150io4jCrl-4i-d9P_M/s1600/swarm5.png" width="100%" /></a></td></tr>
</tbody></table>
<br />
(5) You can verify that the nodes are registered with the following command:<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">etcdctl ls /swarm/docker/swarm/nodes
</pre>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoKfNiA8w6tba52qTukQH7JldET1jYe6S-6SbibDk4XOuZAiWVO8aimpxxB6eyQP4XJXdJcRDo4hv8U9ucP20FjQnE9_-blK4ru6XVxaTlerWrmqab2V43g3YC-nnccu-VcBHOUJmVPKs/s1600/swarm1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoKfNiA8w6tba52qTukQH7JldET1jYe6S-6SbibDk4XOuZAiWVO8aimpxxB6eyQP4XJXdJcRDo4hv8U9ucP20FjQnE9_-blK4ru6XVxaTlerWrmqab2V43g3YC-nnccu-VcBHOUJmVPKs/s1600/swarm1.png" width="100%" /></a></td></tr>
</tbody></table>
<br />
(6) If all nodes are registered successfully, you can now start up the Swarm Master (in this example, on node "hdp1").<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">docker run -p [host port]:2375 -d swarm manage -H tcp://0.0.0.0:2375 etcd://[IP of etcd host]:4001/swarm
</pre>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwUQE7LdT2ro8Zwobx3Xsd4h3Jt863B8eI8SrwOqBTwriB82SuRw4L0dv99mKPythlHt8Jfd_bkRp_688Q_nfdpwiVYGvMmw2qU_QpYfsCSdnBvNA0bsNHreSCEyQwpT9zVceSDP8ROjQ/s1600/swarm2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwUQE7LdT2ro8Zwobx3Xsd4h3Jt863B8eI8SrwOqBTwriB82SuRw4L0dv99mKPythlHt8Jfd_bkRp_688Q_nfdpwiVYGvMmw2qU_QpYfsCSdnBvNA0bsNHreSCEyQwpT9zVceSDP8ROjQ/s1600/swarm2.png" width="100%" /></a></td></tr>
</tbody></table>
<br />
<b>WHERE</b> [in this example]<br />
[host port] = 9999 (or any other free network port you selected - you will use this port to communicate with Swarm Master)<br />
[IP of etcd host = IP address of node "hdp1" where the only etcd instance is running<br />
<br />
Eg.<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">docker run -p 9999:2375 -d swarm manage -H tcp://0.0.0.0:2375 etcd://192.168.0.170:4001/swarm
</pre>
<br />
(7) To verify that the Swarm cluster is now working properly, execute the following command:<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">docker -H [IP of the host where Swarm Master is running]:[port] info
</pre>
<br />
<b>WHERE</b> (in this example)<br />
[IP of the host where Swarm Master is running] = Node "hdp1" (192.168.0.170)<br />
[port] = 9999 (refer to Step (6) above)<br />
<br />
<b>NOTE</b>: You can use any Docker CLI as normal with Swarm cluster =><br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">docker -H [IP of the host where Swarm Master is running]:[port] ps -a
docker -H [IP of the host where Swarm Master is running]:[port] logs [container id]
docker -H [IP of the host where Swarm Master is running]:[port] inspect [container id]
</pre>
<br />
(8) Let's spin up a container and see how your Swarm cluster handles it.<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">docker -H [IP of the host where Swarm Master is running]:[port] run -d --name nginx-1 -p 80 nginx
</pre>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlS6KSZMJ4OleRV-U8g9XGMJlkx-XufMlt7wfRC7pmXVjKLdEMm2M8I34UiTRoYZynCWqtyvdNouIv9TCvGSkRFfRREZJ7_cXqmFiR6MTt9SXXBlOosNH0F4eQnnfsyrNjKVDnH0j_nOs/s1600/swarm3.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlS6KSZMJ4OleRV-U8g9XGMJlkx-XufMlt7wfRC7pmXVjKLdEMm2M8I34UiTRoYZynCWqtyvdNouIv9TCvGSkRFfRREZJ7_cXqmFiR6MTt9SXXBlOosNH0F4eQnnfsyrNjKVDnH0j_nOs/s1600/swarm3.png" width="100%" /></a></td></tr>
</tbody></table>
<br />
(9) Let's check where the container is running.<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">docker -H [IP of the host where Swarm Master is running]:[port] ps -a
</pre>
<br />
(10) You can stop the running container by issuing:<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">docker -H [IP of the host where Swarm Master is running]:[port] stop [container id]
</pre>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-55428551239380395202015-11-03T10:43:00.002+08:002015-11-03T11:08:56.857+08:00Mesos/Kubernetes: How to install and run Kubernetes on Mesos with your local cluster?First of all, let me share with you my test environment:<br />
(1) CentOS 7.1.1503 (nodes = hdp1, hdp2 and hdp3)<br />
(2) HDP 2.3.2 (re-using the installed Zookeeper)<br />
(3) Docker v1.8.3<br />
(4) golang 1.4.2<br />
(5) etcd 2.1.1<br />
<br />
The official documentation for Kubernetes-Mesos integration can be found <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/mesos.md#prerequisites" target="_blank">here</a>. It uses Google Compute Engine (GCE), but this blog entry will share about deploying Kubernetes-Mesos integration on a local cluster.<br />
<br />
Ok, let's begin...<br />
<br />
<b><u>Prerequisites</u></b><br />
<br />
(1) A working local Mesos cluster<br />
<b>NOTE</b>: To build one, please refer to <a href="http://manfrix.blogspot.my/2015/02/apache-mesos-when-all-becomes-one.html" target="_blank">this</a>.<br />
<br />
(2) Install Docker on <b>ALL</b> nodes.<br />
(a) Make sure yum has access to the <a href="https://docs.docker.com/installation/centos/" target="_blank">official Docker repository</a>.<br />
(b) Execute "yum install docker-engine"<br />
(c) Enable docker.service with "systemctl enable docker.service"<br />
(d) Start docker.service with "systemctl start docker.service"<br />
<br />
(3) Install "golang" on the node which you wish to install and deploy Kubernetes-Mesos integration.<br />
(a) Execute "yum install golang"<br />
<br />
(4) Install "etcd" on a selected node (preferably on the node that host the Kubernetes-Mesos integration for testing purposes).<br />
(a) Execute "yum install etcd"<br />
(b) Amend file "/usr/lib/systemd/system/etcd.service" (see below):<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><div style="color: black; font-family: 'Times New Roman'; font-size: medium; line-height: normal; white-space: normal;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><b style="color: black; font-family: 'times new roman'; font-size: medium; line-height: normal; white-space: normal;">[FROM]</b>
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"</pre>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><b style="color: black; font-family: 'times new roman'; font-size: medium; line-height: normal; white-space: normal;">[TO]</b>
<span class="pl-c1" style="box-sizing: border-box; color: black;">ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --listen-client-urls http://0.0.0.0:4001 --advertise-client-urls http://[node_ip]:4001"</span></pre>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><node_ip><b style="color: black; font-family: 'times new roman'; font-size: medium; line-height: normal; white-space: normal;">WHERE</b>
<span class="pl-k" style="box-sizing: border-box; color: black;">[node_ip] = IP Address of the node (hostname -i)
</span></node_ip></span></pre>
</div>
</pre>
<br />
(c) Reload systemctl daemon with "systemctl daemon-reload".<br />
(d) Enable etcd.service with "systemctl enable etcd.service".<br />
(e) Start etcd.service with "systemctl start etcd.service".<br />
<br />
***<br />
<br />
<b><u>Build Kubernetes-Mesos</u></b><br />
<br />
<b>NOTE</b>: Execute the following on the node selected to host the Kubernetes-Mesos integration.<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">cd [directory to install kubernetes-mesos]
git clone https://github.com/kubernetes/kubernetes
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">cd</span> kubernetes
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">export</span> KUBERNETES_CONTRIB=mesos
make</pre>
<br />
***<br />
<br />
<b><u>Export environment variables</u></b><br />
<b><u><br /></u></b>
(1) Export the following environment variables:<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">export</span> KUBERNETES_MASTER_IP=<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">$(</span>hostname -i<span class="pl-pds" style="box-sizing: border-box;">)</span></span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">export</span> KUBERNETES_MASTER=http://<span class="pl-smi" style="box-sizing: border-box;">${KUBERNETES_MASTER_IP}</span>:8888
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">export</span> <span class="pl-smi" style="box-sizing: border-box;">MESOS_MASTER=[zk://.../mesos]</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">export</span> <span class="pl-smi" style="box-sizing: border-box;">PATH="[directory to install kubernetes-mesos]/_output/local/go/bin:$PATH"</span></pre>
<br />
<b>WHERE</b><br />
[zk://.../mesos] = URL of the zookeeper nodes (Eg. zk://hdp1:2181,hdp2:2181,hdp3:2181/mesos)<br />
[directory to install kubernetes-mesos] = Directory used to perform "git clone" (see "Build Kubernetes-Mesos" above).<br />
<br />
(2) Amend .bash_profile to make the variables permanent.<br />
(3) Remember to source the .bash_profile file after amendment (. ~/.bash_profile).<br />
<br />
***<br />
<br />
<b><u>Configure and start Kubernetes-Mesos service</u></b><br />
<br />
<span style="font-family: inherit;">(1) <span style="color: #333333; line-height: 25.6px;">Create a cloud config file</span><span style="color: #333333; line-height: 25.6px;"> </span><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; color: #333333; line-height: 25.6px; margin: 0px; padding: 0.2em 0px;">mesos-cloud.conf</code><span style="color: #333333; line-height: 25.6px;"> </span><span style="color: #333333; line-height: 25.6px;">in the current directory with the following contents:</span></span><br />
<div class="highlight highlight-text-shell-session" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ <span class="pl-s1" style="box-sizing: border-box;">cat <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><<</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">EOF</span> >mesos-cloud.conf</span></span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;">[mesos-cloud]</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> mesos-master = ${MESOS_MASTER}</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;">EOF</span></pre>
</div>
<b>NOTE</b>:<br />
If you have not set ${MESOS_MASTER}, it should be like (example) "zk://hdp1:2181,hdp2:2181,hdp3:2181/mesos".<br />
<br />
(2) Create a script to start all the relevant components (API server, controller manager, and scheduler):<br />
<br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-s1" style="box-sizing: border-box;">km apiserver \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --address=${KUBERNETES_MASTER_IP} \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --etcd-servers=http://${KUBERNETES_MASTER_IP}:4001 \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --service-cluster-ip-range=10.10.10.0/24 \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --port=8888 \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --cloud-provider=mesos \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --cloud-config=mesos-cloud.conf \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --secure-port=0 \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --v=1 >apiserver.log 2>&1 &
</span>
sleep 3
<span class="pl-s1" style="box-sizing: border-box;">km controller-manager \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --master=${KUBERNETES_MASTER_IP}:8888 \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --cloud-provider=mesos \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --cloud-config=./mesos-cloud.conf \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --v=1 >controller.log 2>&1 &</span>
sleep 3
<span class="pl-s1" style="box-sizing: border-box;">km scheduler \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --address=${KUBERNETES_MASTER_IP} \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --mesos-master=${MESOS_MASTER} \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --etcd-servers=http://${KUBERNETES_MASTER_IP}:4001 \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --mesos-user=root \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --api-servers=${KUBERNETES_MASTER_IP}:8888 \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --cluster-dns=10.10.10.10 \</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --cluster-domain=cluster.local \</span>
<span class="pl-mo" style="box-sizing: border-box; color: red;"> <b>--contain-pod-resources=false \</b></span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;"> --v=2 >scheduler.log 2>&1 &</span></pre>
<br />
<b>NOTE</b>:<br />
Since CentOS uses systemd, you will hit this <a href="https://github.com/mesosphere/kubernetes-mesos/issues/460" target="_blank">issue</a>. Hence, you need to add the "--contain-pod-resources=false" to the scheduler (in bold above).<br />
<br />
(3) Give execute permission to the script (chmod 700 <script>).<br />
(4) Execute the script.<br />
<br />
***<br />
<br />
<b><u>Validate Kubernetes-Mesos services</u></b><br />
<div class="highlight highlight-text-shell-session" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ <span class="pl-s1" style="box-sizing: border-box;">kubectl get pods</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;">NAME READY STATUS RESTARTS AGE</span></pre>
</div>
<div class="highlight highlight-text-shell-session" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"># <span class="pl-s1" style="box-sizing: border-box;"><b>NOTE</b>: Your service IPs will likely differ</span>
$ <span class="pl-s1" style="box-sizing: border-box;">kubectl get services</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;">NAME LABELS SELECTOR IP(S) PORT(S)</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;">k8sm-scheduler component=scheduler,provider=k8sm <none> 10.10.10.113 10251/TCP</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;">kubernetes component=apiserver,provider=kubernetes <none> 10.10.10.1 443/TCP</span></pre>
</div>
(4) Lastly, look for Kubernetes in the Mesos web GUI by pointing your browser to http://[mesos-master-ip:port]. Go to the Frameworks tab, and look for an active framework named "Kubernetes".<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEip8ZzcEmpxf906H2znZSHsZdhmDAvHkr-0Ffa1p_ousJ-qnchcmSiAD7V4ZWbhdZ3MgFOJJi0kLTg7v51wIpVJSaAt8p7VeNh1FFRUpZ2qhNPWSvaWz2JTlhHhXrxLhIy51gJsdx_LcME/s1600/km1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEip8ZzcEmpxf906H2znZSHsZdhmDAvHkr-0Ffa1p_ousJ-qnchcmSiAD7V4ZWbhdZ3MgFOJJi0kLTg7v51wIpVJSaAt8p7VeNh1FFRUpZ2qhNPWSvaWz2JTlhHhXrxLhIy51gJsdx_LcME/s1600/km1.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Kubernetest framework is registered with Mesos</b></span></td></tr>
</tbody></table>
***<br />
<br />
<b><u>Let's spin up a pod</u></b><br />
<br />
<span style="color: #333333; line-height: 25.6px;">(1) Write a JSON pod description to a local file:</span><br />
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ cat <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><<</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">EOPOD</span> >nginx.yaml</span></pre>
</div>
<div class="highlight highlight-source-yaml" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">apiVersion:</span> <span class="pl-s" style="box-sizing: border-box;">v1</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">kind:</span> <span class="pl-s" style="box-sizing: border-box;">Pod</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">metadata:</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">name:</span> <span class="pl-s" style="box-sizing: border-box;">nginx</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">spec:</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">containers:</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;">- <span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">name:</span> <span class="pl-s" style="box-sizing: border-box;">nginx</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">image:</span> <span class="pl-s" style="box-sizing: border-box;">nginx</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">ports:</span></span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">- <span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">containerPort:</span> 80</span>
EOPOD</pre>
</div>
<span style="color: #333333; line-height: 25.6px;">(2) Send the pod description to Kubernetes using the "kubectl" CLI:</span><br />
<div class="highlight highlight-text-shell-session" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ <span class="pl-s1" style="box-sizing: border-box;">kubectl create -f ./nginx.yaml</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;">pods/nginx</span></pre>
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmX4D2PS7CFB43r1vLKjUwez8Qgpplhz02Shp8jaSeRCDu4ukYupscMzR9jIjwfnWV1pfpRn_2_e7Ulz77vGc6VCzm_rrKcq0nY-tu6JxJ4Jb49U4jWI9IRAZA7YqXrUNDf_wy6GnXRFA/s1600/km2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmX4D2PS7CFB43r1vLKjUwez8Qgpplhz02Shp8jaSeRCDu4ukYupscMzR9jIjwfnWV1pfpRn_2_e7Ulz77vGc6VCzm_rrKcq0nY-tu6JxJ4Jb49U4jWI9IRAZA7YqXrUNDf_wy6GnXRFA/s1600/km2.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Submitted pod through kubectl</b></span></td></tr>
</tbody></table>
<span style="color: #333333; line-height: 25.6px;">(3) Wait a minute or two while Docker downloads the image layers from the internet. We can use the kubectl interface to monitor the status of our pod:</span><br />
<div class="highlight highlight-text-shell-session" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ <span class="pl-s1" style="box-sizing: border-box;">kubectl get pods</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;">NAME READY STATUS RESTARTS AGE</span>
<span class="pl-mo" style="box-sizing: border-box; color: #1d3e81;">nginx 1/1 Running 0 14s</span></pre>
</div>
<span style="color: #333333; line-height: 25.6px;">(4) Verify that the pod task is running in the Mesos web GUI. Click on the Kubernetes framework. The next screen should show the running Mesos task that started the Kubernetes pod.</span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjY072XBYMekKJtIBaPxyKVNZfeg0MoCQXuvOrdeNC2lx1cB6t1VcIjEPeVh47B49a2AgnBbgPnHjXJDPL6YwGi-kRblcwu8c7t5qhRKCiQuOqialqrA9ruD5DcCAQN0yM1HxwZe6TtS5Y/s1600/km3.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjY072XBYMekKJtIBaPxyKVNZfeg0MoCQXuvOrdeNC2lx1cB6t1VcIjEPeVh47B49a2AgnBbgPnHjXJDPL6YwGi-kRblcwu8c7t5qhRKCiQuOqialqrA9ruD5DcCAQN0yM1HxwZe6TtS5Y/s1600/km3.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Mesos WebGUI shows active Kubernetes task</b></span></td></tr>
</tbody></table>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH34onYkw4vuo_CYi_yHxpPENfppIBFG9z1k1oJVc2uqR-gta-1ZXGuQK6MG74YU9BBLI14eK0ftzdXzT4WTiNKFGUHI0RTI3BEjdMcQB9Lk_3qGEXfjRf3j729DBQzjzxHTr73977ivg/s1600/km4.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH34onYkw4vuo_CYi_yHxpPENfppIBFG9z1k1oJVc2uqR-gta-1ZXGuQK6MG74YU9BBLI14eK0ftzdXzT4WTiNKFGUHI0RTI3BEjdMcQB9Lk_3qGEXfjRf3j729DBQzjzxHTr73977ivg/s1600/km4.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Mesos WebGUI shows that the Kubernetes task is RUNNING</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVJwq3OhafflkD3YKBDbhfulIWEZAoz7ReD_m7RUK8ua9pPbBEAGesVvh-NgS6NTs67JomHBKVa4Ad5XZrx4AYIGL4MRT8HlujtcJFcSlsmwddrEYon45Z4PLnJIPOXmMv_eWfHoDlo_U/s1600/km5.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVJwq3OhafflkD3YKBDbhfulIWEZAoz7ReD_m7RUK8ua9pPbBEAGesVvh-NgS6NTs67JomHBKVa4Ad5XZrx4AYIGL4MRT8HlujtcJFcSlsmwddrEYon45Z4PLnJIPOXmMv_eWfHoDlo_U/s1600/km5.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Click through "Sandbox" link of the task to get to the "executor.log"</b></span></td></tr>
</tbody></table>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiK4OnYZqgsW9iZmStk9FFUQ2JmWpNoWMQOWQlFe9SAs79agy2U_QQcxz5w4HbOr07mcSMrAJ09W22COngX-F9cTNZjxpS_QF41rFlfGq6tGUGkE_BrJ7DuXil1qcOnqMbnE4wdsGmCMf0/s1600/km6.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiK4OnYZqgsW9iZmStk9FFUQ2JmWpNoWMQOWQlFe9SAs79agy2U_QQcxz5w4HbOr07mcSMrAJ09W22COngX-F9cTNZjxpS_QF41rFlfGq6tGUGkE_BrJ7DuXil1qcOnqMbnE4wdsGmCMf0/s1600/km6.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><b>An example of "executor.log"</b></span></td></tr>
</tbody></table>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-8mFZu1IxygDOhx8CGcAyGKvp4u77v_EPTk43c4q125HoIvYda-KHjO5d_PnxacMQNG6L_nqKW0sSflEry4JBdtJKlO1wKl-ZuOnQR_LeXre-RgPQgxJm2IaqlDtSWnvCqCuugfUnGa0/s1600/km8.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-8mFZu1IxygDOhx8CGcAyGKvp4u77v_EPTk43c4q125HoIvYda-KHjO5d_PnxacMQNG6L_nqKW0sSflEry4JBdtJKlO1wKl-ZuOnQR_LeXre-RgPQgxJm2IaqlDtSWnvCqCuugfUnGa0/s1600/km8.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Connected to the node where the container is running</b></span></td></tr>
</tbody></table>
<br />
Getting Kubernetes to work on Mesos can be rather challenging at this point of time.<br />
<div>
<br /></div>
<div>
However, it is possible and hopefully, over time, Kubernetes-Mesos integration can work seamlessly.</div>
<div>
<br /></div>
<div>
Have fun!</div>
<div>
<br />
<div>
<br /></div>
</div>
Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-88563399067428272322015-10-10T14:28:00.000+08:002015-11-07T21:32:43.931+08:00Docker: All Containers Get Automatically Updated /etc/hosts (!?!?!?)While I have always wanted such feature (automatically updated /etc/hosts for all running containers), I understand that Docker does not provide it natively (just yet - or at least AFAIK). I also understand some security issues that might come with such feature (not all running containers want other containers to connect to it).<br />
<br />
Anyway, about 2 weeks ago, while I was dockerizing a system automation application that requires at least 2 running nodes (containers), I found that the feature was silently available*.<br />
<br />
* I went through the <a href="https://docs.docker.com/release-notes/" target="_blank">Release Notes</a> of almost all recent releases and could not find such feature being mentioned. If I got it wrong, please point me to the proper Release Notes. Thanks!<br />
<br />
Before I forget, let me share with you the reason I have always wanted such feature. My reason is simple - I need all my running containers to know the "existence" of other related containers and have a way to communicate with them (in this case, through /etc/hosts).<br />
<br />
My test environment was on <b>CentOS 7.1 and Docker 1.8.2</b>.<br />
<br />
(1) Firstly, I started a container <b>without any hostname and container name</b>. You can see that the /etc/hosts file was updated with:<br />
(a) the <b>container ID as the hostname</b><br />
(b) the <b>container name as the hostname</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqa6xkt5ruYmaHccoBDejJCUwhedQ14PkLde5RPJKhWE6XYYdySRgkzBn52_PN3vuLhINIycT2vWK1ZTyTKshzRltxan66PpLts7XjqwI95QC9mMec-HiuBq0z3d3QsCg5xhfpzOUxHeg/s1600/host1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqa6xkt5ruYmaHccoBDejJCUwhedQ14PkLde5RPJKhWE6XYYdySRgkzBn52_PN3vuLhINIycT2vWK1ZTyTKshzRltxan66PpLts7XjqwI95QC9mMec-HiuBq0z3d3QsCg5xhfpzOUxHeg/s1600/host1.png" width="100%" /></a></div>
<br />
<br />
(2) Next, I started another container with an <b>assigned hostname of "node1"</b>. You can see that now the /etc/hosts was updated with:<br />
(a) the <b>assigned hostname</b><br />
(b) the <b>container name as the hostname</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6BtjAIEhK4oHJtjS-5e61vrBwnei3AAsqA-rKKMUO_bCNKf8GQjVEeqnPT2AcGMQAmbS31wXZtj5r5t2u_O54uKG2WHS_TQIoRrHfOwLW3RzZuaJ_Lk_WGvRyF7t41An_CF1PzUQK7LQ/s1600/host2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6BtjAIEhK4oHJtjS-5e61vrBwnei3AAsqA-rKKMUO_bCNKf8GQjVEeqnPT2AcGMQAmbS31wXZtj5r5t2u_O54uKG2WHS_TQIoRrHfOwLW3RzZuaJ_Lk_WGvRyF7t41An_CF1PzUQK7LQ/s1600/host2.png" width="100%" /></a></div>
<br />
(3) To spice it up a little bit more, I started another container with an assigned hostname and gave the container a name. You can see that the /etc/hosts was updated with:<br />
(a) the <b>assigned hostname</b><br />
(b) the <b>given container name</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaKIYBluk0eL9XR3n2Hb_vojncfzgHz_FcxF37USv7RzSJZbA3sArqSMDRyQ7L3yuvakZJC7rnlJVF-6VnBqgUtv2MHgZFPmyQduGz16_xz-L6s3muD1xa_v6Wj9YObfoV6sR21ff454U/s1600/host3.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaKIYBluk0eL9XR3n2Hb_vojncfzgHz_FcxF37USv7RzSJZbA3sArqSMDRyQ7L3yuvakZJC7rnlJVF-6VnBqgUtv2MHgZFPmyQduGz16_xz-L6s3muD1xa_v6Wj9YObfoV6sR21ff454U/s1600/host3.png" width="100%" /></a></div>
<br />
(4) The next test was to start up 3 containers with different hostname and given name and left all of them running. You can see that "node1" was started and its /etc/hosts was updated accordingly.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-qt8zIbNnflVpQi_3YxreuzKsd1eROwZh4G3FOg8Ryutoi4BFgxbtl-v3amOZiGM2gIEjMbII4TNdggpnrWFzewnD-5dZ6b1NpNUDXm7CQe70WP3uF-pc0BggulYRu6_ipx0FGMjAhno/s1600/host4.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-qt8zIbNnflVpQi_3YxreuzKsd1eROwZh4G3FOg8Ryutoi4BFgxbtl-v3amOZiGM2gIEjMbII4TNdggpnrWFzewnD-5dZ6b1NpNUDXm7CQe70WP3uF-pc0BggulYRu6_ipx0FGMjAhno/s1600/host4.png" width="100%" /></a></div>
<br />
(5) Next, I started "<b><span style="color: blue;">node2</span></b>" and its /etc/hosts was updated with details of "<b><span style="color: red;">node1</span></b>" too.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-G5hc81HAAuAEStkBYDvW-qJ1ksvoiPCxhWm-v94abkJAITYSvLrR-iKNubDMdhMoG7VgdVUUoyRlN7b_mHlpA3nuvS7POynEjBrEcBzcUh3lwHVJzn6H6WDvgBWChs9VDLJ_ZS9nQe8/s1600/host5.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-G5hc81HAAuAEStkBYDvW-qJ1ksvoiPCxhWm-v94abkJAITYSvLrR-iKNubDMdhMoG7VgdVUUoyRlN7b_mHlpA3nuvS7POynEjBrEcBzcUh3lwHVJzn6H6WDvgBWChs9VDLJ_ZS9nQe8/s1600/host5.png" width="100%" /></a></div>
<br />
(6) What happened to the /etc/hosts of "<b><span style="color: red;">node1</span></b>" at this moment? Surprise, surpirse...you can see that its /etc/hosts was updated with details of "<b><span style="color: blue;">node2</span></b>" too.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxFkSYkx57kgQd7Pjg9lGdrCvHmYWCuSmp5Id3Tssdhfkqzulk_rO5gxu_m3OJlNxaus6jcWCVZAMsWl18_S3toe8BDNYNRqKWUJtKkOYXXHuvbIKVydOLaSboFOlBKmwKiQ4wU6OQRiQ/s1600/host6.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxFkSYkx57kgQd7Pjg9lGdrCvHmYWCuSmp5Id3Tssdhfkqzulk_rO5gxu_m3OJlNxaus6jcWCVZAMsWl18_S3toe8BDNYNRqKWUJtKkOYXXHuvbIKVydOLaSboFOlBKmwKiQ4wU6OQRiQ/s1600/host6.png" width="100%" /></a></div>
<br />
(7) Just to make sure it's not devaju. I started the third container ("<b><span style="color: red;">node1</span></b>" and "<b><span style="color: blue;">node2</span></b>" were still running). This time I wasn't surprised to see that the /etc/hosts of "<b><span style="color: #6aa84f;">node3</span></b>" was updated with details of "<b><span style="color: red;">node1</span></b>" and "<b><span style="color: blue;">node2</span></b>".<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMIfkTgVj2SDsL3OVWZuBAG0JYTQn4jAplaDwNzVvuAyoDlsm7o5tPHgpILNEMzuHD3PGu9RytALkMFZQrd8uWLUsKtYaqffQ6oo29qcnDfJ4MnLHmFHnY8cgXQrikyZJs7tpwloROY7g/s1600/host7.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMIfkTgVj2SDsL3OVWZuBAG0JYTQn4jAplaDwNzVvuAyoDlsm7o5tPHgpILNEMzuHD3PGu9RytALkMFZQrd8uWLUsKtYaqffQ6oo29qcnDfJ4MnLHmFHnY8cgXQrikyZJs7tpwloROY7g/s1600/host7.png" width="100%" /></a></div>
<br />
(8) Lastly, let's check the /etc/hosts of "<b><span style="color: red;">node1</span></b>" and "<b><span style="color: blue;">node2</span></b>". Viola, they are updated too!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyut_43JJw9UVBVGX8hKG5jNF9lwEgfayJPl8NGDTsd4wP_2BUtCVxe-pvKJzUqVWgRbNRdzCBruP2hNUxWw5IQwyyCXvY5lkGlga9fm3TqfBidIMCV9z0GMppbHpmXgNaHSd2aVN1DQA/s1600/host8.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyut_43JJw9UVBVGX8hKG5jNF9lwEgfayJPl8NGDTsd4wP_2BUtCVxe-pvKJzUqVWgRbNRdzCBruP2hNUxWw5IQwyyCXvY5lkGlga9fm3TqfBidIMCV9z0GMppbHpmXgNaHSd2aVN1DQA/s1600/host8.png" width="100%" /></a></div>
<br />
Seriously, I am not sure whether this feature has been available for some time or it is an experimental feature. Anyway, I like it...for the thing I do! So, I am not speaking for you :)<br />
<br />Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-49830762443640192042015-10-04T10:53:00.003+08:002015-11-07T21:33:14.775+08:00Docker: Sharing Kernel ModuleThis is more like a request-for-help entry :)<br />
<br />
I was working on a small project to dockerize a system automation application that supports clustering. Everything was working fine until the very last minute - during system startup!!!<br />
<br />
The database was working fine in a primary-standby setup. The system automation application was running fine on both of the nodes in the cluster. Configuration went fine too.<br />
<br />
However, when I started the automation monitoring, KABOOOOMMM, problem happened!<br />
<br />
What happened was that the first node came up fine, but the second node seems to be reporting weird status. A quick read of the log file told me that the second node was unable to load the "softdog" kernel module and would not be able to initialize. The reason is very likely because the first node has already gotten hold of the kernel module.<br />
<br />
Ok, I have to admit that this is a silly mistake I made and an oversight before I started the project.<br />
<br />
Anyway, I would like to share what I learned from this experience:<br />
(1) It is possible to have access to kernel module by:<br />
(a) mounting the /lib/modules directory read-only into the container (-v /lib/modules:/lib/modules:ro).<br />
(b) the kernel version of the host machine must be the same with the kernel version of the image/container.<br />
(c) run the container with --privileged option.<br />
<br />
For those people out there that are trying to dockerize applications that require kernel module, please be reminded that all containers on a same host would share the same underlying (host) kernel. Hence, you have to make sure that the application would still work under such condition.<br />
<br />
I am not a kernel expert and have not ventured into this area in Docker previously, so any help is appreciated - SOS!!!!!Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-50329051976379322162015-09-12T12:32:00.001+08:002015-09-12T12:32:10.246+08:00Running Apache Spark on MesosRunning Apache Spark on Mesos is easier than I thought!<br />
<br />
I have 3 nodes at home (hdp1, hdp2 and hdp3) which are running Hortonworks Data Platform (HDP). I have HDFS, Zookeeper and Spark installed on the cluster.<br />
<br />
To test running Spark on Mesos, I decided to reuse the cluster for simplicity's sake.<br />
<br />
All I did was:<br />
(1) install Mesos master on node hdp1.<br />
(2) install Mesos slave on node hdp2 and hdp3.<br />
(3) configure the master and slaves accordingly.<br />
<br />
<b>NOTE</b>:<br />
(i) For more information about the installation and configuration of Mesos, you can refer to <a href="http://manfrix.blogspot.my/2015/02/apache-mesos-when-all-becomes-one.html" target="_blank">this blog entry</a>.<br />
(ii) I reuse Zookeeper that was installed with HDP.<br />
<br />
Once Mesos is up and running, I decided to carry out a very simple test that is described as follow:<br />
(1) Use "spark-shell" as the Spark client to connect to the Mesos cluster.<br />
(2) Load a sample text file into HDFS.<br />
(3) Use spark-shell to perform a line count on the loaded text file.<br />
<br />
First, let's see what is the command that you can use to connect spark-shell to a running Mesos cluster.<br />
<br />
<b>TIPS</b>: All you need to do is to use the "--master" option to point to the Mesos cluster at "mesos://<IP-or-hostname of Mesos master>:5050".<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj00SY6IAAJB5VWri99Mway01qRndgqzTr3K_wl50SqQxiW-kD6vPiCPMHjtMIJm6fnxQBmlVFqfjZsrx6Rqs-pMx1-XG_PRZ_SyYGOBJIko7dXPafdk40frWmZyiM5iJphKCNqWJI8yDE/s1600/mesos-spark-1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj00SY6IAAJB5VWri99Mway01qRndgqzTr3K_wl50SqQxiW-kD6vPiCPMHjtMIJm6fnxQBmlVFqfjZsrx6Rqs-pMx1-XG_PRZ_SyYGOBJIko7dXPafdk40frWmZyiM5iJphKCNqWJI8yDE/s1600/mesos-spark-1.png" width="100%" /></a></div>
<br />
<br />
Then, let's load a sample text file into HDSF using the "hadoop fs -put" command.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPdZQRZSmhBfxFyy_KSwFurGQQ5IaFKNaX81-PLVDl9k_H_Yp9qDyuuidV3b4juli2Jcik3Y9XUdfVf0CVMmKlGBWpZP_nVf52Ue8_vlwOGtG1Sg1Kome6EWvq8KCsu9PlnpThvYRV7XA/s1600/mesos-spark-2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPdZQRZSmhBfxFyy_KSwFurGQQ5IaFKNaX81-PLVDl9k_H_Yp9qDyuuidV3b4juli2Jcik3Y9XUdfVf0CVMmKlGBWpZP_nVf52Ue8_vlwOGtG1Sg1Kome6EWvq8KCsu9PlnpThvYRV7XA/s1600/mesos-spark-2.png" width="100%" /></a></div>
<br />
<br />
Once the sample text file is loaded, let's create a spark RDD using it.<br />
<br />
<b>TIPS</b>: You can use the "sc.textFile" function and points it to "hdfs://<namenode>:8020/<path to file>".<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiN8FV-ngjqHo0j_9feDWEnDbmez-D4UvOb5ugDYYT_c_bWOiYeOvGY3e2SivUegR_ySmsIfKa2jrO-JhrCA0WHFTbDPbQaOTr65zcYE_a3tlXS2kh6F952OE7Y96QvmQ7ci_8k8u3aTbU/s1600/mesos-spark-5.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiN8FV-ngjqHo0j_9feDWEnDbmez-D4UvOb5ugDYYT_c_bWOiYeOvGY3e2SivUegR_ySmsIfKa2jrO-JhrCA0WHFTbDPbQaOTr65zcYE_a3tlXS2kh6F952OE7Y96QvmQ7ci_8k8u3aTbU/s1600/mesos-spark-5.png" width="100%" /></a></div>
<br />
<br />
After the RDD is created, let's run a count on it using the "count()" function.<br />
<br />
You can see from the screenshot above and below that, Spark (through Mesos) has submitted 2 tasks that are executed on node hdp2 and hdp3 respectively.<br />
<br />
You can also see from the screenshot above that the end result is returned as "4" (which means 4 lines in the file - which is correct!).<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIFXNV6jLX3HSw2sF4AjC2HqtJsxEeWWZ_uZiR470gezRmz941HkgGrZ7_oB0qM5phO8HwlxSKDku2R8TZvb2AGDbAYjMFItLfEOUa6LEXHPaUfQOwmv4LtzszRya3z-hwd9NBftNiK1U/s1600/mesos-spark-3.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIFXNV6jLX3HSw2sF4AjC2HqtJsxEeWWZ_uZiR470gezRmz941HkgGrZ7_oB0qM5phO8HwlxSKDku2R8TZvb2AGDbAYjMFItLfEOUa6LEXHPaUfQOwmv4LtzszRya3z-hwd9NBftNiK1U/s1600/mesos-spark-3.png" width="100%" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEe1w3tN6_umq52VKm3CyPS-E4uEB9Bjq3Ym8WxisEjFcr3SgiBuQpEg5ubvjk42ZolYCIvS-r3RUq1efpOncysRMQ_NJp6jGJYquBynUj3j22allFoXXs2BZKOjuSXdt0Oo06P9Dezls/s1600/mesos-spark-4.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEe1w3tN6_umq52VKm3CyPS-E4uEB9Bjq3Ym8WxisEjFcr3SgiBuQpEg5ubvjk42ZolYCIvS-r3RUq1efpOncysRMQ_NJp6jGJYquBynUj3j22allFoXXs2BZKOjuSXdt0Oo06P9Dezls/s1600/mesos-spark-4.png" width="100%" /></a></div>
<br />
<br />
So, that is how easy it is to run Spark on Mesos.<br />
<br />
Hope that you are going to try it out!<br />
<br />
<br />Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-59160049435638896182015-09-05T19:10:00.000+08:002015-09-05T19:36:09.824+08:00Neo4j: How to import data using neo4j-import?One of the few first things to do while learning a new database is to load some sample data and there is no difference when it comes to Neo4j.<br />
<br />
With Neo4j, my preferred way of data loading is through the <b>neo4j-import</b> command.<br />
<br />
While the documentation available is decent, new users might face some challenges accomplishing it (at least I was struggling at one point of time :).<br />
<br />
That is the purpose of me writing this blog entry - to help those that need a quick-start guide for loading data into Neo4j.<br />
<br />
Let's do it in a cookbook manner...<br />
<br />
(1) Read the following documentations:<br />
<a href="http://neo4j.com/docs/stable/import-tool.html" target="_blank">http://neo4j.com/docs/stable/import-tool.html</a><br />
<br />
It's ok if you do not understand fully. Please allow me to share with you an example.<br />
<br />
(2) Let's create the CSV needed to represent NODE and RELATIONSHIP.<br />
<br />
The example that I have here is a simple banking system that contains the following <b>NODEs</b>:<br />
(a) CUSTOMER<br />
(b) ACCOUNT<br />
(c) ATM<br />
<br />
and the following <b>RELATIONSHIPs</b>:<br />
(a) CUSTOMER->ACCOUNT <b>[OWNS]</b><br />
(b) ACCOUNT->ACCOUNT <b>[TXFER] </b><br />
(c) CUSTOMER->ATM <b>[WITHDRAW]</b><br />
<br />
(3) First, let's create the CSV files for the NODEs.<br />
<br />
According to the documentation (see excerpt below), the CSV header for a NODE must contain an <b>ID</b> and a <b>LABEL</b>.<br />
[<a href="http://neo4j.com/docs/stable/import-tool-header-format.html" target="_blank">http://neo4j.com/docs/stable/import-tool-header-format.html</a>]<br />
<br />
<div class="titlepage" style="background-color: white; box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px;" xmlns="">
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box;">
<h3 class="title" style="box-sizing: border-box; color: inherit; cursor: pointer; font-family: inherit; font-size: 24px; line-height: 1.1; margin-bottom: 10px; margin-top: 20px;" xmlns="http://www.w3.org/1999/xhtml">
Nodes</h3>
</div>
</div>
</div>
<div style="background-color: white; box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; margin-bottom: 10px;">
The following field types do additionally apply to node data sources:</div>
<div class="variablelist" style="background-color: white; box-sizing: border-box;">
<dl class="variablelist" style="box-sizing: border-box; margin-bottom: 20px; margin-top: 0px;">
<dt style="box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; font-weight: 700; line-height: 1.42857;"><span class="term" style="box-sizing: border-box;">ID</span></dt>
<dd style="box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 1.42857; margin-bottom: 0.75em; margin-left: 0px;">Each node must have a unique id which is used during the import. The ids are used to find the correct nodes when creating relationships. Note that the id has to be unique across all nodes in the import, even nodes with different labels.</dd>
<dt style="box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; font-weight: 700; line-height: 1.42857;"><span class="term" style="box-sizing: border-box;">LABEL</span></dt>
<dd style="box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 1.42857; margin-bottom: 0.75em; margin-left: 0px;">Read one or more labels from this field. For multiple labels, the values are separated by the array delimiter.</dd><dd style="box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 1.42857; margin-bottom: 0.75em; margin-left: 0px;"><br /></dd><dd style="box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 1.42857; margin-bottom: 0.75em; margin-left: 0px;">How do we do that? Simple...</dd><dd style="box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 1.42857; margin-bottom: 0.75em; margin-left: 0px;"><br /></dd>
<dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">[danielyeap@myhost neo4j-data]$ cat <b>customer.csv </b></span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;"><b>custID:ID(CUSTOMER),firstname,lastname,since:int,:LABEL</b></span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">1,Daniel,Yeap,1999,CUSTOMER</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">2,John,Smith,1999,CUSTOMER</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">3,Tod,Pitt,2010,CUSTOMER</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">4,Isabel,Grager,2014,CUSTOMER</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">[danielyeap@myhost neo4j-data]$ cat <b>atm.csv </b></span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;"><b>atmId:ID(ATM),location,:LABEL</b></span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">1,Damansara Uptown,ATM</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">2,Kota Damansara,ATM</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">3,Jalan Ipoh,ATM</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">[danielyeap@myhost neo4j-data]$ cat <b>account.csv </b></span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;"><b>custId:int,acctno:ID(ACCOUNT),:LABEL</b></span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">1,11111,ACCOUNT</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">1,11112,ACCOUNT</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">2,22221,ACCOUNT</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">2,22222,ACCOUNT</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">3,33331,ACCOUNT</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><span style="font-family: PT Sans, Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 15px; line-height: 21.4286px;">4,44441,ACCOUNT</span></span></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><br /></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;">(4) Since the NODEs are taken care of, let's craft our RELATIONSHIP CSV files.</dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><br /></dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"><div class="titlepage" style="box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px;" xmlns="">
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box;">
<h3 class="title" style="box-sizing: border-box; color: inherit; cursor: pointer; font-family: inherit; font-size: 24px; line-height: 1.1; margin-bottom: 10px; margin-top: 20px;" xmlns="http://www.w3.org/1999/xhtml">
Relationships</h3>
</div>
</div>
</div>
<div style="box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; margin-bottom: 10px;">
For relationship data sources, there’s three mandatory fields:</div>
<div class="variablelist" style="box-sizing: border-box; font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px;">
<dl class="variablelist" style="box-sizing: border-box; margin-bottom: 20px; margin-top: 0px;">
<dt style="box-sizing: border-box; font-weight: 700; line-height: 1.42857;"><span class="term" style="box-sizing: border-box;">TYPE</span></dt>
<dd style="box-sizing: border-box; line-height: 1.42857; margin-bottom: 0.75em; margin-left: 0px;">The relationship type to use for the relationship.</dd>
<dt style="box-sizing: border-box; font-weight: 700; line-height: 1.42857;"><span class="term" style="box-sizing: border-box;">START_ID</span></dt>
<dd style="box-sizing: border-box; line-height: 1.42857; margin-bottom: 0.75em; margin-left: 0px;">The id of the start node of the relationship to create.</dd>
<dt style="box-sizing: border-box; font-weight: 700; line-height: 1.42857;"><span class="term" style="box-sizing: border-box;">END_ID</span></dt>
<dd style="box-sizing: border-box; line-height: 1.42857; margin-bottom: 0.75em; margin-left: 0px;">The id of the end node of the relationship to create.</dd></dl>
</div>
</dd><dd style="box-sizing: border-box; margin-bottom: 0.75em; margin-left: 0px;"></dd>
<div>
<br /></div>
<div>
[danielyeap@myhost neo4j-data]$ cat <b>rel-cust-acct.csv </b><br />
<b>:START_ID(CUSTOMER),:END_ID(ACCOUNT),:TYPE</b><br />
1,11111,OWNS<br />
1,11112,OWNS<br />
2,22221,OWNS<br />
2,22222,OWNS<br />
3,33331,OWNS<br />
4,44441,OWNS<br />
[danielyeap@myhost neo4j-data]$ cat <b>rel-txfer.csv </b><br />
<b>TXID,:START_ID(ACCOUNT),:END_ID(ACCOUNT),amount:int,:TYPE</b><br />
1,11111,11112,100,TXFER<br />
2,11111,22222,200,TXFER<br />
3,22221,33331,300,TXFER<br />
4,44441,11112,400,TXFER<br />
5,33331,22221,500,TXFER<br />
6,11111,22222,600,TXFER<br />
7,11111,33331,700,TXFER<br />
8,11111,33331,800,TXFER<br />
[danielyeap@myhost neo4j-data]$ cat <b>rel-withdraw.csv </b><br />
<b>TXID,:START_ID(CUSTOMER),:END_ID(ATM),amount:int,:TYPE</b><br />
1,1,1,100,WITHDRAW_FROM<br />
2,1,2,200,WITHDRAW_FROM<br />
3,2,3,300,WITHDRAW_FROM<br />
4,4,1,400,WITHDRAW_FROM<br />
5,3,2,500,WITHDRAW_FROM<br />
6,1,2,600,WITHDRAW_FROM<br />
7,1,3,700,WITHDRAW_FROM<br />
8,1,3,800,WITHDRAW_FROM</div>
<div>
<div>
<br /></div>
</div>
<div>
<div>
<br /></div>
</div>
<div>
<div>
(5) Now that we have all the files that we need to perform the data loading, it is time to execute the command:</div>
</div>
<div>
<div>
<br /></div>
</div>
<div>
<div>
<div>
[danielyeap@myhost neo4j-data]$ neo4j-import <b>--into</b> ~danielyeap/neo4j/data/graph.db <b>--nodes</b> customer.csv <b>--nodes</b> account.csv <b>--nodes</b> atm.csv <b>--relationships</b> rel-txfer.csv <b>--relationships</b> rel-cust-acct.csv <b>--relationships</b> rel-withdraw.csv </div>
</div>
</div>
<div>
<div>
<br /></div>
</div>
<div>
<div>
<div>
Importing the contents of these files into /home/danielyeap/neo4j/data/graph.db:</div>
<div>
Nodes:</div>
<div>
/home/danielyeap/neo4j-data/customer.csv</div>
<div>
<br /></div>
<div>
/home/danielyeap/neo4j-data/account.csv</div>
<div>
<br /></div>
<div>
/home/danielyeap/neo4j-data/atm.csv</div>
<div>
Relationships:</div>
<div>
/home/danielyeap/neo4j-data/rel-txfer.csv</div>
<div>
<br /></div>
<div>
/home/danielyeap/neo4j-data/rel-cust-acct.csv</div>
<div>
<br /></div>
<div>
/home/danielyeap/neo4j-data/rel-withdraw.csv</div>
<div>
<br /></div>
<div>
Available memory:</div>
<div>
Free machine memory: 11.08 GB</div>
<div>
Max heap memory : 2.73 GB</div>
<div>
<br /></div>
<div>
Nodes</div>
<div>
[*>:??------------------------------------------------------------------------------||NODE:7.||] 10k</div>
<div>
Done in 481ms</div>
<div>
Prepare node index</div>
<div>
[*DETECT:7.63 MB-------------------------------------------------------------------------------] 0</div>
<div>
Done in 11ms</div>
<div>
Calculate dense nodes</div>
<div>
[*>:??-----------------------------------------------------------------------------------------] 0</div>
<div>
Done in 297ms</div>
<div>
Relationships</div>
<div>
[*>:??--------------------------------------------------------------------------------------|P|] 10k</div>
<div>
Done in 68ms</div>
<div>
Node --> Relationship</div>
<div>
[*>:??-----------------------------------------------------------------------------------------] 10k</div>
<div>
Done in 10ms</div>
<div>
Relationship --> Relationship</div>
<div>
[*>:??-----------------------------------------------|v:??-------------------------------------] 10k</div>
<div>
Done in 10ms</div>
<div>
Node counts</div>
<div>
[*COUNT:76.29 MB-------------------------------------------------------------------------------] 10k</div>
<div>
Done in 218ms</div>
<div>
Relationship counts</div>
<div>
[*>:??------------------------------------------|COUNT-----------------------------------------] 10k</div>
<div>
Done in 10ms</div>
<div>
<br /></div>
<div>
<b>IMPORT DONE in 2s 664ms. Imported:</b></div>
<div>
<b> 13 nodes</b></div>
<div>
<b> 22 relationships</b></div>
<div>
<b> 66 properties</b></div>
</div>
</div>
<div>
<div>
<b><br /></b></div>
</div>
<div>
<div>
<b>==============================</b></div>
</div>
<div>
<div>
<br /></div>
</div>
<div>
<b>NOTE</b>:</div>
<div>
(i) You can refer to the documentation about the command line <a href="http://neo4j.com/docs/stable/import-tool-usage.html" target="_blank">HERE</a>.</div>
<div>
(ii) The "--into" option must point to the directory where neo4j database is located (look in <neo4j_install_dir/conf/neo4j-server.properties) =></div>
<div>
<br /></div>
<div>
org.neo4j.server.database.location=/home/danielyeap/neo4j/data/graph.db</div>
<div>
<br /></div>
<div>
(iii) If there is an error about existing database, you can use the following procedure to remove it:</div>
<div>
~ While Neo4j is up and running, go to "org.neo4j.server.database.location" and run "rm -rf *".</div>
<div>
~ After all the files under that directory are removed, execute the "neo4j-import" command again.</div>
<div>
~ After the import is successful, restart Neo4j (command = neo4j restart). </div>
<div>
~ Login to Neo4j web interface and you should see your data loaded.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUnaZ7sw3IvJcztPcqg9f53LLHEWXAF6pkQcWKSLYy6Ti2yrGAycwkrluSEv09wYp5bs4MiXtigplB48EVFo-vS9AtbX_h0cF-Ppa3pIGz4bnRNImzFyi6E1syK_6ExFxDPx86xbcauMs/s1600/neo4j.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUnaZ7sw3IvJcztPcqg9f53LLHEWXAF6pkQcWKSLYy6Ti2yrGAycwkrluSEv09wYp5bs4MiXtigplB48EVFo-vS9AtbX_h0cF-Ppa3pIGz4bnRNImzFyi6E1syK_6ExFxDPx86xbcauMs/s1600/neo4j.png" width="100%" /></a></div>
<div>
<br /></div>
</dl>
</div>
Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-38332199525021965122015-08-22T14:02:00.004+08:002015-08-22T14:35:36.875+08:00D3.js: How to draw a Node Relationship Graph like Neo4J?First of all, I am deeply sorry for the lack of update for this blog! I have been really busy with my work, family and self-learning.<br />
<br />
Anyway, this entry is about some cool technologies that I have been spending time lately - <b><a href="http://neo4j.com/" target="_blank">Neo4</a>j</b> and <b><a href="http://d3js.org/">D3.js</a></b>.<br />
<br />
I have always been curious about graph database. Recently, I have finally conceded to my better curious half and started to get my hands on Neo4j (one of the more popular graph database engine).<br />
<br />
I am greatly impressed by its Web GUI that is able to draw a proper Node -> Relationship graph that is both pretty and practical.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://dev.assets.neo4j.com.s3.amazonaws.com/wp-content/uploads/2014/08/T1cJSxYSi_a8FQ0JR0KrkFSo3lunMT1cau-LWPimMn0G8bSkgKTcci7BGopGM8KYOYcODy-Ifiv1S0LXVkIS_oRde3FveTTIzWc-dOhXGx_Ep2E0-8nxVyOm.png?_ga=1.212580756.1259378330.1431051962" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="http://dev.assets.neo4j.com.s3.amazonaws.com/wp-content/uploads/2014/08/T1cJSxYSi_a8FQ0JR0KrkFSo3lunMT1cau-LWPimMn0G8bSkgKTcci7BGopGM8KYOYcODy-Ifiv1S0LXVkIS_oRde3FveTTIzWc-dOhXGx_Ep2E0-8nxVyOm.png?_ga=1.212580756.1259378330.1431051962" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Courtesy from Neo4j website</b></td></tr>
</tbody></table>
<br />
At work, I have a sudden need for such a graph to analyze some really complex text-based source-target relationships. That was how D3.js came into play!<br />
<br />
I have always known that I need to lay my hands on D3.js one day for data visualisation. This need at work just justified it.<br />
<br />
After some googling and reading, I ended up with the codes below:<br />
<b>NOTE</b>: Simplified to increase readability!<br />
<br />
<input class="spoilerbutton" onclick="this.value=this.value=='Show Codes'?'Hide Codes':'Show Codes';" type="button" value="Show Codes" />
<div class="spoiler">
<div>
<table cellpadding="2" cellspacing="0" style="border: 1px solid #ddd;">
<tbody>
<tr>
<td><pre class="brush: csharp"><!DOCTYPE html>
<html lang="en">
<head>
<%@page pageEncoding="UTF-8" contentType="text/html; charset=UTF-8"%>
<title>Node Relationship Graph</title>
<script type="text/javascript" src="d3/d3.min.js"></script>
<style>
path.link {
fill: none;
stroke: #666;
stroke-width: 1.5px;
}
circle {
fill: #ccc;
stroke: #fff;
stroke-width: 1.5px;
}
text {
fill: #000;
font: 10px sans-serif;
pointer-events: none;
}
</style>
</head>
<BODY>
<script type="text/javascript">
d3.csv("data/lsrel.csv", function(error, links) {
var nodes = {};
var rel = {};
// Compute the distinct nodes from the links.
links.forEach(function(link) {
link.id = "rel" + link.relnum;
// link.relnum = link.relnum;
var sLinkSrc = link.source;
var sLinkTgt = link.target;
link.source = nodes[link.source] ||
(nodes[link.source] = {name: link.source, relcnt: 0, srccnt: 0, tgtcnt: 0});
link.target = nodes[link.target] ||
(nodes[link.target] = {name: link.target, relcnt: 0, srccnt: 0, tgtcnt: 0});
link.relationship = link.relationship;
if (nodes[sLinkSrc])
{
nodes[sLinkSrc]["relcnt"] = nodes[sLinkSrc]["relcnt"]+1;
nodes[sLinkSrc]["srccnt"] = nodes[sLinkSrc]["srccnt"]+1;
}
if (nodes[sLinkTgt])
{
nodes[sLinkTgt]["relcnt"] = nodes[sLinkTgt]["relcnt"]+1;
nodes[sLinkTgt]["tgtcnt"] = nodes[sLinkTgt]["tgtcnt"]+1;
}
// console.log(JSON.stringify(nodes));
// console.log("NODEPROP: " + nodes[sLinkSrc].name);
});
var width = screen.width-80,
height = screen.height-80;
console.log("Width: " + width);
console.log("Height: " + height);
var force = d3.layout.force()
.nodes(d3.values(nodes))
.links(links)
.size([width, height])
.linkDistance(350)
.charge(-800)
.on("tick", tick)
.start();
var drag = force.drag()
.on("dragstart", dragstart);
var svg = d3.select("body").append("svg")
.attr("width", width)
.attr("height", height);
// build the arrow.
svg.append("svg:defs").selectAll("marker")
.data(["end"])
.enter().append("svg:marker")
.attr("id", String)
.attr("viewBox", "0 -5 10 10")
.attr("refX", 22)
.attr("refY", -1)
.attr("markerWidth", 8)
.attr("markerHeight", 8)
.attr("orient", "auto")
.append("svg:path")
.attr("d", "M0,-5L10,0L0,5");
// add the links and the arrows
var path = svg.append("svg:g").selectAll("path")
.data(force.links())
.enter()
.append("svg:path")
.attr("id", function(d) { return d.id; } )
.attr("class", "link")
.attr("marker-end", "url(#end)");
var mytext = svg.append("svg:g").selectAll("text")
.data(force.links())
.enter()
.append("text")
.attr("dx", "150")
.attr("dy", "-8")
.append("textPath")
.attr("xlink:href", function(d) { return "#" + d.id; })
.attr("style", "fill:magenta; font-weight:bold; font-size:12")
.text(function(d) { return d.relationship; } );
// define the nodes
var node = svg.selectAll(".node")
.data(force.nodes())
.enter().append("g")
.attr("class", "node")
.call(force.drag);
// add the nodes
node.append("circle")
.attr("r", 12)
.attr("fill", "grey")
.append("svg:title")
.text(function(d) { return "Source: " + d.srccnt + " ~ Target: " + d.tgtcnt; });
// add the text
node.append("text")
.attr("x", 12)
.attr("dy", ".35em")
.attr("style", "fill:blue; font-weight:bold; font-size:16")
.text(function(d) { return d.name; });
node.append("text")
.attr("text-anchor", "middle")
// .attr("style", "font-weight:bold; font-size:12")
.attr("style", function(d) {
if (d.relcnt >= 3)
{
return "font-weight:bold; font-size:12; fill:red"
}
else
{
return "font-weight:bold; font-size:12"
}
})
.text(function(d) { return d.relcnt; });
// add the curvy lines
function tick() {
path.attr("d", function(d) {
var dx = d.target.x - d.source.x,
dy = d.target.y - d.source.y,
dr = Math.sqrt(dx * dx + dy * dy);
return "M" +
d.source.x + "," +
d.source.y + "A" +
dr + "," + dr + " 0 0,1 " +
d.target.x + "," +
d.target.y;
});
node
.attr("transform", function(d) {
return "translate(" + d.x + "," + d.y + ")"; });
}
function dragstart(d)
{
d3.select(this).classed("fixed", d.fixed = true);
}
if (error)
{
console.log(error);
}
else
{
console.log(nodes);
console.log(links);
console.log(path);
console.log(rel);
}
});
</script>
</BODY>
</HTML>
</pre>
</td></tr>
</tbody></table>
</div>
</div>
<br />
A sample of the input file "lsrel.csv" is as follow:<br />
<br />
<b>relnum,source,target,relationship</b><br />
6,c,a,DependsOn<br />
5,c,d,Anti-Collocated<br />
1,a,b,DependsOn<br />
2,a,c,StartAfter<br />
3,b,c,StopAfter<br />
4,b,d,Collocated<br />
<div>
<br />
The output of the D3.js script based on the data above:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZ4Nmi6TBkWYhnEeG8MRT9hCX0JnwO5dEpsCWFcvL9kNuEVitFY61KzMGG0riPrkBSUaTITFQu1Y7gfsGUswkXiMZxBcOyue6EhqAcRS3e7rw4tsa9pH2DeooH0DeI5Qibc1nWWpQeVCs/s1600/result.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZ4Nmi6TBkWYhnEeG8MRT9hCX0JnwO5dEpsCWFcvL9kNuEVitFY61KzMGG0riPrkBSUaTITFQu1Y7gfsGUswkXiMZxBcOyue6EhqAcRS3e7rw4tsa9pH2DeooH0DeI5Qibc1nWWpQeVCs/s1600/result.png" width="100%" /></a></div>
Hope you guys like it!<br />
<br /></div>
Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com6tag:blogger.com,1999:blog-4943934699922259568.post-25230210591167379822015-04-18T20:50:00.002+08:002015-04-18T21:11:31.004+08:00Docker: OpenDayLight (ODL)I have always wanted to learn more about Software Defined Network (SDN) when I first heard about it from a colleague (now ex-colleague) about 2 years ago.<br />
<br />
However, I have been really busy (a.k.a lazy) with work and family. Not until recently when I stumbled upon an interesting opportunity that drove me to take a peek at it.<br />
<br />
Ok, the first big question is "OpenFlow or OpenDayLight"? Since I do not have a lot of time to perform research on their respective strengths and weaknesses, I have resorted to a very "scientific" way to choose among the two. In the end, I chose OpenDayLight because (wait for it...) I prefer their website :).<br />
<br />
Since I am a big fan of Docker, it doesn't make sense for me to not install OpenDayLight on Docker. A quick google landed me with <a href="https://github.com/opendaylight/integration/tree/master/packaging/docker" target="_blank">this</a>. I am not interested in Debian, so I made some changes to the Dockerfile to switch the base image to Centos 6.6.<br />
<br />
<input class="spoilerbutton" onclick="this.value=this.value=='Show Codes'?'Hide Codes':'Show Codes';" type="button" value="Show Codes" />
<br />
<div class="spoiler">
<div>
<table cellpadding="2" cellspacing="0" style="border: 1px solid #ddd;">
<tbody>
<tr>
<td><pre class="brush: csharp">FROM centos:6.6
# Install required software (170MB)
RUN yum update -y && yum install -y tar wget java-1.7.0-openjdk
# Download and install ODL
WORKDIR /opt
RUN mkdir opendaylight
RUN wget -q "https://nexus.opendaylight.org/content/groups/public/org/opendaylight/integration/distribution-karaf/0.2.3-Helium-SR3/distribution-karaf-0.2.3-Helium-SR3.tar.gz" && \
tar -xf distribution-karaf-0.2.3-Helium-SR3.tar.gz -C opendaylight --strip-components=1 && \
rm -rf distribution-karaf-0.2.3-Helium-SR3.tar.gz
EXPOSE 162 179 1088 1790 1830 2400 2550 2551 2552 4189 4342 5005 5666 6633 6640 6653 7800 8000 8080 8101 8181 8383 12001
WORKDIR /opt/opendaylight
ENV JAVA_HOME /usr/lib/jvm/jre-1.7.0-openjdk.x86_64
CMD ["./bin/karaf", "server"]
</pre>
</td></tr>
</tbody></table>
</div>
</div>
<br />
<b>NOTE</b>:<br />
(i) My test shows that Java 8 is currently not supported and will give weird Java NullPointerException while executing command.<br />
(ii) You can download the Dockerfile <a href="https://docs.google.com/uc?authuser=0&id=0B6EFlN-9iDHtSVBlVS1DeU5XZDA&export=download" target="_blank">here</a>.<br />
<br />
By executing "docker build -t opendaylight:helium ." in the directory where the Dockerfile is stored, I ended up with a Docker image that contains OpenDayLight.<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinJkALvs_C-dDgdpHJOM4yXw5pWNpsURG9_8GRAWv87sgmgvyFuqOXUf_gszci8rKo7K5DfajNF-z4QPgizmVi_hO_hMxZH6BacmFVayA1cMoDuL4ZGqlqFJqE-Qvo2JJ0uZwb_ocASIg/s1600/odl1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinJkALvs_C-dDgdpHJOM4yXw5pWNpsURG9_8GRAWv87sgmgvyFuqOXUf_gszci8rKo7K5DfajNF-z4QPgizmVi_hO_hMxZH6BacmFVayA1cMoDuL4ZGqlqFJqE-Qvo2JJ0uZwb_ocASIg/s1600/odl1.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>OpenDayLight on Docker</b></span></td></tr>
</tbody></table>
<br />
After starting the container using "docker run" command (refer to the above screenshot), it is time to SSH connect to it and install some ODL components.<br />
<br />
This can be achieved by using the Karaf client (<a href="https://karaf.apache.org/index/community/download.html" target="_blank">download</a>).
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUnLhdkIcT2E3At2R8wzizICn2604hrw3KBw0xnA-vS_hyphenhyphenBouFidTD1urHo0FyOHMqvy5jInWBOLUpCCn8h_CjANgYvQrAEAfOSKRb4yFCH23eAWBKzHmHnU8fQKPGhW_ZzfHcpMqBRoc/s1600/odl2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUnLhdkIcT2E3At2R8wzizICn2604hrw3KBw0xnA-vS_hyphenhyphenBouFidTD1urHo0FyOHMqvy5jInWBOLUpCCn8h_CjANgYvQrAEAfOSKRb4yFCH23eAWBKzHmHnU8fQKPGhW_ZzfHcpMqBRoc/s1600/odl2.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>Connect to the ODL container through Karaf client</b></span></td></tr>
</tbody></table>
<div>
To list all supported features, you can execute the "feature:list" command.</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWhMEEPbm6YEi5Sqk7W_EawSdJFsDnnEOllWuEvZ7GoIKCSUOcdnUhrFN9lNBjlRSLR4ukHQt7awx_EElde_Wd9n8-Nw1PrIte7GXW6B1Lyr40tZ-OXFPKuAjzz3uJYiGt2IycYVwqkLY/s1600/odl3.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWhMEEPbm6YEi5Sqk7W_EawSdJFsDnnEOllWuEvZ7GoIKCSUOcdnUhrFN9lNBjlRSLR4ukHQt7awx_EElde_Wd9n8-Nw1PrIte7GXW6B1Lyr40tZ-OXFPKuAjzz3uJYiGt2IycYVwqkLY/s1600/odl3.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>List all supported features</b></span></td></tr>
</tbody></table>
<div>
To list currently installed features, you can use the "feature:list -i" command.</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaCyCcQIVZGGuEkRd1fLeSLDSZB1QLyKe9SuubUo3_rgGcfOd736OmNvbs_4dV9TOmnmamexg97C-dQM72kZ4G6Gg8mcaYjUiq3-iv-skGQEnKJItqQMgBQR9psmAc7UmfoMGQYKj5nH4/s1600/odl4.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaCyCcQIVZGGuEkRd1fLeSLDSZB1QLyKe9SuubUo3_rgGcfOd736OmNvbs_4dV9TOmnmamexg97C-dQM72kZ4G6Gg8mcaYjUiq3-iv-skGQEnKJItqQMgBQR9psmAc7UmfoMGQYKj5nH4/s1600/odl4.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>Default installed features</b></span></td></tr>
</tbody></table>
To install some basic features, you can execute "feature:install odl-restconf odl-l2switch-switch odl-mdsal-apidocs odl-dlux-core".<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBEv2_P7UUwJB-sahSTEejFowZhnYfuyVVlA4fOKX5QnA7ZqZMR5U5gLv9GIMtD-GAHDgYbs0030xiwjZv064DIJKxt5aU7dUmx84D1ZNsoAGp20Nwk0kdzZdHg1V9I4A2EKg7RIIPDdg/s1600/odl5.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBEv2_P7UUwJB-sahSTEejFowZhnYfuyVVlA4fOKX5QnA7ZqZMR5U5gLv9GIMtD-GAHDgYbs0030xiwjZv064DIJKxt5aU7dUmx84D1ZNsoAGp20Nwk0kdzZdHg1V9I4A2EKg7RIIPDdg/s1600/odl5.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>Install some basic features</b></span></td></tr>
</tbody></table>
Now, you can proceed to login to OpenDayLight User Experience (DLUX) using URL "<IP of the Docker container>:8181/dlux/index.html.<br />
<div>
The default user ID and password are "admin" and "admin" respectively.</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfvJyl89SeK-yPZGm-HuWBS47Gy9gY_P9AlV-UleHJSsxsryFA79lfWvLz6kp1_xMIElVIGiyeJWPPhX3zcaQNH1a8lgaUi_3T4ssSjLl47jIOqr-G-tfkl2cc-EUuKWKsf6U_OxP7DRY/s1600/odl6.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfvJyl89SeK-yPZGm-HuWBS47Gy9gY_P9AlV-UleHJSsxsryFA79lfWvLz6kp1_xMIElVIGiyeJWPPhX3zcaQNH1a8lgaUi_3T4ssSjLl47jIOqr-G-tfkl2cc-EUuKWKsf6U_OxP7DRY/s1600/odl6.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>DLUX login page</b></span></td></tr>
</tbody></table>
<div>
Voila, you are in!</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLNV7qD588zV-VeTdsdPICH70jhpLJv7Pu58JT0F3j5jpR6Vm5GY2xbkn3VFZ7NG6pIgz8f70jp8680lI78PfpqC2RwigyFJjdEi6ksdWSv2vbKVOFtgSFdP_7BSVwmL1WVct35ySyFI8/s1600/odl7.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLNV7qD588zV-VeTdsdPICH70jhpLJv7Pu58JT0F3j5jpR6Vm5GY2xbkn3VFZ7NG6pIgz8f70jp8680lI78PfpqC2RwigyFJjdEi6ksdWSv2vbKVOFtgSFdP_7BSVwmL1WVct35ySyFI8/s1600/odl7.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>DLUX login page</b></span></td></tr>
</tbody></table>
To learn ODL further, you can download its documentation from this <a href="http://www.opendaylight.org/software/downloads/helium-sr3" target="_blank">page</a>.<br />
<div>
<br /></div>
<div>
I am still far from really knowing how to use ODL properly. I might even explore OpenFlow one day.</div>
<div>
<br /></div>
<div>
Anyway, I had fun trying it out and hope that you would too!</div>
<div>
<br />
<div>
<br /></div>
</div>
Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-7285137008187885552015-04-12T16:35:00.002+08:002015-04-16T05:33:51.457+08:00Java: Stream Control Transmission Protocol (SCTP)If you have not heard about SCTP, don't fret! Me neither, until recently.<br />
<br />
For a good explanation about SCTP, you can read <a href="http://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol" target="_blank">this</a> and <a href="http://www.oracle.com/technetwork/articles/javase/index-139946.html" target="_blank">this</a> and <a href="http://www.ibm.com/developerworks/library/l-sctp/" target="_blank">this</a>. In short, SCTP is basically TCP on steroids!<br />
<br />
People who knows me well knew that I am always curious about new technology or technology new to me (sorry if I confuse you :).<br />
<br />
How can I miss the chance to explore SCTP futher? Of course, the language of choice would be Java - still my favorite language after all these years.<br />
<br />
First, I created the SctpServer class to serve as a SCTP server.<br />
Next, I created the SctpClient class to serve as the SCTP client that will connect to multiple SCTP servers started on different ports (same host in this case) through a single socket (YES, you read it right, single socket!).<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-xPhYD0zgF1qkpIqpovZzBiC3Djj3x1pqptZAA29rtU4zyDJvx_2mVzGTvhOOanfTbzrHHm2rmAvVYY2VnnyEsUlqDiutqdg9Z7P9J8SlHYcTb1S7vEssPwXno_WynjMCoWuuOz8JJi0/s1600/sctp1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-xPhYD0zgF1qkpIqpovZzBiC3Djj3x1pqptZAA29rtU4zyDJvx_2mVzGTvhOOanfTbzrHHm2rmAvVYY2VnnyEsUlqDiutqdg9Z7P9J8SlHYcTb1S7vEssPwXno_WynjMCoWuuOz8JJi0/s1600/sctp1.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>Two SctpServers listening on port 12000 and 12001. An SCTP client sends message to them on the same socket.</b></span></td></tr>
</tbody></table>
From the image above, you can see that there are 2 SctpServer listening on port 12000 and 12001 respectively.<br />
<br />
Once the ScptClient connects to them and send them messages, you can see <i><b>both</b></i> servers reported that the connection came from port <b>43975</b> with stream number of <b>0</b> (SctpServer that listens on port 12000) and <b>1</b> (SctpServer that listens on port 12001) respectively.<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_ePqHs3ua8sJG0NCqqp29PZ8HYYBdogHlJkIbJeBSoeXwlAWwQEoCPbAX5eWn4PCYrRSqcrR-CuWjDy-cIr8eEfkpHvrhRgX5xkWVMKuvrMI8nCT1MM4nr-GqfCEHsPOM1z3WTbriSJM/s1600/sctp2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_ePqHs3ua8sJG0NCqqp29PZ8HYYBdogHlJkIbJeBSoeXwlAWwQEoCPbAX5eWn4PCYrRSqcrR-CuWjDy-cIr8eEfkpHvrhRgX5xkWVMKuvrMI8nCT1MM4nr-GqfCEHsPOM1z3WTbriSJM/s1600/sctp2.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>SctpClient uses SctpMultiChannel for its communication.</b></span></td></tr>
</tbody></table>
<br />
Now, if you take a look at the codes for SctpClient, there is no "connect" statement at all. That is because a new association will automatically be created for every "send" command. [<a href="http://docs.oracle.com/javase/8/docs/jre/api/nio/sctp/spec/com/sun/nio/sctp/SctpMultiChannel.html" target="_blank">reference</a>]<br />
<br />
The destination of your message/data is now encapsulated in the <a href="http://docs.oracle.com/javase/7/docs/jre/api/nio/sctp/spec/com/sun/nio/sctp/class-use/MessageInfo.html" target="_blank">MessageInfo</a> object.<br />
<br />
If you have the interest to explore SCTP further, you can download the sample codes <a href="https://docs.google.com/uc?authuser=0&id=0B6EFlN-9iDHtaWpRY0kzbUpKamM&export=download">here</a>.<br />
<br />
Enjoy hacking!<br />
<br />
<br />Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-29089318982855455092015-04-08T23:40:00.001+08:002015-04-12T16:44:07.148+08:00Docker User Group MalaysiaI created the Docker User Group Malaysia on Facebook a few months ago.<br />
<br />
The intention is to share information and knowledge with all Docker users within Malaysia.<br />
<br />
From 2 members (myself and a colleague of mine), it has now grown to 8 members (some nice folks from Mindvalley).<br />
<br />
If you are interested to join or learn more about Docker, you can visit <a href="https://www.facebook.com/groups/632712500166002/" target="_blank">this Facebook group</a> anytime!Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-5179630868730666992015-04-06T17:32:00.001+08:002015-04-12T16:50:16.086+08:00Apache Storm: HBase Bolt<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;">Starting with version 0.9.2-incubating, Storm included support for HBase as bolt. You no longer need to seek third party packages to do that, which is a great news!</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;"><br /></span>
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;">To refer to the APIs and documentation, you can go <a href="https://github.com/apache/storm/tree/master/external/storm-hbase" target="_blank">here</a>.</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;"><br /></span>
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;">I have coded a simple Storm application that takes Kafka as spout and HBase as bolt. In other words, the application will get its data from Kafka and then write to HBase.</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;"><br /></span>
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;">One interesting item within Apache Storm is stream grouping (how tuples produced by spouts will be handled by available bolts). To read more about stream grouping, you can refer <a href="https://storm.apache.org/documentation/Concepts.html" target="_blank">here</a>.</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;"><br /></span><span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;">Lastly, if you want to try it out yourself, you can download the sample codes <a href="https://docs.google.com/uc?authuser=0&id=0B6EFlN-9iDHtZnhiVTl1WEZJVFE&export=download">here</a>.</span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzmf89yLlR1yS4I7ekCB7CSVPCQH495COfYnKEudTagjO-b6gSy04-KbSGi0JTkdbYGzOyfN8iitOp8YPMondZmFld3J-vfKr2gJPIdx3uL97w6j0iocZwBp8KvX-DsFXgy9A2XjLXXBQ/s1600/hbasebolt1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzmf89yLlR1yS4I7ekCB7CSVPCQH495COfYnKEudTagjO-b6gSy04-KbSGi0JTkdbYGzOyfN8iitOp8YPMondZmFld3J-vfKr2gJPIdx3uL97w6j0iocZwBp8KvX-DsFXgy9A2XjLXXBQ/s1600/hbasebolt1.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>Storm UI that shows Kafka spout and HBase bolt processing</b></span></td></tr>
</tbody></table>
Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-77289852297727771692015-03-10T16:32:00.000+08:002015-04-12T16:44:59.520+08:00Apache Storm: How to add external JARs or packages into CLASSPATH while running 'storm jar'?<span style="font-family: Arial, Helvetica, sans-serif;">I have been playing with </span>'<a href="https://github.com/apache/storm/tree/master/external/storm-kafka" target="_blank">storm-kafka</a>' and '<a href="https://github.com/apache/storm/tree/master/external/storm-hbase" target="_blank">storm-hbase</a>' lately. Basically, they are projects/tools that one can use to integrate Kafka and HBase with storm.<br />
<br />
My project was to have Kafka as spout and HBase as bolt. In other words, my application will pull data from Kafka and then write the output to HBase. In other other words (pun intended :), I need to have the storm-kafka and storm-hbase JAR included when I run my Storm topology.<br />
<br />
There are a few ways to do this:<br />
(1) Put the JARs under STORM_BASE_DIR<br />
(2) Put the JARs under STORM_BASE_DIR/lib<br />
(3) Put the package under STORM_CONF_DIR<br />
(4) Include the package into the topology JAR<br />
<br />
After trying the above few methods, my favourite is method #4. However, it is not without its own pain points.<br />
<br />
Let me explain why I do not like the other methods.<br />
<br />
Method #1<br />
=======<br />
By putting JARs into STORM_BASE_DIR, I have a feeling that I have 'corrupted' the directory. Messing up a standard directory of a product is not my cup of tea.<br />
<br />
Method #2<br />
=======<br />
See Method #1 above.<br />
<br />
Method #3<br />
=======<br />
Since the storm.py codes do not search the directory declared as STORM_CONF_DIR (or USER_CONF_DIR) for JARs, you would have to put the package files in that directory. How many times have I said 'messy'? :)<br />
<br />
Now, let's discuss Method #4. I say it is my favourite, but I never say it is the best. That is because it will grow your JAR file size greatly if you have some really big external JARs to include (beside your topology). However, I feel that it is the most acceptable approach because it is more manageable than the other methods (at least to me :).<br />
<br />
Hence, if you are looking into including or adding external JARs while running your Storm topology, I would suggest you to include those JARs into your topology JAR for the time being until there is a neater way to do this!<br />
<br />
<b>NOTE</b>:<br />
Environment = HDP 2.2 (Storm 0.9.3)<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWODWnr3x54UL_1dukvuhWIn22mOm7WcyqbKpGGt_XYapd2tS6b-8SXvGezcEn1y0ILavITt8x5pGpM2t2RgiF138Imz8GGVBQnYwa5o6iqzbSTqaWYO6khhBj0jvb32kGb2i5O1uIp6o/s1600/storm1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWODWnr3x54UL_1dukvuhWIn22mOm7WcyqbKpGGt_XYapd2tS6b-8SXvGezcEn1y0ILavITt8x5pGpM2t2RgiF138Imz8GGVBQnYwa5o6iqzbSTqaWYO6khhBj0jvb32kGb2i5O1uIp6o/s1600/storm1.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>Source codes of storm.py and how CLASSPATH is determined</b></span></td></tr>
</tbody></table>
Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com7tag:blogger.com,1999:blog-4943934699922259568.post-23611471820321745662015-03-03T10:09:00.000+08:002015-04-12T16:45:26.146+08:00Apache Mesos: Mesos + Marathon + Docker = ?<span style="font-family: Arial, Helvetica, sans-serif;">In my previous <a href="http://manfrix.blogspot.com/2015/02/apache-mesos-when-all-becomes-one.html" target="_blank">post</a>, I talked about merging resources from multiple nodes into one using Apache Mesos.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">I also talked about the 2 reasons I decided to pick up Mesos:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(1) Google Kubernetes</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(2) Docker</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">In this post, I am going to share the method you can use to deploy Docker container on a Mesos cluster.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">* Mesos and Zookeeper have to be up and running. For installation instruction, please refer to my previous <a href="http://manfrix.blogspot.com/2015/02/apache-mesos-when-all-becomes-one.html" target="_blank">post</a>.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">** I used CentOS 6.6 as the platform. So, some commands might differ on other platforms.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">*** Make sure "docker-io" "(the Docker package) is installed and the daemon is running on all Mesos slave nodes.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(1) Check and update (if needed) the /etc/mesos/zk file on the node you wish to install Marathon.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">vi /etc/mesos/zk</span>
<br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">** Make sure the IP address of the Zookeeper server is written there</span></td>
</tr>
</tbody></table>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">(2) Install <a href="https://github.com/mesosphere/marathon" target="_blank">Marathon</a> (using the Mesosphere repository created in the previous post) on the node selected above (Master node is recommended for testing purpose and ease of maintenance):</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">yum install marathon</span>
</td>
</tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(3) Make sure Marathon is up and running after the installation. Otherwise, start it using:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">initctl start marathon</span> </td>
</tr>
</tbody></table>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">** Use "ps -ef" command to verify the Marathon process is running with the proper IP addresses (instead of 'localhost') of the zookeeper-server and Master node. If it is not, check the /etc/mesos/zk file again and restart.</span><br />
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(4) Update all the Mesos slave nodes with the following:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">echo 'docker,mesos' > /etc/mesos-slave/containerizers</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">echo '5 mins' > /etc/mesos-slave/executor_registration_timeout</span></td>
</tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(5) Restart all the Mesos slave nodes:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">initctl restart mesos-slave</span> </td>
</tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(6) By now, you should be ready to deploy Docker container on the Mesos cluster. To do so, you have to create a JSON file for the Docker container you wish to deploy:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">Eg.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">{</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> "container": {</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> "type": "DOCKER",</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> "docker": {</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> "image": "192.168.0.210:5000/centos63:httpd",</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> "network": "HOST"</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> }</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> },</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> "id": "centos63",</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> "instances": 1,</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> "cpus": 4,</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> "mem": 2048,</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> "uris": [],</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> "cmd": "/usr/sbin/httpd -DFOREGROUND"</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">}</span><br />
<div>
<br /></div>
</td>
</tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">There are a few things to take note:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(a) For the "image" parameter, you would need to specify an image that is reachable by all of the Mesos slaves, because you would not know for sure which slave or slaves the Master will select to run the container.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">** If you are using your own insecured private registry, please make sure you edit the docker "default" file (eg. /etc/sysconfig/docker or /etc/default/docker) to declare the registry as insecured and restart the Docker service (service docker restart):</span></div>
<div>
<br /></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">other_args="--insecure-registry 192.168.0.210:5000"</span> </td>
</tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(b) AFAIK, Mesos (0.21.1) only support 2 network modes now - HOST or BRIDGE. </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(c) The "cmd" parameter works like CMD in Docker.</span></div>
<div>
<br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">(7) Once the JSON file is ready, you can submit to Marathon using the POST method:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">curl -X POST -H "Content-Type: application/json" http://<marathon host>:8080/v2/apps -d@<JSON filename></span> </td>
</tr>
</tbody></table>
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjDNaikfXavDj50kXmf28GHi7n2LPnCtxGW54fQKZMM4B1dtZsEYxg_i0L8y-LyIp39lwwjLUbV0ZzYmFDtzmN5mefu9zavkKsPZu-FuLmhxvUBsEubUrxeH-ZCFC3UufYvhlZwMuavLg/s1600/marathon1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjDNaikfXavDj50kXmf28GHi7n2LPnCtxGW54fQKZMM4B1dtZsEYxg_i0L8y-LyIp39lwwjLUbV0ZzYmFDtzmN5mefu9zavkKsPZu-FuLmhxvUBsEubUrxeH-ZCFC3UufYvhlZwMuavLg/s1600/marathon1.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>Marathon GUI: After the CURL command and when Mesos is deploying the container</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9VaM_5z5rl9MlA7GF-HsWUmLlBBRtrO_wMoHYijA-AZuaVZ1QgxqV4D60fG43XqSFs8euKzaXHaNTKWhc0IzaeGG9QmfHYZgPhI_gnlH5TinKaFyTdkysdgF5UXLmhtLxGH0sIxiCcJs/s1600/marathon2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9VaM_5z5rl9MlA7GF-HsWUmLlBBRtrO_wMoHYijA-AZuaVZ1QgxqV4D60fG43XqSFs8euKzaXHaNTKWhc0IzaeGG9QmfHYZgPhI_gnlH5TinKaFyTdkysdgF5UXLmhtLxGH0sIxiCcJs/s1600/marathon2.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>Marathon GUI: The container is successfully deployed and RUNNING</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPwKyi5XTZwxHBcias3vJYZnvF9H3hpDSz6G0OAKE81etXlAMuH7VU7iGzGXpAZhIHSEN5knmz-5pXVaEQyRpvTsLE5iQ54qesla9s8jxpEegsoBrQcjKmgjvg3xxcVz1UjAMhFzPNaFY/s1600/marathon3.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPwKyi5XTZwxHBcias3vJYZnvF9H3hpDSz6G0OAKE81etXlAMuH7VU7iGzGXpAZhIHSEN5knmz-5pXVaEQyRpvTsLE5iQ54qesla9s8jxpEegsoBrQcjKmgjvg3xxcVz1UjAMhFzPNaFY/s1600/marathon3.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>Mesos GUI: Shows one active task running on "mesos4" slave node</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKNGakSOC65yzRQp8ckXY_5uEydAs1AlOFRLSkWmTcaQxlDrYfdkCsN5Ybnr9ccGb4-2r4G7PgAygn1BIaVUR6N7y-fZF0kKMBlPUcX4Afbbaa1GwPiHam5GNeM1Oe6u67ViQdDebmgXA/s1600/marathon4.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKNGakSOC65yzRQp8ckXY_5uEydAs1AlOFRLSkWmTcaQxlDrYfdkCsN5Ybnr9ccGb4-2r4G7PgAygn1BIaVUR6N7y-fZF0kKMBlPUcX4Afbbaa1GwPiHam5GNeM1Oe6u67ViQdDebmgXA/s1600/marathon4.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>Mesos GUI: Clicking on the task shows the details (it's a SANDBOX)</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicqFI7FWdQz3K8czKI-IC9-yFhysZWtRRCuVvyqjhUUfZEJ-2tiHjP1zLxk8l0Q3WbJ3k43r6dtYsGdI02K9qfof34rBdG6NAlV9NeKp0w7vYuz1B-aIUwb_M-UvposWAI97ils3Y0-W0/s1600/marathon5.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicqFI7FWdQz3K8czKI-IC9-yFhysZWtRRCuVvyqjhUUfZEJ-2tiHjP1zLxk8l0Q3WbJ3k43r6dtYsGdI02K9qfof34rBdG6NAlV9NeKp0w7vYuz1B-aIUwb_M-UvposWAI97ils3Y0-W0/s1600/marathon5.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>Mesos GUI: STDOUT and STDERR are streamed from the container to the sandbox</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS_b7EpGdX3m9e-5fTKb8pjXAJXJDTlVROhyphenhyphenFSpggK907yMKNmxLH7V4qV00n2XcNC2tV3wh-2ijKKepUg9O5Cq_bVLIPMzfiv__2qUGCyA302A-PN8cabxfowCUqjLBSj_gsnL7w6J90/s1600/marathon6.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS_b7EpGdX3m9e-5fTKb8pjXAJXJDTlVROhyphenhyphenFSpggK907yMKNmxLH7V4qV00n2XcNC2tV3wh-2ijKKepUg9O5Cq_bVLIPMzfiv__2qUGCyA302A-PN8cabxfowCUqjLBSj_gsnL7w6J90/s1600/marathon6.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>On 'mesos4' node, the image is downloaded from the private repo and a container is running</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_XlvLpQIZBaEOkDSlHhTHl19xlbGQaFYqZKsEuKhtz5-zXPElYSAx2hXkUiLOrXxlKr3eZEpKffe7FoP-3p1O5r1vGfTJyvRpeERpKdBivq5djWOxDusDiq7AO61ymZSXCXjjg-wOuxo/s1600/marathon7.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_XlvLpQIZBaEOkDSlHhTHl19xlbGQaFYqZKsEuKhtz5-zXPElYSAx2hXkUiLOrXxlKr3eZEpKffe7FoP-3p1O5r1vGfTJyvRpeERpKdBivq5djWOxDusDiq7AO61ymZSXCXjjg-wOuxo/s1600/marathon7.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>On 'mesos4' node, 'docker inspect <container id>' shows the networking mode is HOST as configured</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCiuLwpO3eR_UvDTr3zjeN_3P-3vmvcXERRVcvJNVhvsXxwkmayNnefzUH0btAQbttUy_dWTwxam5H8hi21FbVQ5k6nDxfNDVWtCR08SfeAZqO9ZqtGnduTxZIJ-gD_DxU0hgaFUQd1Qc/s1600/marathon8.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCiuLwpO3eR_UvDTr3zjeN_3P-3vmvcXERRVcvJNVhvsXxwkmayNnefzUH0btAQbttUy_dWTwxam5H8hi21FbVQ5k6nDxfNDVWtCR08SfeAZqO9ZqtGnduTxZIJ-gD_DxU0hgaFUQd1Qc/s1600/marathon8.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>On 'mesos3' node, a HTTP connection shows HTTPD container is indeed running on 'mesos4' node</b></span></td></tr>
</tbody></table>
Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-83855207968360184162015-02-27T09:13:00.000+08:002015-04-12T16:45:52.541+08:00Apache Mesos: When All Becomes One<span style="font-family: Arial, Helvetica, sans-serif;">First of all, for the benefits of those unfamiliar with <a href="http://mesos.apache.org/">Apache Mesos</a>, this is the "what is" taken from its official website:</span><br />
<br />
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><h3 style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 24px; font-weight: 500; line-height: 1.1; margin-bottom: 10px; margin-top: 20px;">
What is Mesos?</h3>
<h4 style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 18px; font-weight: 500; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
A distributed systems kernel</h4>
<div style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-bottom: 10px;">
Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments.</div>
</td>
</tr>
</tbody></table>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Beside the fact that I have always wanted to learn a technology that can merge all the resources available on my multiple machines into one, I am out to learn Apache Mesos for the following</span><span style="font-family: Arial, Helvetica, sans-serif;"> 2 reasons (currently):</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(1) <a href="http://kubernetes.io/" target="_blank">Google Kubernetes</a></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(2) <a href="https://www.docker.com/" target="_blank">Docker</a></span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">Installing Apache Mesos isn't too hard if you follow the instruction available on its <a href="http://mesos.apache.org/gettingstarted/" target="_blank">official website</a>, but I would like to share an alternative installation method which I found is more straightforward:</span>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">* Instructions only suitable for RHEL/CentOS 6 (tested on CentOS 6.6). For other platforms, refer <a href="http://mesosphere.com/docs/getting-started/datacenter/install/" target="_blank">here</a>.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">** Run all instructions as 'root' user for simplicity.</span>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(1) On the node or VM image that you would like to designate as the <b>Master and all slave nodes</b>, execute the following command to create the Mesosphere repository:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">rpm -Uvh http://repos.mesosphere.io/el/6/noarch/RPMS/mesosphere-el-repo-6-2.noarch.rpm</span>
</td>
</tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(2) On the <b>Master and all the slave nodes</b>, install Mesos:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">yum -y install mesos
</span></td>
</tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(3) Even though you only plan to have a single Master node, it is advisable to install Zookeeper (just in case you want to expand in the future):</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">
rpm -Uvh http://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">yum -y install zookeeper-server
</span></td>
</tr>
</tbody></table>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">* You can install the Zookeeper server on <b>either the Master node</b> (preferred for ease of maintenance) <b>or any of the slave node</b>.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">** You need to have <b>Java</b> installed for Zookeeper to work properly.</span><br />
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(4) On the <b>Master</b> node, initialize Zookeeper:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">
service zookeeper-server init </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">echo 1 | sudo tee -a /var/lib/zookeeper/myid >/dev/null
</span></td>
</tr>
</tbody></table>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">(5) On the <b>Master</b> node, stop and disable mesos-slave:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">initctl stop mesos-slave</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">
cd /etc/init/ </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">mv mesos-slave.conf mesos-slave.disable
</span></td></tr>
</tbody></table>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">(6) On <b>all the slave</b> <b>nodes</b>, stop and disable mesos-master:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">initctl stop mesos-master</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">
cd /etc/init/ </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">mv mesos-master.conf mesos-master.disable
</span></td></tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(7) On the <b>Master</b> node, set the IP address:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">
echo <IP of the Master node> | sudo tee /etc/mesos-master/ip
</span></td></tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(8) On the <b>Master</b> node, set the name of the cluster:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">
echo <cluster name> | sudo tee /etc/mesos-master/cluster </span></td></tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(9) On the <b>Master and all slave</b> <b>nodes</b>, set the URL of the Zookeeper server:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">
echo zk://<IP of the Zookeeper server>:2181/mesos | sudo tee /etc/mesos/zk
</span></td></tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(10) On <b>all the slave nodes</b>, set their respective IP address:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">
echo </span><span style="font-family: Arial, Helvetica, sans-serif;"><IP of the Slave node> </span><span style="font-family: Arial, Helvetica, sans-serif;"> | sudo tee /etc/mesos-slave/ip </span></td></tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(11) On the <b>Master</b> node, restart mesos-master and Zookeeper (if it is installed there):</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">
service zookeeper restart </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">initctl restart mesos-master
</span></td></tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(12) On <b>all the slave nodes</b>, restart mesos-slave:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">
initctl restart mesos-slave
</span></td></tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(12) Verify that the Master is running and all slaves are registered with it:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<table>
<tbody>
<tr>
<td style="border: 1px solid gray;"><span style="font-family: Arial, Helvetica, sans-serif;">
http://<IP of the Master node>:5050</span></td></tr>
</tbody></table>
<br />
<div>
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSNBOUuxLgrsrGwFwtZpWn-Znf51Vze89aY44jBDOHvulm1Fk1FS8lp0et3sHhV_-oe9vOwvqC0ffzZuypnSCI754SEAoUMwsPtEUMxopM1F3xE5YVebFAtuh4FtemXKhN_475cgBqn2M/s1600/mesos1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSNBOUuxLgrsrGwFwtZpWn-Znf51Vze89aY44jBDOHvulm1Fk1FS8lp0et3sHhV_-oe9vOwvqC0ffzZuypnSCI754SEAoUMwsPtEUMxopM1F3xE5YVebFAtuh4FtemXKhN_475cgBqn2M/s1600/mesos1.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>When the Master is first initialized</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgaiKVvimHyJCU9iH5Vaqt1NXl-QBjKowh3cxntnbzHmClBGPjMomj_LgHPn4f_qVioSZm5VK47zx0A0a3yPPaCw0T9p1meQtG9iEOmf86vIxyMaTFJYsQfRZz8PVgCcnunLm3qdcffcY/s1600/mesos2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgaiKVvimHyJCU9iH5Vaqt1NXl-QBjKowh3cxntnbzHmClBGPjMomj_LgHPn4f_qVioSZm5VK47zx0A0a3yPPaCw0T9p1meQtG9iEOmf86vIxyMaTFJYsQfRZz8PVgCcnunLm3qdcffcY/s1600/mesos2.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>When the first slave joined</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAhF-MERnfKEhGiO2qTXlzrMpJJBGdsBjE0NeSDKf5uf52lRg_yl0glPJAQlZ3cb_yq-cCkd7M5ccmwqegdAW3khn7WMAVCjGm3Vrusz52r_EpmFjiy2bimXR_xMR2ORo9LEowzGA7agY/s1600/mesos3.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAhF-MERnfKEhGiO2qTXlzrMpJJBGdsBjE0NeSDKf5uf52lRg_yl0glPJAQlZ3cb_yq-cCkd7M5ccmwqegdAW3khn7WMAVCjGm3Vrusz52r_EpmFjiy2bimXR_xMR2ORo9LEowzGA7agY/s1600/mesos3.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>When the second slave joined</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgsxvpUFNj1VM99mPYRqof1BDOQwtmjaUHJhFhr1kP24rAW5U23Yom8qbCfTIO4jlCP9318NqXqjUFCeC82OEsFtI53HCd2pwULXtQeOCFoxs40FlAuN2Xq9KeEPPmZzEtVwEoZZOqgEI/s1600/mesos4.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgsxvpUFNj1VM99mPYRqof1BDOQwtmjaUHJhFhr1kP24rAW5U23Yom8qbCfTIO4jlCP9318NqXqjUFCeC82OEsFtI53HCd2pwULXtQeOCFoxs40FlAuN2Xq9KeEPPmZzEtVwEoZZOqgEI/s1600/mesos4.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>When the third slave joined</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiirDvpREiL8iR817ocNCV7UyVlYB-VBhFRR_fVeZUnEA_h6rWaEMc0oNHR_8-DOE3DNiVvSOPidlTu1Vj8dmugNZlxXBllmuEZ0uUmSc-xJi26m21o2QgoHv1kgyBXR8R5PqO1Nb5qAC0/s1600/mesos5.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiirDvpREiL8iR817ocNCV7UyVlYB-VBhFRR_fVeZUnEA_h6rWaEMc0oNHR_8-DOE3DNiVvSOPidlTu1Vj8dmugNZlxXBllmuEZ0uUmSc-xJi26m21o2QgoHv1kgyBXR8R5PqO1Nb5qAC0/s1600/mesos5.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>All the slaves</b></span></td></tr>
</tbody></table>
Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-40372219657303024892015-02-16T09:09:00.003+08:002015-04-18T20:32:45.802+08:00Docker: How to Create Your Own Base Image (CentOS/RHEL)<span style="font-family: Arial, Helvetica, sans-serif;">Normally, we will pull base images from the Docker Hub registry to build our own images.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">However, there might be times when you want more control over the base image (size, packages, etc.). Luckily, Docker provides a way to do that.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">For general information about building your own image, refer <a href="https://docs.docker.com/articles/baseimages/" target="_blank">here</a>. Since I am a fan of CentOS/RHEL, I would normally go <a href="https://github.com/docker/docker/blob/master/contrib/mkimage-yum.sh" target="_blank">here</a>. For a more "friendly" version of the script, go <a href="https://raw.githubusercontent.com/docker/docker/master/contrib/mkimage-yum.sh" target="_blank">here</a>.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">To create your own CentOS/RHEL image, follow these instructions:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(1) Copy or download the script to a running CentOS/RHEL system with yum properly setup. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>NOTE</b>: I tested the script on CentOS 6.5.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<input class="spoilerbutton" onclick="this.value=this.value=='Show Codes'?'Hide Codes':'Show Codes';" type="button" value="Show Codes" />
<br />
<div class="spoiler">
<div>
<table cellpadding="2" cellspacing="0" style="border: 1px solid #ddd;">
<tbody>
<tr>
<td><pre class="brush: csharp">#!/usr/bin/env bash
#
# Create a base CentOS Docker image.
#
# This script is useful on systems with yum installed (e.g., building
# a CentOS image on CentOS). See contrib/mkimage-rinse.sh for a way
# to build CentOS images on other systems.
usage() {
cat <<EOOPTS
OPTIONS:
-y <yumconf> The path to the yum config to install packages from. The
default is /etc/yum.conf.
EOOPTS
exit 1
}
# option defaults
yum_config=/etc/yum.conf
while getopts ":y:h" opt; do
case $opt in
y)
yum_config=$OPTARG
;;
h)
usage
;;
\?)
echo "Invalid option: -$OPTARG"
usage
;;
esac
done
shift $((OPTIND - 1))
name=$1
if [[ -z $name ]]; then
usage
fi
#--------------------
target=$(mktemp -d --tmpdir $(basename $0).XXXXXX)
set -x
mkdir -m 755 "$target"/dev
mknod -m 600 "$target"/dev/console c 5 1
mknod -m 600 "$target"/dev/initctl p
mknod -m 666 "$target"/dev/full c 1 7
mknod -m 666 "$target"/dev/null c 1 3
mknod -m 666 "$target"/dev/ptmx c 5 2
mknod -m 666 "$target"/dev/random c 1 8
mknod -m 666 "$target"/dev/tty c 5 0
mknod -m 666 "$target"/dev/tty0 c 4 0
mknod -m 666 "$target"/dev/urandom c 1 9
mknod -m 666 "$target"/dev/zero c 1 5
yum -c "$yum_config" --installroot="$target" --releasever=/ --setopt=tsflags=nodocs \
--setopt=group_package_types=mandatory -y groupinstall Core
yum -c "$yum_config" --installroot="$target" -y clean all
cat > "$target"/etc/sysconfig/network <<EOF
NETWORKING=yes
HOSTNAME=localhost.localdomain
EOF
# effectively: febootstrap-minimize --keep-zoneinfo --keep-rpmdb
# --keep-services "$target". Stolen from mkimage-rinse.sh
# locales
rm -rf "$target"/usr/{{lib,share}/locale,{lib,lib64}/gconv,bin/localedef,sbin/build-locale-archive}
# docs
rm -rf "$target"/usr/share/{man,doc,info,gnome/help}
# cracklib
rm -rf "$target"/usr/share/cracklib
# i18n
rm -rf "$target"/usr/share/i18n
# sln
rm -rf "$target"/sbin/sln
# ldconfig
rm -rf "$target"/etc/ld.so.cache
rm -rf "$target"/var/cache/ldconfig/*
version=
if [ -r "$target"/etc/redhat-release ]; then
version="$(sed 's/^[^0-9\]*\([0-9.]\+\).*$/\1/' "$target"/etc/redhat-release)"
fi
if [ -z "$version" ]; then
echo >&2 "warning: cannot autodetect OS version, using '$name' as tag"
version=$name
fi
tar --numeric-owner -c -C "$target" . | docker import - $name:$version
docker run -i -t $name:$version echo success
rm -rf "$target"
</pre>
</td></tr>
</tbody></table>
</div>
</div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(2) Save the script and give it execute permission (chmod 700 <script>).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(3) Login as 'root'.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(4) Amend the script as necessary to suit your purpose (eg. more packages, etc).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(5) Execute the script (eg. createDockerImage.sh centos).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>WHERE</b> "centos" is the name that you would like the resultant image to have</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>NOTE</b>: The script will tag the image using the version of the OS (or the "name" provided if the version cannot be determined).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(6) Once the script returns, run "docker images" to confirm the image is created.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(7) Verify that the image is ok by launching a container using "docker run -t -i <image> /bin/bash".</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">If everything works out fine, you would now have your custom built base image!</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-88942065248572814522015-02-11T21:01:00.004+08:002015-04-12T16:46:56.428+08:00Apache Storm: Integration with Kafka using Kafka Spout<span style="font-family: Arial, Helvetica, sans-serif;">If you have been playing with either Apache Kafka or Apache Storm, you would have read so much articles about integration between the two. From my experience, reading too much can be a bad thing sometimes (pun intended :). In this case, there were multiple efforts that try to offer such integration. Thus, it might caused confusion about which is the best or standard way to do it.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">It is good to know that starting from version 0.9.2-incubating, Apache Storm has decided to include such support officially. Read more <a href="https://storm.apache.org/2014/11/25/storm093-released.html" target="_blank">here</a>.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Anyway, how does such integration work?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">In this blog entry, I am only going to share information about using Kafka as a Storm spout. Yes, starting from Storm version 0.9.3, you can use Kafka as a bolt too. If you want to know more about Topology, Spout and Bolt, read <a href="http://storm.apache.org/documentation/Tutorial.html" target="_blank">this</a>.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Basically, the classes you need for the Storm-Kafka integration are available under storm.kafka.* package. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">If you want to get up to speed quick, try out the sandbox offered by Hortonworks <a href="http://hortonworks.com/hdp/downloads/" target="_blank">here</a>. After you have downloaded the sandbox (or if you are gutsy enough to install the system through Ambari), it is advisable to try out the <a href="http://hortonworks.com/tutorials/" target="_blank">tutorial</a> too. If you want to jump straight to the tutorial related to the Storm-Kafka integration, you can go <a href="http://hortonworks.com/hadoop-tutorial/ingesting-processing-real-time-events-apache-storm/" target="_blank">here</a>. Please take note that the tutorial contains the source codes too, so make sure you check them out!</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Once you get a hang of it, you can move over to this <a href="https://github.com/apache/storm/blob/v0.9.3/external/storm-kafka/README.md" target="_blank">website</a> to learn more about Storm Kafka.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">If you do not want to compile the Storm Kafka package yourself, you can download it from the <a href="http://repo.hortonworks.com/content/repositories/releases/org/apache/storm/storm-kafka/" target="_blank">Hortonworks maven repository</a>.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />The information offered here should get you going for a while, and I will share some tips and traps regarding the integration in future entries.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Happy hacking!</span>Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-56005019336383102702015-02-09T11:31:00.000+08:002015-04-12T16:47:31.006+08:00Docker: Why delete does not reduce the image size?<span style="font-family: Arial, Helvetica, sans-serif;">If you wonder why your Docker image does not shrink in size after you have deleted some very large files, then you might want to read further for a brief explanation and solution!</span><br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">In short, the cause of all these drama is the union filesystem (read <a href="https://docs.docker.com/terms/layer/" target="_blank">here</a>). If you work on an image and have performed multiple commits (either through DockerFile or manually), then you might run into this problem later in the stage. The large files might have been committed in one of the layers during the earlier stage.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Eg.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(1) We have this image called "<b>centos63:omnibus731fp8-hm2000</b>" which is about <b>2.11GB</b> in size.</span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEYx3s6VEuWdu6hylA1zUQgKHep2dVxiqWB02XrrIaZ-egkWItTS0jOgm85vXQ9GWV5dbjFcV43hNfS2Pj1Trkz_uxuZCYZ6f34mf_wMYa-NVyIiNpTEYV5wWwoiy_pEZKH993EksFjK4/s1600/docker1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEYx3s6VEuWdu6hylA1zUQgKHep2dVxiqWB02XrrIaZ-egkWItTS0jOgm85vXQ9GWV5dbjFcV43hNfS2Pj1Trkz_uxuZCYZ6f34mf_wMYa-NVyIiNpTEYV5wWwoiy_pEZKH993EksFjK4/s1600/docker1.png" width="100%" /></a></td></tr>
</tbody></table>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(2) Next, let us run the image and delete a very large directory in there.</span></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXbWvSL89FtNbHmKGz_5KIsayqJKCuTL8lpRJTxQ4XRym6daYC52MKum9M52FrfJ-hdHXFHWm9zWJ8ejC-UC5_HLF7VWlZQJhloac1qF-DgX7oejz9BsrCzMkmFHOCv4HUB8cmE8PbRvc/s1600/docker2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXbWvSL89FtNbHmKGz_5KIsayqJKCuTL8lpRJTxQ4XRym6daYC52MKum9M52FrfJ-hdHXFHWm9zWJ8ejC-UC5_HLF7VWlZQJhloac1qF-DgX7oejz9BsrCzMkmFHOCv4HUB8cmE8PbRvc/s1600/docker2.png" width="100%" /></a></td></tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(3) To persist the change, I committed the image and tagged the new image as "<b>centos63:omnibus731fp8-hm2000-<span style="color: #3d85c6;">rm</span></b>".</span></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLVR4DDGIIZXdDYuyLblxxLsn3o31WZTZn9xA2k4_EPrrwCUknsT6o9NwuJu0644Ywh63DVOwtFdU2H5TjA6U1MuLvh-j_zGCOVvWe0hkdN8KRXVA8eHaGLvstLUeJu97Xu-bE70vk3D8/s1600/docker3.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLVR4DDGIIZXdDYuyLblxxLsn3o31WZTZn9xA2k4_EPrrwCUknsT6o9NwuJu0644Ywh63DVOwtFdU2H5TjA6U1MuLvh-j_zGCOVvWe0hkdN8KRXVA8eHaGLvstLUeJu97Xu-bE70vk3D8/s1600/docker3.png" width="100%" /></a></td></tr>
</tbody></table>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(4) Surprise, surprise! The size is still <b>2.11GB</b>!?!?</span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsdDxFCBx4CBld13hdkNbIAg7Z55ygvBynyA9uMmW0ngj4KHBowK7utOZHGW_fQ-dLYhEkrrabG7rMBaQ5nqqsqWvjbhj6KkOCUMnqFDBZXrM64YmxCcSZqWnUGtUQWijIPDt8qGmoH6A/s1600/docker4.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsdDxFCBx4CBld13hdkNbIAg7Z55ygvBynyA9uMmW0ngj4KHBowK7utOZHGW_fQ-dLYhEkrrabG7rMBaQ5nqqsqWvjbhj6KkOCUMnqFDBZXrM64YmxCcSZqWnUGtUQWijIPDt8qGmoH6A/s1600/docker4.png" width="100%" /></a></td></tr>
</tbody></table>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(5) What happened? Let's check it out. Is the large directory still there? Nope!!!</span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXCVxG6XY92wUGM8-um00t7uWP-sahzi0lvUrKC5YITZjv2-DRr4Ty2KzuT4zKtt_AOQ844lwD0PrPLlKETJGZasUXehcVoyT0_id6uGhFvw3wc9s8fDz564M30LR_vTrWImJnx1tt_c0/s1600/dcoker5.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXCVxG6XY92wUGM8-um00t7uWP-sahzi0lvUrKC5YITZjv2-DRr4Ty2KzuT4zKtt_AOQ844lwD0PrPLlKETJGZasUXehcVoyT0_id6uGhFvw3wc9s8fDz564M30LR_vTrWImJnx1tt_c0/s1600/dcoker5.png" width="100%" /></a></td></tr>
</tbody></table>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(6) Then why didn't the "rm" work?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">You can see that the large directory was most probably (sorry I can't tell for sure because the image was created with a previous very bad habit of manual commits :) added in image "28729cfb27de" that was created very early in the stage. Ever since, there were multiple commits (represented by those multiple different images in the history). </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Another thing to pay attention to is the "rm" command that was persisted in image "ae11f957b955" (latest - highlighted in <span style="color: #3d85c6;">BLUE</span> box) and the size was only 7 bytes!!! It shows clearly that Docker has only recorded the "rm" command and not the result of it.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">That is the "power" of an union filesystem. But sorry, your large directory was still technically in there.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjo0mXIeTvZCtaVDk6-SEF7677CrqpISzVTqXAfeiGLECZKG2Fv2RnzqBZBzdnuoOtCfCpmYiW5YHrNTzifR3AZkTDH6Evbp4szdGbi3MtHFZsZkYB4UvsAGU10CCu2DYmGVm14nIMO0Qk/s1600/docker6.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjo0mXIeTvZCtaVDk6-SEF7677CrqpISzVTqXAfeiGLECZKG2Fv2RnzqBZBzdnuoOtCfCpmYiW5YHrNTzifR3AZkTDH6Evbp4szdGbi3MtHFZsZkYB4UvsAGU10CCu2DYmGVm14nIMO0Qk/s1600/docker6.png" width="100%" /></a></td></tr>
</tbody></table>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(7) So, what now? How do you trim those extra fat off? The answer is with this awesome tool called <a href="https://github.com/jwilder/docker-squash" target="_blank">docker-squash</a>!<br /><br />I executed the tool with an option to name the output image "<b>centos63:omnibus731fp8-hm2000-squashed</b>".</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">You can see how the tool successfully determined all the layers belonging to the image (check the UUID).</span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwBEJL6MRSRv8ZtcDBFtTk0uWc91u-hzPekvAESxfaj8RDScJ2yGz9wGOVBjMqEBjJl9JQ58YsLNnd-UnWYwnhnE9sD3maygkIBXggmogEbnV9Q_VxvOzuYNGIpgQfHP0obJViWdR3qAo/s1600/docker7.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwBEJL6MRSRv8ZtcDBFtTk0uWc91u-hzPekvAESxfaj8RDScJ2yGz9wGOVBjMqEBjJl9JQ58YsLNnd-UnWYwnhnE9sD3maygkIBXggmogEbnV9Q_VxvOzuYNGIpgQfHP0obJViWdR3qAo/s1600/docker7.png" width="100%" /></a></td></tr>
</tbody></table>
<span style="font-family: Arial, Helvetica, sans-serif;"><br />(8) From the output of the tool, it seems that it has successfully trimmed down the image.</span><br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEStP-J2-nWo8wQlq1doIK5H0V-HjlZfPdOsDKVkjFYk-f28lwOc6Hf2eDOnCMskX8wLNKyaFwM8CV66XJPgR3iuVuXjFt8tLfpAsXg3vPkryXSjLqLV4XhBaldU0nALGI62MvzPa_8Iw/s1600/docker8.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEStP-J2-nWo8wQlq1doIK5H0V-HjlZfPdOsDKVkjFYk-f28lwOc6Hf2eDOnCMskX8wLNKyaFwM8CV66XJPgR3iuVuXjFt8tLfpAsXg3vPkryXSjLqLV4XhBaldU0nALGI62MvzPa_8Iw/s1600/docker8.png" width="100%" /></a></td></tr>
</tbody></table>
<span style="font-family: Arial, Helvetica, sans-serif;"><br />(9) Let's verify whether the new image "omnibus731fp8-hm2000-squashed" is significantly smaller in size. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">It seems that the tool does deliver after all! :)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmKARqr07Or32Z_GZRFi6UdixsWAlKnapQlxvD8yACe-gkURWHxoy5Hmw3rIjHDomkIISGdVoh7_2wHD8gjas_ui-iiST-wWy9U6JqfHTE45a54R2ixTEJj1I3TNs6Bk13muGfs9DpOjc/s1600/docker9.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmKARqr07Or32Z_GZRFi6UdixsWAlKnapQlxvD8yACe-gkURWHxoy5Hmw3rIjHDomkIISGdVoh7_2wHD8gjas_ui-iiST-wWy9U6JqfHTE45a54R2ixTEJj1I3TNs6Bk13muGfs9DpOjc/s1600/docker9.png" width="100%" /></a></td></tr>
</tbody></table>
<span style="font-family: Arial, Helvetica, sans-serif;"></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br />I have read that Docker Inc is working on having this feature (shrinking the image size) built-in. So, while waiting for it, you have this tool!</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-45690124343463057992015-02-05T18:00:00.000+08:002015-04-12T16:47:40.813+08:00Java: Telnet Client<span style="font-family: Arial, Helvetica, sans-serif;">I was going through my personal projects in search for a topic to write for this blog when I stumbled upon this very old codes.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">I wrote this back in Year 2003 and that was 12 years ago!</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Back then, I was learning Java and somehow got hooked on reading RFCs. So, I guess it was natural for me to write a Telnet Client based on <a href="http://en.wikipedia.org/wiki/Telnet#Related_RFCs" target="_blank">Telnet RFCs</a>. I have to admit upfront that I did not implement all of them, but basic things do work :)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJAUnfetRYtOoYi0fWMR1zfoayE9CbVWDd2oMosPhTBz7Og2D7_veJgfcJlCdI1uHmBDZjbnYfejASyirLUHMr43NAakpX3omsVPuU7Fn97Eaj3ZcvEZWXXgYP12gCnYfzUe2b08obNw0/s1600/telnet-rfc.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJAUnfetRYtOoYi0fWMR1zfoayE9CbVWDd2oMosPhTBz7Og2D7_veJgfcJlCdI1uHmBDZjbnYfejASyirLUHMr43NAakpX3omsVPuU7Fn97Eaj3ZcvEZWXXgYP12gCnYfzUe2b08obNw0/s1600/telnet-rfc.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>RFC</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPlpxQYm8s7mBg3UlQZNxG9XceNqB7TEEWcuiy_IDVK16502lJzseIdDaO8bJ9_Fkn-H_iCiJev53K3XBjcUmU1HrHvB5OmLGs229YMsrcDG7sPfdko-q-9GIWFwl1rQNntQl8LEsufQg/s1600/telnet-codes.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPlpxQYm8s7mBg3UlQZNxG9XceNqB7TEEWcuiy_IDVK16502lJzseIdDaO8bJ9_Fkn-H_iCiJev53K3XBjcUmU1HrHvB5OmLGs229YMsrcDG7sPfdko-q-9GIWFwl1rQNntQl8LEsufQg/s1600/telnet-codes.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>CODES</b></span></td></tr>
</tbody></table>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Just for the fun of it, I installed a telnet server on a CentOS 7 VM image and had the Telnet Client connects to it.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Guess what? It still works (phew :).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEispCjaiBpEDUwJEYaF1BrWEQU-hTDTZaRCXZvdwIepGrY3L8B8I50lM84B5lq4lEn8yGKPqXwmqKjQTrvJwRNzhrtRN4onQXFYMTYXojrJ5GZxWOs1KFNJ8gU8HcO96vgaYEojE9utvRk/s1600/telnet-conn.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEispCjaiBpEDUwJEYaF1BrWEQU-hTDTZaRCXZvdwIepGrY3L8B8I50lM84B5lq4lEn8yGKPqXwmqKjQTrvJwRNzhrtRN4onQXFYMTYXojrJ5GZxWOs1KFNJ8gU8HcO96vgaYEojE9utvRk/s1600/telnet-conn.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>RFC</b></span></td></tr>
</tbody></table>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">If you want to have a good laugh on some codes written by a Java newbie back in Year 2003, you can download it <a href="https://drive.google.com/uc?export=download&id=0B6EFlN-9iDHtMHY2Y1A4SVplcEE">here</a>.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Happy hacking!</span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-12028096659029962422015-02-02T13:50:00.001+08:002015-04-12T16:48:07.990+08:00Apache Kafka: A simple producer<span style="font-family: Arial, Helvetica, sans-serif;">If you are into big data and analytic, you must have heard of <a href="http://kafka.apache.org/" target="_blank">Apache Kafka</a> lately.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">I have and set out to learn this cool technology.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Before proceeding further, let me share a little bit about Apache Kafka (excerpt taken from its official website):</span><br />
<table cellpadding="2" cellspacing="0" style="border: 1px solid #ddd;">
<tbody>
<tr>
<td><span style="font-family: Arial, Helvetica, sans-serif;">Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.
</span></td>
</tr>
</tbody></table>
<div>
<br /></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">In a summary, this is what I did:<br />(1) Installed <a href="http://hortonworks.com/hdp/downloads/" target="_blank">Hortonworks HDP 2.2</a> (with Ambari)</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(2) Installed Apache Kafka (and all required components) through <a href="http://docs.hortonworks.com/HDPDocuments/Ambari-1.7.0.0/Ambari_Install_v170/Ambari_Install_v170.pdf" target="_blank">Ambari</a></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(3) Configured 3 Kafka brokers.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(4) Downloaded a large sample data set (about 10GB).<br /><b>NOTE</b>: If you would like to download some sample data sets, you can refer <a href="http://aws.amazon.com/datasets/" target="_blank">here</a>.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(<string tring=""><string string=""><string string=""><string string="">5) Wrote a simple producer (see below).</string></string></string></string></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(6) Executed the producer to load data into Kafka.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">(7) Executed the console consumer to check on the data.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">Eg. kafka-console-consumer.sh --topic datatest --zookeeper hdp1:2181 --from-beginning</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<input class="spoilerbutton" onclick="this.value=this.value=='Show Codes'?'Hide Codes':'Show Codes';" type="button" value="Show Codes" />
<div class="spoiler">
<div>
<table cellpadding="2" cellspacing="0" style="border: 1px solid #ddd;">
<tbody>
<tr>
<td><pre class="brush: csharp">import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Properties;
import java.io.*;
import kafka.producer.*;
import kafka.javaapi.producer.*;
public class KafkaProducer {
private final static String TOPIC = "datatest";
private final static String DELIMITER = "~~";
public static void main(String[] argv)
{
Properties properties = new Properties();
properties.put("metadata.broker.list","hdp1:6667,hdp2:6667,hdp3:6667");
properties.put("serializer.class","kafka.serializer.StringEncoder");
ProducerConfig producerConfig = new ProducerConfig(properties);
kafka.javaapi.producer.Producer<String, String> producer = new kafka.javaapi.producer.Producer<String, String>(producerConfig);
KeyedMessage<String, String> message = null;
try
{
FileReader fr = new FileReader(new File("/opt/data/movies.txt"));
BufferedReader br = new BufferedReader(fr);
String s = null;
String msg = null;
long ctr = 0;
while ((s = br.readLine()) != null)
{
if (s.length() != 0)
{
if (msg == null)
{
msg = s;
}
else
{
msg = msg + DELIMITER + s;
}
}
else
{
ctr++;
message = new KeyedMessage<String, String>(TOPIC, msg);
producer.send(message);
msg = null;
message = null;
}
}
}
catch (IOException fie)
{
System.out.println(fie);
}
producer.close();
}
}
</pre>
</td></tr>
</tbody></table>
</div>
</div>
Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com0tag:blogger.com,1999:blog-4943934699922259568.post-26824865940847398352015-01-31T17:56:00.000+08:002015-04-12T16:48:21.375+08:00VIMScript: Highlight Multiple Regex<span style="font-family: Arial, Helvetica, sans-serif;">My job requires me to read some very large and complex log files for troubleshooting.<br /><br />A few months down the road, I got so sicked that I started to find a way to ease my job. Since I was (and am) using vim and gvim to open those files, it was natural for me to explore whether VIMScript can do some wonders.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Google led me to <a href="http://vim.wikia.com/wiki/Highlight_multiple_words" target="_blank">this awesome VIMScript</a> maintained by John Beckett. However, based on my brief and limited trials (hence I could be wrong :), the script lacks a few things I need:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />(1) the script provides functionality to highlight words dynamically, but I need something that can remember my settings (keywords, highlight style, etc.) since I will be re-using the configuration a lot to read files of the same format and with same keywords.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(2) I need a script that would allow me to update or switch configuration on the fly without needing to re-open the input file. Just imagine that you suddenly want a different highlight style for some keywords or you have multiple different configurations for a single file type that you can switch to and fro dynamically.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Hence, despite not being a VIMScript expert, I started to explore how to re-use some of the codes from the original script (after getting permission from John to reuse and share) to do what I need.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The result is this:</span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVJQldHypKPrPsO6g0mqrFYUmO1lsWavBuKxp_BNWXR95sejXDtUNjyoCrwE19B-e9d_NXYEwboTho-YPmtm2Gg9pQ4TTE59Qhl-5FdmmEnQ12xMKUhIiHMirf3m4P11zEthHrxxEDwdc/s1600/vim1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVJQldHypKPrPsO6g0mqrFYUmO1lsWavBuKxp_BNWXR95sejXDtUNjyoCrwE19B-e9d_NXYEwboTho-YPmtm2Gg9pQ4TTE59Qhl-5FdmmEnQ12xMKUhIiHMirf3m4P11zEthHrxxEDwdc/s1600/vim1.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>FROM</b></span></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: center; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOmmzPs4CYu_lDUIzwzwOq0Vi89b0OowaQaJ7nkm0dqefXcwQy9GFfQRQQKtiRHnT57y7t3oyCRm2L3SXobisgawCZqyn4SZZmhl8Sy-EwFKo8UrxVGOlGGfll9JzemNleGJRTH5XmxZ0/s1600/vim2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOmmzPs4CYu_lDUIzwzwOq0Vi89b0OowaQaJ7nkm0dqefXcwQy9GFfQRQQKtiRHnT57y7t3oyCRm2L3SXobisgawCZqyn4SZZmhl8Sy-EwFKo8UrxVGOlGGfll9JzemNleGJRTH5XmxZ0/s1600/vim2.png" width="100%" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>TO</b></span></td></tr>
</tbody></table>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">If this interests you enough to try it out, you can follow the instructions below:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">For <b>WINDOWS</b>:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(1) Install GVIM (version 7.2 and above) from http://www.vim.org/download.php</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(2) Once GVIM is installed, make the directory C:\Users\<USERNAME>\vimfiles\plugin</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(3) Download the following archive <a href="https://drive.google.com/uc?export=download&id=0B6EFlN-9iDHtRXJtNm9qTjBRRVk">file</a> into the newly created directory and extract the content (only 2 files) there.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(4) File "highlight.csv" is the configuration file (details on how to configure can be found in the file itself).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(5) Configure as needed.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(6) Open a text file and press "\m".</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(7) Enjoy the color coded text :)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;">For <b>LINUX</b>:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(1) You have a choice of using terminal or GVIM (install vim-x11). If you choose to use terminal, you are limited to a small amount of colors (see below) =></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>NOTE</b>: If you decide to use terminal, please configure using the ctermfg and ctermbg columns in the "highlight.csv" file.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> NR-16 NR-8 COLOR NAME ~</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 0<span class="Apple-tab-span" style="white-space: pre;"> </span> 0<span class="Apple-tab-span" style="white-space: pre;"> </span> Black</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 1<span class="Apple-tab-span" style="white-space: pre;"> </span> 4<span class="Apple-tab-span" style="white-space: pre;"> </span> DarkBlue</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 2<span class="Apple-tab-span" style="white-space: pre;"> </span> 2<span class="Apple-tab-span" style="white-space: pre;"> </span> DarkGreen</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 3<span class="Apple-tab-span" style="white-space: pre;"> </span> 6<span class="Apple-tab-span" style="white-space: pre;"> </span> DarkCyan</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 4<span class="Apple-tab-span" style="white-space: pre;"> </span> 1<span class="Apple-tab-span" style="white-space: pre;"> </span> DarkRed</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 5<span class="Apple-tab-span" style="white-space: pre;"> </span> 5<span class="Apple-tab-span" style="white-space: pre;"> </span> DarkMagenta</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 6<span class="Apple-tab-span" style="white-space: pre;"> </span> 3<span class="Apple-tab-span" style="white-space: pre;"> </span> Brown, DarkYellow</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 7<span class="Apple-tab-span" style="white-space: pre;"> </span> 7<span class="Apple-tab-span" style="white-space: pre;"> </span> LightGray, LightGrey, Gray, Grey</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 8<span class="Apple-tab-span" style="white-space: pre;"> </span> 0*<span class="Apple-tab-span" style="white-space: pre;"> </span> DarkGray, DarkGrey</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 9<span class="Apple-tab-span" style="white-space: pre;"> </span> 4*<span class="Apple-tab-span" style="white-space: pre;"> </span> Blue, LightBlue</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 10<span class="Apple-tab-span" style="white-space: pre;"> </span> 2*<span class="Apple-tab-span" style="white-space: pre;"> </span> Green, LightGreen</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 11<span class="Apple-tab-span" style="white-space: pre;"> </span> 6*<span class="Apple-tab-span" style="white-space: pre;"> </span> Cyan, LightCyan</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 12<span class="Apple-tab-span" style="white-space: pre;"> </span> 1*<span class="Apple-tab-span" style="white-space: pre;"> </span> Red, LightRed</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 13<span class="Apple-tab-span" style="white-space: pre;"> </span> 5*<span class="Apple-tab-span" style="white-space: pre;"> </span> Magenta, LightMagenta</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 14<span class="Apple-tab-span" style="white-space: pre;"> </span> 3*<span class="Apple-tab-span" style="white-space: pre;"> </span> Yellow, LightYellow</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span> 15<span class="Apple-tab-span" style="white-space: pre;"> </span> 7*<span class="Apple-tab-span" style="white-space: pre;"> </span> White</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;">(2) Make a directory <user home directory>/.vim/plugin.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(3) Download the following archive <a href="https://drive.google.com/uc?export=download&id=0B6EFlN-9iDHtRXJtNm9qTjBRRVk">file</a> into the newly created directory and extract the content (only 2 files) there.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(4) File "highlight.csv" is the configuration file (details on how to configure can be found in the file itself).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(5) Configure as needed.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(6) Open a text file and press "\m".</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">(7) Enjoy the color coded text :)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>NOTE</b>: "vi" is not "vim". If you have installed vim, you can create an alias in <user home directory>/.bashrc file to map "vi" to "vim". Remember to source the .bashrc file after amending it.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;">Eg. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">alias vi="vim"</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">alias view="vim -R"</span><br />
<div>
<br /></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>Daniel Yeaphttp://www.blogger.com/profile/16883267638415675465noreply@blogger.com2