Anything Linux, Big Data, Containerization, Graph Database and Cloud

Tuesday, March 10, 2015

Apache Storm: How to add external JARs or packages into CLASSPATH while running 'storm jar'?

I have been playing with 'storm-kafka' and 'storm-hbase' lately. Basically, they are projects/tools that one can use to integrate Kafka and HBase with storm.

My project was to have Kafka as spout and HBase as bolt. In other words, my application will pull data from Kafka and then write the output to HBase. In other other words (pun intended :), I need to have the storm-kafka and storm-hbase JAR included when I run my Storm topology.

There are a few ways to do this:
(1) Put the JARs under STORM_BASE_DIR
(2) Put the JARs under STORM_BASE_DIR/lib
(3) Put the package under STORM_CONF_DIR
(4) Include the package into the topology JAR

After trying the above few methods, my favourite is method #4. However, it is not without its own pain points.

Let me explain why I do not like the other methods.

Method #1
=======
By putting JARs into STORM_BASE_DIR, I have a feeling that I have 'corrupted' the directory. Messing up a standard directory of a product is not my cup of tea.

Method #2
=======
See Method #1 above.

Method #3
=======
Since the storm.py codes do not search the directory declared as STORM_CONF_DIR (or USER_CONF_DIR) for JARs, you would have to put the package files in that directory. How many times have I said 'messy'? :)

Now, let's discuss Method #4. I say it is my favourite, but I never say it is the best. That is because it will grow your JAR file size greatly if you have some really big external JARs to include (beside your topology). However, I feel that it is the most acceptable approach because it is more manageable than the other methods (at least to me :).

Hence, if you are looking into including or adding external JARs while running your Storm topology, I would suggest you to include those JARs into your topology JAR for the time being until there is a neater way to do this!

NOTE:
Environment = HDP 2.2 (Storm 0.9.3)

Source codes of storm.py and how CLASSPATH is determined

Tuesday, March 3, 2015

Apache Mesos: Mesos + Marathon + Docker = ?

In my previous post, I talked about merging resources from multiple nodes into one using Apache Mesos.

I also talked about the 2 reasons I decided to pick up Mesos:
(1) Google Kubernetes
(2) Docker

In this post, I am going to share the method you can use to deploy Docker container on a Mesos cluster.

* Mesos and Zookeeper have to be up and running. For installation instruction, please refer to my previous post.
** I used CentOS 6.6 as the platform. So, some commands might differ on other platforms.
*** Make sure "docker-io" "(the Docker package) is installed and the daemon is running on all Mesos slave nodes.

(1) Check and update (if needed) the /etc/mesos/zk file on the node you wish to install Marathon.

vi /etc/mesos/zk

** Make sure the IP address of the Zookeeper server is written there

(2) Install Marathon (using the Mesosphere repository created in the previous post) on the node selected above (Master node is recommended for testing purpose and ease of maintenance):

yum install marathon

(3) Make sure Marathon is up and running after the installation. Otherwise, start it using:

initctl start marathon

** Use "ps -ef" command to verify the Marathon process is running with the proper IP addresses (instead of 'localhost') of the zookeeper-server and Master node. If it is not, check the /etc/mesos/zk file again and restart.

(4) Update all the Mesos slave nodes with the following:

echo 'docker,mesos' > /etc/mesos-slave/containerizers

echo '5 mins' > /etc/mesos-slave/executor_registration_timeout

(5) Restart all the Mesos slave nodes:

initctl restart mesos-slave

(6) By now, you should be ready to deploy Docker container on the Mesos cluster. To do so, you have to create a JSON file for the Docker container you wish to deploy:

Eg.

{
"container": {
"type": "DOCKER",
"docker": {
"image": "192.168.0.210:5000/centos63:httpd",
"network": "HOST"
}
},
"id": "centos63",
"instances": 1,
"cpus": 4,
"mem": 2048,
"uris": [],
"cmd": "/usr/sbin/httpd -DFOREGROUND"

}

There are a few things to take note:

(a) For the "image" parameter, you would need to specify an image that is reachable by all of the Mesos slaves, because you would not know for sure which slave or slaves the Master will select to run the container.

** If you are using your own insecured private registry, please make sure you edit the docker "default" file (eg. /etc/sysconfig/docker or /etc/default/docker) to declare the registry as insecured and restart the Docker service (service docker restart):

other_args="--insecure-registry 192.168.0.210:5000"

(b) AFAIK, Mesos (0.21.1) only support 2 network modes now - HOST or BRIDGE.

(7) Once the JSON file is ready, you can submit to Marathon using the POST method:

curl -X POST -H "Content-Type: application/json" http://<marathon host>:8080/v2/apps -d@<JSON filename>

Marathon GUI: After the CURL command and when Mesos is deploying the container

Marathon GUI: The container is successfully deployed and RUNNING

Mesos GUI: Shows one active task running on "mesos4" slave node

Mesos GUI: Clicking on the task shows the details (it's a SANDBOX)

Mesos GUI: STDOUT and STDERR are streamed from the container to the sandbox

On 'mesos4' node, the image is downloaded from the private repo and a container is running

On 'mesos4' node, 'docker inspect <container id>' shows the networking mode is HOST as configured

On 'mesos3' node, a HTTP connection shows HTTPD container is indeed running on 'mesos4' node

Friday, February 27, 2015

Apache Mesos: When All Becomes One

First of all, for the benefits of those unfamiliar with Apache Mesos, this is the "what is" taken from its official website:

What is Mesos?

A distributed systems kernel

Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments.

Beside the fact that I have always wanted to learn a technology that can merge all the resources available on my multiple machines into one, I am out to learn Apache Mesos for the following 2 reasons (currently):
(1) Google Kubernetes
(2) Docker

Installing Apache Mesos isn't too hard if you follow the instruction available on its official website, but I would like to share an alternative installation method which I found is more straightforward:

* Instructions only suitable for RHEL/CentOS 6 (tested on CentOS 6.6). For other platforms, refer here.
** Run all instructions as 'root' user for simplicity.

(1) On the node or VM image that you would like to designate as the Master and all slave nodes, execute the following command to create the Mesosphere repository:

rpm -Uvh http://repos.mesosphere.io/el/6/noarch/RPMS/mesosphere-el-repo-6-2.noarch.rpm

(2) On the Master and all the slave nodes, install Mesos:

yum -y install mesos

(3) Even though you only plan to have a single Master node, it is advisable to install Zookeeper (just in case you want to expand in the future):

rpm -Uvh http://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm

yum -y install zookeeper-server

* You can install the Zookeeper server on either the Master node (preferred for ease of maintenance) or any of the slave node.
** You need to have Java installed for Zookeeper to work properly.

(4) On the Master node, initialize Zookeeper:

service zookeeper-server init

echo 1 | sudo tee -a /var/lib/zookeeper/myid >/dev/null

(5) On the Master node, stop and disable mesos-slave:

initctl stop mesos-slave

cd /etc/init/

mv mesos-slave.conf mesos-slave.disable

(6) On all the slave nodes, stop and disable mesos-master:

initctl stop mesos-master

cd /etc/init/

mv mesos-master.conf mesos-master.disable

(7) On the Master node, set the IP address:

echo <IP of the Master node> | sudo tee /etc/mesos-master/ip

(8) On the Master node, set the name of the cluster:

echo <cluster name> | sudo tee /etc/mesos-master/cluster

(9) On the Master and all slave nodes, set the URL of the Zookeeper server:

echo zk://<IP of the Zookeeper server>:2181/mesos | sudo tee /etc/mesos/zk

(10) On all the slave nodes, set their respective IP address:

echo <IP of the Slave node> | sudo tee /etc/mesos-slave/ip

(11) On the Master node, restart mesos-master and Zookeeper (if it is installed there):

service zookeeper restart

initctl restart mesos-master

(12) On all the slave nodes, restart mesos-slave:

initctl restart mesos-slave

(12) Verify that the Master is running and all slaves are registered with it:

http://<IP of the Master node>:5050

When the Master is first initialized

When the first slave joined

When the second slave joined

When the third slave joined

All the slaves

Monday, February 16, 2015

Docker: How to Create Your Own Base Image (CentOS/RHEL)

Normally, we will pull base images from the Docker Hub registry to build our own images.

However, there might be times when you want more control over the base image (size, packages, etc.). Luckily, Docker provides a way to do that.

For general information about building your own image, refer here. Since I am a fan of CentOS/RHEL, I would normally go here. For a more "friendly" version of the script, go here.

To create your own CentOS/RHEL image, follow these instructions:
(1) Copy or download the script to a running CentOS/RHEL system with yum properly setup.
NOTE: I tested the script on CentOS 6.5.

#!/usr/bin/env bash
#
# Create a base CentOS Docker image.
#
# This script is useful on systems with yum installed (e.g., building
# a CentOS image on CentOS).  See contrib/mkimage-rinse.sh for a way
# to build CentOS images on other systems.

usage() {
    cat <<EOOPTS
OPTIONS:
  -y <yumconf>  The path to the yum config to install packages from. The
                default is /etc/yum.conf.
EOOPTS
    exit 1
}

# option defaults
yum_config=/etc/yum.conf
while getopts ":y:h" opt; do
    case $opt in
        y)
            yum_config=$OPTARG
            ;;
        h)
            usage
            ;;
        \?)
            echo "Invalid option: -$OPTARG"
            usage
            ;;
    esac
done
shift $((OPTIND - 1))
name=$1

if [[ -z $name ]]; then
    usage
fi

#--------------------

target=$(mktemp -d --tmpdir $(basename $0).XXXXXX)

set -x

mkdir -m 755 "$target"/dev
mknod -m 600 "$target"/dev/console c 5 1
mknod -m 600 "$target"/dev/initctl p
mknod -m 666 "$target"/dev/full c 1 7
mknod -m 666 "$target"/dev/null c 1 3
mknod -m 666 "$target"/dev/ptmx c 5 2
mknod -m 666 "$target"/dev/random c 1 8
mknod -m 666 "$target"/dev/tty c 5 0
mknod -m 666 "$target"/dev/tty0 c 4 0
mknod -m 666 "$target"/dev/urandom c 1 9
mknod -m 666 "$target"/dev/zero c 1 5

yum -c "$yum_config" --installroot="$target" --releasever=/ --setopt=tsflags=nodocs \
    --setopt=group_package_types=mandatory -y groupinstall Core
yum -c "$yum_config" --installroot="$target" -y clean all

cat > "$target"/etc/sysconfig/network <<EOF
NETWORKING=yes
HOSTNAME=localhost.localdomain
EOF

# effectively: febootstrap-minimize --keep-zoneinfo --keep-rpmdb
# --keep-services "$target".  Stolen from mkimage-rinse.sh
#  locales
rm -rf "$target"/usr/{{lib,share}/locale,{lib,lib64}/gconv,bin/localedef,sbin/build-locale-archive}
#  docs
rm -rf "$target"/usr/share/{man,doc,info,gnome/help}
#  cracklib
rm -rf "$target"/usr/share/cracklib
#  i18n
rm -rf "$target"/usr/share/i18n
#  sln
rm -rf "$target"/sbin/sln
#  ldconfig
rm -rf "$target"/etc/ld.so.cache
rm -rf "$target"/var/cache/ldconfig/*

version=
if [ -r "$target"/etc/redhat-release ]; then
    version="$(sed 's/^[^0-9\]*\([0-9.]\+\).*$/\1/' "$target"/etc/redhat-release)"
fi

if [ -z "$version" ]; then
    echo >&2 "warning: cannot autodetect OS version, using '$name' as tag"
    version=$name
fi


tar --numeric-owner -c -C "$target" . | docker import - $name:$version
docker run -i -t $name:$version echo success

rm -rf "$target"

(2) Save the script and give it execute permission (chmod 700 <script>).

(3) Login as 'root'.

(4) Amend the script as necessary to suit your purpose (eg. more packages, etc).

(5) Execute the script (eg. createDockerImage.sh centos).
WHERE "centos" is the name that you would like the resultant image to have
NOTE: The script will tag the image using the version of the OS (or the "name" provided if the version cannot be determined).

(6) Once the script returns, run "docker images" to confirm the image is created.

(7) Verify that the image is ok by launching a container using "docker run -t -i <image> /bin/bash".

If everything works out fine, you would now have your custom built base image!

Wednesday, February 11, 2015

Apache Storm: Integration with Kafka using Kafka Spout

If you have been playing with either Apache Kafka or Apache Storm, you would have read so much articles about integration between the two. From my experience, reading too much can be a bad thing sometimes (pun intended :). In this case, there were multiple efforts that try to offer such integration. Thus, it might caused confusion about which is the best or standard way to do it.

It is good to know that starting from version 0.9.2-incubating, Apache Storm has decided to include such support officially. Read more here.

Anyway, how does such integration work?

In this blog entry, I am only going to share information about using Kafka as a Storm spout. Yes, starting from Storm version 0.9.3, you can use Kafka as a bolt too. If you want to know more about Topology, Spout and Bolt, read this.

Basically, the classes you need for the Storm-Kafka integration are available under storm.kafka.* package.

If you want to get up to speed quick, try out the sandbox offered by Hortonworks here. After you have downloaded the sandbox (or if you are gutsy enough to install the system through Ambari), it is advisable to try out the tutorial too. If you want to jump straight to the tutorial related to the Storm-Kafka integration, you can go here. Please take note that the tutorial contains the source codes too, so make sure you check them out!

Once you get a hang of it, you can move over to this website to learn more about Storm Kafka.

If you do not want to compile the Storm Kafka package yourself, you can download it from the Hortonworks maven repository.

The information offered here should get you going for a while, and I will share some tips and traps regarding the integration in future entries.

Happy hacking!

Monday, February 9, 2015

Docker: Why delete does not reduce the image size?

If you wonder why your Docker image does not shrink in size after you have deleted some very large files, then you might want to read further for a brief explanation and solution!

In short, the cause of all these drama is the union filesystem (read here). If you work on an image and have performed multiple commits (either through DockerFile or manually), then you might run into this problem later in the stage. The large files might have been committed in one of the layers during the earlier stage.

Eg.

(1) We have this image called "centos63:omnibus731fp8-hm2000" which is about 2.11GB in size.

(2) Next, let us run the image and delete a very large directory in there.

(3) To persist the change, I committed the image and tagged the new image as "centos63:omnibus731fp8-hm2000-rm".

(4) Surprise, surprise! The size is still 2.11GB!?!?

(5) What happened? Let's check it out. Is the large directory still there? Nope!!!

(6) Then why didn't the "rm" work?

You can see that the large directory was most probably (sorry I can't tell for sure because the image was created with a previous very bad habit of manual commits :) added in image "28729cfb27de" that was created very early in the stage. Ever since, there were multiple commits (represented by those multiple different images in the history).

Another thing to pay attention to is the "rm" command that was persisted in image "ae11f957b955" (latest - highlighted in BLUE box) and the size was only 7 bytes!!! It shows clearly that Docker has only recorded the "rm" command and not the result of it.

That is the "power" of an union filesystem. But sorry, your large directory was still technically in there.

(7) So, what now? How do you trim those extra fat off? The answer is with this awesome tool called docker-squash!

I executed the tool with an option to name the output image "centos63:omnibus731fp8-hm2000-squashed".
You can see how the tool successfully determined all the layers belonging to the image (check the UUID).

(8) From the output of the tool, it seems that it has successfully trimmed down the image.

(9) Let's verify whether the new image "omnibus731fp8-hm2000-squashed" is significantly smaller in size.

It seems that the tool does deliver after all! :)

I have read that Docker Inc is working on having this feature (shrinking the image size) built-in. So, while waiting for it, you have this tool!

Thursday, February 5, 2015

Java: Telnet Client

I was going through my personal projects in search for a topic to write for this blog when I stumbled upon this very old codes.

I wrote this back in Year 2003 and that was 12 years ago!

Back then, I was learning Java and somehow got hooked on reading RFCs. So, I guess it was natural for me to write a Telnet Client based on Telnet RFCs. I have to admit upfront that I did not implement all of them, but basic things do work :)

RFC

CODES

Just for the fun of it, I installed a telnet server on a CentOS 7 VM image and had the Telnet Client connects to it.

Guess what? It still works (phew :).

RFC

If you want to have a good laugh on some codes written by a Java newbie back in Year 2003, you can download it here.

Happy hacking!