Perfect End-to-End Data Platform for Today (IMHO)

Hi Guys,
I am back (like Superman :)!!

I had been bogged down with work, family and health issues for past 1+ year. But then, I decided that my passion for technology is too great for me to stop sharing :)

So, here we go again...this time, get ready for more interesting stuff that I learned in the past year.

***

Now, let me start by sharing a data platform diagram that I have based my past year's learning on...

** At the DATA STORAGE layer, I am still contemplating which NoSQL database to select.


Before you get too excited or disappointed, let me just say that this is a platform that "I" think is perfect (and I am still constantly re-adjusting by adding/removing technologies) for today.

Each and individual selected technology is the cream of the crop. I do not declare that they are perfect, but then they are good enough to perform the work consistently well.

Everyone of them plays a specific role from the source to the destination (reporting).

I am still learning most (if not all) of them, so stay tuned for more exciting sharing!

Running Apache Spark on Mesos

Running Apache Spark on Mesos is easier than I thought!

I have 3 nodes at home (hdp1, hdp2 and hdp3) which are running Hortonworks Data Platform (HDP). I have HDFS, Zookeeper and Spark installed on the cluster.

To test running Spark on Mesos, I decided to reuse the cluster for simplicity's sake.

All I did was:
(1) install Mesos master on node hdp1.
(2) install Mesos slave on node hdp2 and hdp3.
(3) configure the master and slaves accordingly.

NOTE:
(i) For more information about the installation and configuration of Mesos, you can refer to this blog entry.
(ii) I reuse Zookeeper that was installed with HDP.

Once Mesos is up and running, I decided to carry out a very simple test that is described as follow:
(1) Use "spark-shell" as the Spark client to connect to the Mesos cluster.
(2) Load a sample text file into HDFS.
(3) Use spark-shell to perform a line count on the loaded text file.

First, let's see what is the command that you can use to connect spark-shell to a running Mesos cluster.

TIPS: All you need to do is to use the "--master" option to point to the Mesos cluster at "mesos://<IP-or-hostname of Mesos master>:5050".

Then, let's load a sample text file into HDSF using the "hadoop fs -put" command.

Once the sample text file is loaded, let's create a spark RDD using it.

TIPS: You can use the "sc.textFile" function and points it to "hdfs://<namenode>:8020/<path to file>".

After the RDD is created, let's run a count on it using the "count()" function.

You can see from the screenshot above and below that, Spark (through Mesos) has submitted 2 tasks that are executed on node hdp2 and hdp3 respectively.

You can also see from the screenshot above that the end result is returned as "4" (which means 4 lines in the file - which is correct!).

So, that is how easy it is to run Spark on Mesos.

Hope that you are going to try it out!