Perfect End-to-End Data Platform for Today (IMHO)

Hi Guys,
I am back (like Superman :)!!

I had been bogged down with work, family and health issues for past 1+ year. But then, I decided that my passion for technology is too great for me to stop sharing :)

So, here we go again...this time, get ready for more interesting stuff that I learned in the past year.

***

Now, let me start by sharing a data platform diagram that I have based my past year's learning on...

** At the DATA STORAGE layer, I am still contemplating which NoSQL database to select.


Before you get too excited or disappointed, let me just say that this is a platform that "I" think is perfect (and I am still constantly re-adjusting by adding/removing technologies) for today.

Each and individual selected technology is the cream of the crop. I do not declare that they are perfect, but then they are good enough to perform the work consistently well.

Everyone of them plays a specific role from the source to the destination (reporting).

I am still learning most (if not all) of them, so stay tuned for more exciting sharing!

Apache Storm: Integration with Kafka using Kafka Spout

If you have been playing with either Apache Kafka or Apache Storm, you would have read so much articles about integration between the two. From my experience, reading too much can be a bad thing sometimes (pun intended :). In this case, there were multiple efforts that try to offer such integration. Thus, it might caused confusion about which is the best or standard way to do it.

It is good to know that starting from version 0.9.2-incubating, Apache Storm has decided to include such support officially. Read more here.

Anyway, how does such integration work?

In this blog entry, I am only going to share information about using Kafka as a Storm spout. Yes, starting from Storm version 0.9.3, you can use Kafka as a bolt too. If you want to know more about Topology, Spout and Bolt, read this.

Basically, the classes you need for the Storm-Kafka integration are available under storm.kafka.* package. 

If you want to get up to speed quick, try out the sandbox offered by Hortonworks here. After you have downloaded the sandbox (or if you are gutsy enough to install the system through Ambari), it is advisable to try out the tutorial too. If you want to jump straight to the tutorial related to the Storm-Kafka integration, you can go here. Please take note that the tutorial contains the source codes too, so make sure you check them out!

Once you get a hang of it, you can move over to this website to learn more about Storm Kafka.

If you do not want to compile the Storm Kafka package yourself, you can download it from the Hortonworks maven repository.

The information offered here should get you going for a while, and I will share some tips and traps regarding the integration in future entries.

Happy hacking!

Apache Kafka: A simple producer

If you are into big data and analytic, you must have heard of Apache Kafka lately.

I have and set out to learn this cool technology.

Before proceeding further, let me share a little bit about Apache Kafka (excerpt taken from its official website):

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

In a summary, this is what I did:
(1) Installed Hortonworks HDP 2.2 (with Ambari)

(2) Installed Apache Kafka (and all required components) through Ambari

(3) Configured 3 Kafka brokers.

(4) Downloaded a large sample data set (about 10GB).
NOTE: If you would like to download some sample data sets, you can refer here.

(5) Wrote a simple producer (see below).

(6) Executed the producer to load data into Kafka.

(7) Executed the console consumer to check on the data.
Eg. kafka-console-consumer.sh --topic datatest --zookeeper hdp1:2181 --from-beginning

import java.text.SimpleDateFormat; import java.util.Date; import java.util.Properties; import java.io.*; import kafka.producer.*; import kafka.javaapi.producer.*; public class KafkaProducer { private final static String TOPIC = "datatest"; private final static String DELIMITER = "~~"; public static void main(String[] argv) { Properties properties = new Properties(); properties.put("metadata.broker.list","hdp1:6667,hdp2:6667,hdp3:6667"); properties.put("serializer.class","kafka.serializer.StringEncoder"); ProducerConfig producerConfig = new ProducerConfig(properties); kafka.javaapi.producer.Producer<String, String> producer = new kafka.javaapi.producer.Producer<String, String>(producerConfig); KeyedMessage<String, String> message = null; try { FileReader fr = new FileReader(new File("/opt/data/movies.txt")); BufferedReader br = new BufferedReader(fr); String s = null; String msg = null; long ctr = 0; while ((s = br.readLine()) != null) { if (s.length() != 0) { if (msg == null) { msg = s; } else { msg = msg + DELIMITER + s; } } else { ctr++; message = new KeyedMessage<String, String>(TOPIC, msg); producer.send(message); msg = null; message = null; } } } catch (IOException fie) { System.out.println(fie); } producer.close(); } }