Apache Storm: HBase Bolt

Starting with version 0.9.2-incubating, Storm included support for HBase as bolt. You no longer need to seek third party packages to do that, which is a great news!

To refer to the APIs and documentation, you can go here.

I have coded a simple Storm application that takes Kafka as spout and HBase as bolt. In other words, the application will get its data from Kafka and then write to HBase.

One interesting item within Apache Storm is stream grouping (how tuples produced by spouts will be handled by available bolts). To read more about stream grouping, you can refer here.

Lastly, if you want to try it out yourself, you can download the sample codes here.

Storm UI that shows Kafka spout and HBase bolt processing

Apache Storm: How to add external JARs or packages into CLASSPATH while running 'storm jar'?

I have been playing with 'storm-kafka' and 'storm-hbase' lately. Basically, they are projects/tools that one can use to integrate Kafka and HBase with storm.

My project was to have Kafka as spout and HBase as bolt. In other words, my application will pull data from Kafka and then write the output to HBase. In other other words (pun intended :), I need to have the storm-kafka and storm-hbase JAR included when I run my Storm topology.

There are a few ways to do this:
(1) Put the JARs under STORM_BASE_DIR
(2) Put the JARs under STORM_BASE_DIR/lib
(3) Put the package under STORM_CONF_DIR
(4) Include the package into the topology JAR

After trying the above few methods, my favourite is method #4. However, it is not without its own pain points.

Let me explain why I do not like the other methods.

Method #1
=======
By putting JARs into STORM_BASE_DIR, I have a feeling that I have 'corrupted' the directory. Messing up a standard directory of a product is not my cup of tea.

Method #2
=======
See Method #1 above.

Method #3
=======
Since the storm.py codes do not search the directory declared as STORM_CONF_DIR (or USER_CONF_DIR) for JARs, you would have to put the package files in that directory. How many times have I said 'messy'? :)

Now, let's discuss Method #4. I say it is my favourite, but I never say it is the best. That is because it will grow your JAR file size greatly if you have some really big external JARs to include (beside your topology). However, I feel that it is the most acceptable approach because it is more manageable than the other methods (at least to me :).

Hence, if you are looking into including or adding external JARs while running your Storm topology, I would suggest you to include those JARs into your topology JAR for the time being until there is a neater way to do this!

NOTE:
Environment = HDP 2.2 (Storm 0.9.3)

Source codes of storm.py and how CLASSPATH is determined