Enterprise Integration Zone is brought to you in partnership with:

Mitch Pronschinske is the Lead Research Analyst at DZone. Researching and compiling content for DZone's research guides is his primary job. He likes to make his own ringtones, watches cartoons/anime, enjoys card and board games, and plays the accordion. Mitch is a DZone Zone Leader and has posted 2576 posts at DZone. You can read more from them at their website. View Full User Profile

VMware's RabbitMQ vs. LinkedIn's Kafka

01.16.2013
| 14882 views |
  • submit to reddit

Although quantitative, data-driven comparisons like benchmarks are the most fundamental comparisons we have in technology, it can often be more helpful to get a great qualitative analysis in the form of user experiences.

That's the kind of comparison I recently found between the RabbitMQ and Kafka messaging queues.  IT consultant Stuart Charlton had a rich analysis on these two technologies from VMware (RabbitMQ) and LinkedIn (Kafka) in the Quora question "RabbitMQ vs Kafka: which one for durable messaging with good query features?".  I've picked out some of the best bits of insight from the post, but if you have the time you should read his answer in full, especially if you're currently using, or considering the use of Kafka or RabbitMQ.
____________________________________________________________________________

Use Kafka if you have a fire hose of events (100k+/sec) you need delivered in partitioned order 'at least once' with a mix of online and batch consumers, you want to be able to re-read messages, you can deal with current limitations around node-level HA (or can use trunk code), and/or you don't mind supporting incubator-level software yourself via forums/IRC. 

Use RabbitMQ if you have messages (20k+/sec) that need to be routed in complex ways to consumers, you want per-message delivery guarantees, you don't care about ordered delivery, you need HA at the cluster-node level now, and/or you need 24x7 paid support in addition to forums/IRC.

Both have similar distributed deployment goals but they differ on message model semantics.  

Neither offers great "filter/query" capabilities

-- From Stuart Charlton