Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Besides the operational concerns mentioned in the sibling comment, Kafka is simply not a great queue. You can't work-steal, you can't easily retry out-of-order, you can't size retention based on "is it processed yet", and you may need to manually implement DLQ behavior.

If you already have Kafka for other (more WAL-y, or maybe older log-shippy) reasons it can be an OK queue, especially if you've got a team that can use Kafka as a WAL they can easily work around using most of the downsides of using it as queue. But I wouldn't take it as a first choice.



Additionally, you can't easily increase/decrease consumer counts such that all consumers quickly get assigned roughly equivalent workloads.

It will be interesting to watch progress on KIP-932 as the Kafka community thinks about adding message queue behavior: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...


Great point. The basic semantics are very different too. In MQs you partition/namespace/channel (whatever you want to call it) based on how data flows in your application (e.g. fanout). In Kafka you're tied more to a persistance model so you end up with fat linear topics and the "filtering"/flow management happen on the consumer's side.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: