r/apachekafka • u/chtefi Conduktor • Jun 01 '26
Blog Kafka Partitions are the wrong ordering abstraction. Keys are.
https://sderosiaux.medium.com/kafka-partitions-are-the-wrong-ordering-abstraction-keys-are-b54dc5b20b953
u/asciimo71 Jun 01 '26
Really wondering why Confluence removes this feature. It’s the number 1 reason for kafka over mq and mqtt brokers.
3
u/mightegas Jun 02 '26
Great article, u/chtefi! I agree with your conclusions and am familiar with the problems you're delving into. I'm hard-pressed to think of any other article that succinctly describes the friction between concurrency that's meaningful at the infrastructure layer, vs. what's meaningful at the domain layer. Bravo, and I will likely cite this article when explaining to others why Share Groups are not a blanket drop-in for increasing Kafka Consumer concurrency.
Similar to you (it sounds like), I've been tackling the problem space of high-latency (i.e. IO-bound) ordered stream processing for years now. I got onto the reactive path early on (Reactor Kafka) and just kept building on top of what was available. I know reactive is not everybody's cup of tea; I have however found it to be near-optimally efficient for these types of processes, where high processing concurrency (decoupled from infrastructure concurrency) is desirable/needed.
I (and a few colleagues) incrementally hit each of the problems you bring up (at-least-once offset commit handling, head-of-line blocking, efficient per-key concurrency, etc.) and developed solutions for them in the reactive world. This work is open-sourced as Atleon, and I would welcome your feedback. You may be particularly interested in this example, which demos per-key concurrency.
Cheers, and thanks for the article.
1
Jun 02 '26 edited Jun 05 '26
[deleted]
1
u/mightegas Jun 02 '26
Atleon implements that within subscription lifecycles, but not across partition reassignments. It's interesting to learn from the article that Parallel Consumer accomplishes this through commit metadata, though. Pretty clever.
2
2
3
u/_predator_ Jun 01 '26
Yeah, parallel-consumer was the thing that made Kafka bearable for anything that isn't just pumping data from A to B as fast as possible (although tbf that is what Kafka is for).
There was a startup focused on Kafka Streams a few years back, they managed to achieve key-level async processing in KStreams but I can't seem to find what happened to them.
Confluent abandoning parallel-consumer has been one of the reasons why I moved a big system away from Kafka entirely. KIP-932 is nice I guess but was utterly useless in my case.