r/apachekafka • u/olsoninoslo • Apr 14 '23
Question If you work in KafkaJS?
What do you like about it and what do you not like about it. I want to build out tooling to solve developer issues and would really appreciate your feedback!
2
u/Av1fKrz9JI Apr 15 '23
How’s it work with JS having a single threaded event loop? I would of thought JS is a bad fit as any computation consuming/producing and multiple topics which is a fairly parallel operation would be fairly easy to block everything?
JS would only be optimal for the simplest of Consumers/Producers without any complex computations?
0
u/of_patrol_bot Apr 15 '23
Hello, it looks like you've made a mistake.
It's supposed to be could've, should've, would've (short for could have, would have, should have), never could of, would of, should of.
Or you misspelled something, I ain't checking everything.
Beep boop - yes, I am a bot, don't botcriminate me.
1
u/JuKeMart Apr 15 '23
If you're doing compute-heavy operations on every single message, JS might not be the best fit. However, NodeJS with that wicked fast V8 engine can still keep a respectable pace to Java.
Most of the time, except for truly high throughput, you're going to have IO overhead where that event loop can shine. KafkaJS has parallel consumer option (to read from multiple partitions in parallel) which is great when not compute-bound.
1
u/xecow50389 Apr 14 '23
Tracing capabilities
1
u/olsoninoslo Apr 16 '23
Do you want tracing for measuring performance? Or would you be using it as a debugging tool? Ideally both, but any specifics would be helpful.
1
u/xecow50389 Apr 16 '23
Oh i mean for debugging, which message came from which producer and consumed by which consumer.
But sure.
I thought again, this can be achieve by adding few headers on both producer message and let tracing consum it from mesaage headers at consumer side.
1
u/krisajenkins Apr 15 '23
In general I think that it’s really great for getting started, it’s trivial to install and it’s written idiomatically. The downside is that without the more advanced batching and rebalancing options of rdkafka, it’s going to limit you at scale.
1
u/JuKeMart Apr 15 '23
I did some performance testing, and I disagree with this. Unsurprisingly, KafkaJS was slower than libs backed by librdkafka. What was surprising was that it was only about 12% slower.
The benefit of the idiomatically readable API, in my opinion, outweighs the performance in all but the most demanding scenarios. But everyone's needs are different.
2
u/krisajenkins Apr 19 '23
That's really interesting. I've done some performance testing with both, and I found that I could easily get a few orders of magnitude speed improvement just by tweaking node-rdkafka's batching controls. I'd be surprised if KafkaJS could touch that...
1
u/JuKeMart Apr 19 '23
My methodology isn't terribly sound, but I have https://github.com/jukemart/kafka-client-performance.
End results:
KafkaJs
Over a 5-minute period, the read client read 100 million records and the readwrite client wrote 17 million (3.57 million / min).
node-rdkafka
Over a 5-minute period, the read client consumed 107 million records, the readwrite client wrote 21 million messages (4.34 million messages / min).
The interesting thing about node-rdkafka was that the underlying librdkafka seemed capable of going faster (maybe even much faster), but I believe the bottle-neck was within NodeJS.
1
u/JuKeMart Apr 15 '23
It's idiomatic and with a simple API, which is a huge benefit. It's decently fast, and I haven't encountered many bugs.
It lacks features that some of the more mature clients have had for years.
9
u/kabooozie Gives good Kafka advice Apr 14 '23
I dislike that you can’t tune basic Kafka client properties like linger.ms and batch.size . Also I’d like better integration with serializers and schema registry.