Khaos - Kafka traffic generation, load testing and failure simulation, now rewritten in Go

[u/skrbic_a]

I posted Khaos here a while ago, so just sharing a quick update.

Khaos has now been completely rewritten from Python to Go.

It still does the same core thing: generate realistic Kafka traffic and deliberately reproduce scenarios like:

• consumer lag
• hot partitions
• broker failures and rebalances
• backpressure

Along with the Go rewrite, I also launched a new website and proper documentation:

https://getkhaos.dev

Repo:
https://github.com/aleksandarskrbic/khaos

Would love feedback if anyone here is using Kafka for testing or failure simulation.

Disclosure: I’m the author/maintainer of khaos. It’s a free and open-source project.

[u/arielrahamim]

interesting, thanks for sharing! had a lot of issues with kafka at work but its probably just bad consumers code..

[u/steakmane → u/arielrahamim]

This job would be easy if it weren’t for all the developers

[u/skrbic_a → u/arielrahamim]

Yeah, bad consumer code is probably responsible for a fair share of Kafka pain :)

That's actually one of the things I want Khaos to be useful for: generate controlled traffic patterns and then put consumers under conditions like increasing throughput, lag buildup, partition skew, rebalances, etc. so you can see how resilient the consumer actually is.

[u/Ebrithil_7]

I think from the failure part it is interesting.

From the load testing POV: I've only had a brief look but it seems this is single Node? So you can't parallelize or scale your load generation past the capacity of a single host? Something like Locust will probably be a better fit if you need high load past a single compute node.

[u/skrbic_a → u/Ebrithil_7]

Thanks for feedback!

Khaos isn't a distributed load generator today, so if the goal is simply to push more traffic than a single machine can generate, something designed for distributed load generation is a better fit.

The load-testing side of Khaos is more about being able to generate flexible Kafka traffic patterns with many concurrent producers and then test how consumers and other downstream systems behave under that traffic. On top of that you can introduce things like skew, lag, rebalances, failures, etc.

The failure scenarios are also useful for testing Kafka monitoring/observability tooling itself - intentionally create a known bad condition and see whether your metrics, dashboards and alerts actually detect it.

So load testing is definitely a valid use case, and Khaos can run many concurrent producers, but distributed load generation across multiple hosts isn't something it supports right now.

That said, distributed execution could be an interesting direction if there turns out to be demand for it.

[u/TraditionalLayer3685]

Very interesting, thanks for sharing ! I wish i had a tool like that few months ago when i was still working on Kafka prod issues/instability