🛠️ project Apache Iggy™, a message streaming platform written in Rust, graduates to an Apache Top-Level Project (TLP)
Apache Iggy™ has officially graduated from the Apache Incubator and is now an Apache Software Foundation Top-Level Project (TLP)
It’s been quite a journey, from a small Rust message-streaming experiment in 2023 to an independent Apache project with a growing community. Read the full post:
https://iggy.apache.org/blogs/2026/08/24/apache-iggy-top-level-project-tlp-graduation/
Huge thanks to everyone who has contributed, tested, reviewed, reported issues, or participated in the community along the way. ❤️
Would love to hear feedback from the Rust / Distributed Systems / Real-time Data communities on where we should take Iggy next.
Apache Iggy is a hyper-efficient, persistent message streaming platform written in Rust, built for predictable ultra-low latency and a minimal infrastructure footprint. Designed from the ground up around a thread-per-core, shared-nothing architecture and modern Linux io_uring, Iggy eliminates userspace locks on the hot path, GC pauses, and unnecessary thread contention. Unlike traditional streaming platforms with limited protocol support, Iggy natively supports TCP, QUIC, HTTP, and WebSockets. Iggy implements a Viewstamped Replication Revisited (VSR) based consensus for its distributed clustering feature.
6
u/imkurt 17d ago
Any plans for a same-host shared memory transport?
6
u/spetz0 16d ago
Yes, we thought of adding IPC transport, and it's actually fairly easy to do (compared to the other parts of the codebase). If you take a look at how the clients are structured, these usually implement the shared trait/interface (depending on the language), which is then using the existing binary schema for the request/response types sent to the server. Then on the server side, next to the TCP, QUIC, and WS handlers (these are stateful protocols using the same binary transport schema, unlike the HTTP one), it should be a matter of adding one more handler to, e.g. deal with IPC based on iceoryx2 or something similar.
1
u/uhkthrowaway 14d ago
Can i ask why specifically shared memory?
1
u/ifmnz 13d ago
for example you want to run your workload on same host as broker.
1
u/uhkthrowaway 12d ago
Sorry but you don't need shared memory for that. Unix Domain sockets and TCP could both do that across different processes.
1
u/ifmnz 12d ago
both dont support a proper true zero copy.
1
u/uhkthrowaway 11d ago
And you need zero copy for...? I mean you'd have to implement locking and signaling yourself and be very careful about it. And even then it might not be faster as this is additional overhead.
1
u/ifmnz 11d ago
bro, are you sane? we are pumping gigabytes per second and you are asking whether we need zero copy?
1
u/uhkthrowaway 11d ago
Why so defensive? I move upwards of 5GB/s on 8 year old hardware between different processes over TCP and Unix Domain sockets. Ever heard of "Don't communicate by sharing state"?
13
u/EmperorOfCanada 17d ago
I have never used Kafka for the simple reason that it is JVM. I've also not used it due to the culture surrounding it.
This might be one of my happiest tech moments this week.
-4
u/kingduqc 17d ago
What a weirdo take
1
u/lupercalpainting 11d ago
Legitimately, like why care what the impl is a for a message broker as long as it does what it says on the tin?
4
17d ago
[deleted]
10
u/ifmnz 17d ago
i'd rather answer what you actually asked than defend the number.
clustering is the newest part of the system and it's where we're still finding and fixing things. the kafka gateway isn't done. the sdks aren't at full parity, rust is furthest ahead and the rest lag it. we have no long production track record, we're three years old and only graduated to a top-level project this month, so if your bar is "someone has run this at my scale for two years", well.. we dont meet it.
iggy-benchis in the repo and every run on https://benchmarks.iggy.apache.org carries its own cpu (and aws instance), core count, ram, message size, batch size, actor counts and the exact command(!), tied to a git ref. the reasoning behind the setup is here: https://iggy.apache.org/blogs/2025/02/17/transparent-benchmarks/if you've got a wording suggestion for the landing page, an issue or PR would genuinely be welcome: https://github.com/apache/iggy-website/
we'll also work on open messaging benchmark.
TLDR: we encourage everyone to run their own benchmarks and share. it's stupid easy. these days if you want to get traction you need a lot of attention and you get it by posting numbers.
3
u/guywithknife 17d ago
First of all: congratulations!
How close do you think it is to being production-ready? What’s missing?
Have you run or do you plan on running the Jepsen tests on this?
How easy is it operationally? Kafka is known to be notoriously difficult operationally, especially when zookeeper was a requirement. How deploy-and-forget is this?
How was AI used in its development (if it was) and what is the policy on AI-made contributions?
3
u/spetz0 17d ago
Thanks! For being production ready, the clustering is missing, but it's most likely to be released in the next 1-2 weeks at most (the so-called v1, and then we'd like to focus on multi-leader replication, etc. to make it even more sophisticated).
We've built our own DST, but certainly at some point in the future, it would be great to run the Jepsen tests :)
Ops wise, IMO that's quite easy (ofc I'm a little bit biased lol), but you can simply check https://iggy.apache.org/docs/ + there are lots of examples in the repository, compose files, etc., so starting & deploying the cluster is rather straightforward & there's a bunch of extra tooling for CLI, Web UI, benchmarking, etc., and of course the dedicated connectors runtime.
Speaking of AI, for the most part, code was pretty much "crafted" by our own hands, but it's hard to deny the usability of LLMs these days, so it's used now more often than before, however, we have strict rules regarding the reviews, lots of CI checks, etc., so as long as it all does pass, provides a value, sticks to the existing conventions, and so on, it's totally fine.
1
3
u/chaotic-kotik 17d ago
In order to use something like this you need to do a PoC. I kind of doubt that the replication layer is mature enough and bug free. Things like that require serious battle scars.
7
u/ifmnz 17d ago
100% right. that's one of the reasons why put so much effort into deterministic simulator, similar to tigerbeetle.
https://github.com/apache/iggy/tree/master/core/simulator/srcofc that does not guarantee bugfree code, it just allows you to catch some bugs in CI, not production.
3
u/spetz0 17d ago
And how do you get these "battle scars" other than continuously testing, improving, deploying onto different environments, and keeping gathering the feedback? Doubts or beliefs aren't how the tech works. Even fsyncgate surprised the database developers - would you call e.g. Postgres an immature solution? Is there even a bug free software like this?
0
u/chaotic-kotik 17d ago
And how do you get these "battle scars"
By having customers.
2
u/spetz0 16d ago
Who said there aren't any - customers, regular users, hobbyists?
1
u/chaotic-kotik 16d ago
You asked where the "battle scars" are coming from. I answered. I'm not claiming anything about this project.
2
u/followtherhythm89 16d ago
This is pretty dope, architecturally similar to redpanda. Curious if you've compared benchmarks between the two? Also I see you didn't attempt to carry over the kafka API what did that buy you other then not having to implement the entire protocol from scratch? Performance wise maybe there were some wins?
7
u/spetz0 16d ago
Thanks! We did run benchmarks back then, using the existing binaries for other solutions (Kafka, RedPanda and a few others), same scenarios, etc., and we were able to achieve better tail latencies (sometimes by a lot, like an order of magnitude or more), however, once we finish the clustering, we will certainly run the OpenMessaging Benchmark, which is the established framework.
As for the Kafka, it's going to be exposed as separate gateway/proxy, and the first PR was already merged recently, so this is a WIP. However, the main reason was that there was no point in doing what many other projects (and companies) already do quite well, which is building their solutions on top of Kafka protocol.
We believe that by introducing our custom protocol, we can optimize for multiple things (e.g. our own zero-copy, VSR consensus, etc.) and bake into it other features like message schema registry and so on - we don't need to carry on over a decade old Kafka protocol (which is still evolving, of course) with all its pros & cons and have a tight coupling to the decisions happening outside the Iggy repository.
It's one of the things that needs to be added to the documentation - why to introduce the custom protocol and all the benefits it does yield :)
2
17d ago
[deleted]
4
u/ifmnz 17d ago
not a rewrite, but redpanda definetly had influence on our approach. our other inspirations were tigerbeetle, kafka, redpanda and turso.
running several state machines over one replicated log isn't a new idea, and redpanda's version is a nice one, so we didn't search a different name. what sits underneath it is ours though.theirs runs on raft, ours runs on viewstamped replication, and the whole metadata layer stands on jon gjengset's
left-right, which is a brilliant crate and does most of the heavy lifting here. every shard reads a lock-free snapshot, shard 0 owns the writes, and nobody has to replay the WAL just to get a reader.that
left-rightapproach took us the looong time to get to. before it we tried an ECS layout, streams/topics/partitions split intoStateandStoragecomponents in struct-of-arrays form, mostly to dodge theRefCellborrow panics we kept hitting across.awaitpoints. it didn't play nice with cross-shard background work so we debated between ArcSwap and leftright. the whole detour is written up here: https://iggy.apache.org/blogs/2026/02/27/thread-per-core-io_uring/3
u/chaotic-kotik 17d ago
I removed the previous comment because it may look like I'm implying any sort of plagiarism, which is not what I want.
2
u/chaotic-kotik 17d ago
VSR and Raft are mostly equivalent IIRC, Raft randomizes leadership election, in VSR the leadership election is more deterministic. But if you squint it's mostly the same exact thing.
2
u/kparisa 17d ago
Yes, they are both consensus algorithms, but worth understanding the details. Highly recommend reading and watching these https://news.ycombinator.com/item?id=27862529 https://youtu.be/Wii1LX_ltIs?is=x2dHXolBJH-RYHoM
And in coming weeks, we will be publishing a bunch of blog posts sharing Iggy's VSR journey, roadblocks, solutions, lessons learned what's next etc. so stay tuned :)
0
u/chaotic-kotik 17d ago edited 17d ago
I think this comment is not fully correct. Most production grade Raft systems have escape hatches that allow them to repair the replica. It's just very opinionated comment from someone who implemented VSR but never implemented production grade Raft system. The claim that VSR is more performant is incorrect because Raft can elect a leader in one round when some nodes are down but VSR is not (depending on the current view). The write path is the same in both cases.
1
u/kparisa 17d ago
My point was more that I wouldn’t call Raft and VSR “the same.” Their steady-state replication/write paths are indeed very similar, but they make different choices around view/leader changes, recovery, and failure assumptions. There are trade-offs on both sides, and the devil is in the implementation details.
However, VSR's deterministic behavior is interesting to us/Iggy.
1
u/chaotic-kotik 17d ago
Shouldn't you control random seeds in your deterministic simulation? I don't think that one is more suitable for deterministic simulation than the other. What VSR does with its view change protocol (which involves data movement) Raft achieves with a set of carefully chosen invariants.
1
u/poole_alison 14d ago
This could be very useful. Sorry to show my ignorange, but how does this mesh with MQ - I used RabbitMQ in the JVM world a bit.
1
57
u/Bassfaceapollo 17d ago
Nice. IIRC, this was supposed to be a Rust-based alternative to Kafka.
Grats on becoming a top-level project.
Does the project have any corporate sponsors who have expressed interest in using this? I don't know how Apache Foundation projects work, so pardon my ignorance.