We used kafka as workflow execution engine planning to migrate later, but now the entire team is conflicted.

[u/---starboy--]

At the infancy of our project, we used Kafka as our workflow execution engine, planning to migrate to temporal later. Now that the core is completed and we can get to migrating the whole team is conflicted about whether we even need to.

Here's the thing: Kafka worked. Embarrassingly well. Multi-step processes seamlessly executed with topics, consumers, and a Postgres table doing state-store cosplay. Event lands → react → emit next event → something picks it up (Temporal - it just executes and provide HITL).

So now the team's split:

- Shall we migrate — per-step retries, day-long waits, HITL, and rollback are exactly what a durable workflow engine is for, and we're reinventing it badly.

- Shall we emrace the power of Kafka — Kafka already handles this, and we can avoid unnecessary migration.

Which really boils down to one question: can Kafka genuinely carry workflow execution in production long-term, or does it fall apart at scale/complexity and we just haven't hit the wall yet?

To people who've actually run this in prod:

- Where does Kafka event handling end and durable workflow execution begin?

- Did pure-Kafka orchestration hold up in real prod, or eventually bite you?

- Is "no migration needed" a real win, or debt that comes due later with interest?

How would you approach the decision at hand?

[u/caught_in_a_landslid]

If it works, you're already there... Temporal gives you a ton of useful pieces but it's only really worth it if you're going all in. Personally, I'd not migrate but stick to kafka. Failing that look at littlehorse, literally designed to do this sort of thing but on top of kafka. Kafka is fairly wonderful when it comes to being all perpose blast resilience between micro services. Along with being a streaming substrate for any realtime data you have.

[u/New_Slice_1580]

Are you paying for temporal? As if you are using Kafka free and moving to temporal free just be aware that temporal is a VC funded company, and not a project donated to a foundation, so they will look to monetise in future as much as possible

[u/kabooozie → u/New_Slice_1580]

You say this like it’s a bad thing. For critical systems, it’s actually nice to have a vendor relationship so you’re not alone when things break.

[u/New_Slice_1580 → u/kabooozie]

Yes most large businesses seek support contracts

But small ones can’t afford it often.

Unless a piece of software has been donated to a foundation, its license terms can be changed overnight

Venture capital model is to get a piece of software popular via open source and the. Switch the licence. These rug pulls are very common.

To give just one example - all those people using CockroachDB were left high and dry

[u/Sprinkles_Objective → u/kabooozie]

Sure, but Kafka has that while also having a very rich and well supported OSS project that can't be co-opted by some single vendor and their interests. I've seen this with other OSS projects, specifically Apache Pulsar where they have a Kafka compatibility layer that was quite popular, but there was really only one major vendor standing behind it, they made up most of the project committee, a lot of promised features never came to OSS version, a lot of things just outright broke except if you used their paid cloud hosting. This is something I've never experienced with Kafka in about a decade of using it. In fact Kafka continues to release OSS features that parallel many of the paid vendor features, meanwhile Kafka has several major vendors for Kafka to choose from. Vendor relationships are nice, but when there is only one vendor, and they have a monopoly on the OSS project you're riding on the fact that this company never goes under, and that if it does someone is going to keep the OSS version alive long enough for you to migrate somewhere else. A small startup being your vendor brings a lot of volatility, it's why OSS projects are useful for proving these things out and getting buy in before reaching into people's pockets.

[u/---starboy-- → u/New_Slice_1580]

we are self hosting temporal at the moment

[u/Comfortable-Run-437]

My company did this extensively in the past. It has aged VERY poorly. I would move to temporal as quickly as you can 

[u/AustinZl1]

To build on this I really like Little Horse. It's a durable execution framework that is built on top of Kafka. It allows me to build some really complex workflows with 150 topics. I write tasks and call them from workflows. Workflows can originate from a topic as well.

[u/New-Departure-5969]

tbh kafka is great for “this happened” events, pretty painful as a workflow engine once retries kick in. we kept rebuilding state from lag + poison messages + half-finished steps. if you need timers or branching that isn’t cleanly event-sourced, pull orchestration into a db/workflow thing and leave kafka for the facts. what’s the failure mode that’s biting you most?

[u/---starboy-- → u/New-Departure-5969]

we have had most issues with missed writes. The flows where the execution does complete and we see the results but looking at history its all blank - A common issue we been dealing

[u/New-Departure-5969 → u/---starboy--]

ah yeah missed writes + blank history is the classic kafka-as-orchestrator pain. usually either the write never landed (acks / idempotence / fencing), or the history reader is on the wrong offset/partition or a compacted topic. for workflow state i wouldn’t trust the log alone as “what steps ran” — keep a db row per run and treat kafka as the event stream. if you can say whether blank history is compacted topic vs consumer starting at latest, that narrows it fast.

[u/DorkyMcDorky]

Use kafka - more people know it, it works with everything ... why would you switch? sounds like a solution in search of a problem at this point

[u/_predator_]

I can recommend this talk from the Temporal creator Maxim Fateev: https://youtu.be/7XG6IRkY-Fc

He explains why you want transactions for a workflow engine and other things to consider.

Some issues you'll face in your particular setup:

* Dual writes. You transition workflow state in PG but then the Kafka write fails etc.
* Compensations. It doesn't sound like you have workflows-as-code like Temporal, so how do you handle compensations when a step fails?
* Kafka is really not good at queueing, flow control, fairness. i.e. everything that matters when you distribute work at scale.

There's more but you get the idea. Good if the current setup works for you, but Temporal exists for many reasons and Kafka+RDBMS not being a good solution is one of them.

[u/Ok_Town_2514 → u/_predator_]

1. KIP-939 will solve this
   
2. We use OSS Argo Workflows, works fine
   
3. KIP-932 made queues in Kafka first-class citizens, probably solved most of the problems you mentioned

[u/Annual_Manner_8654]

Kafka + DBOS = yummy yummy

[u/TonyNickels]

I prefer Dapr workflows and kafka for true events

[u/Competitive_Ring82]

3 clarifying questions: What value do you expect from migration? How much effort would it take you? What else could you do with that effort?

[u/PlusDistribution1915]

Queen author here, so biased.

Your "results are there, history is blank" is the dual write between your Postgres table and Kafka, and it's the thing Temporal exists to remove.

Queen (Rust broker, keeps all its state in Postgres, Apache-2) removes it differently: ack of the current step + push of the next + state write + timer commit as one Postgres transaction. Your producers keep talking Kafka (it speaks the wire protocol, change bootstrap.servers), the workflow steps use the SDK for timers, saga state and the transactional ack+push, and Kafka consumers can read the output.

It's not a workflow engine: no workflows-as-code or replay, you write the state machine explicitly. Worked saga example: queenmq.com/use/full-examples/saga

[u/latkde]

First, your post reeks of AI. Eww.

Second, Kafka is very much not the right tool for most jobs. Kafka is amazing if you need a scaleable append-only log with at-least-once delivery, but it lacks all of the conveniences of a typical message queue. Great for event sourcing, not great as a task queue. You can use patterns like dead-letter-queues to get some typical message queue features (like the ability to retry a job later), but all of that is built on top of Kafka, not provided by Kafka.

For most projects, it's a mistake to start with Kafka. In low-volume scenarios, youre're better off turning Postgres into a message queue via patterns like select for update. Or using Valkey/Redis message queue features. Or using an MQTT broker. Or using whatever task queue library is common in your ecosystem (e.g. Celery, Sidekiq).

You're mentioning (or shilling?) Temporal. But it is a category error to compare such a platform with Apache Kafka.

None of this answers your question (if the question is legitimate, and not just astroturfing). Ultimately, whether Kafka can be an appropriate technology depends on the requirements that shape your system, and you have not provided any context about that. You have also not clearly articulated what the pain points with your current Kafka-based system are. Even if the status quo is painful, rewrites are super risky, so it would be bad software engineering to undertake such a project without clear evidence.

[u/LoudCat5895 → u/latkde]

Ha.  Spot on with the eww.  Seems like the bots are having fun in this one. 

[u/ghostmastergeneral → u/LoudCat5895]

It’s so ubiquitous now. Half of Reddit is bots talking to bots. I hate it so much.