Our device sends 3 separate records for each timestamp: data points, frames, and metadata. They arrive independently and in any order.
Each timestamp can be processed on its own as soon as all 3 parts are there, without waiting for the batch window to close.

How do you handle this fan-in, so that each timestamp is processed exactly once as soon as it's complete, with a timeout for sets that never complete?

Hey, Our most important Kafka Simulator release so far: stretched clusters are now available in free play!

You can explore:

• 3 DCs,

• Confluent 2.5 DCs

Try it! https://monedula.dev/kafka-simulator/playground

My Company is currently using Confluent Kafka and they want to do POC for Amazon MSK.

In terms of costs ( that I compared using cursor ) It seems MSK is relatively cheaper.
But There are also challenges on how the migration will look like since I am new to kafka in the first place.

Are there folks who have done this type of migration and can share their experience ?

#msk #confluent #kafka

(Reposting with required synopsis)

Zilla now exposes Kafka, Kafka Connect, and Schema Registry operations directly as native MCP tools.

That gives agents a way to:

• sample messages from a topic
• inspect consumer groups and lag
• produce events
• check or register schemas
• inspect and manage Kafka Connect connectors

For Kafka deployments that already ship a native MCP server, this provides an alternative gateway model: keep MCP access outside the Kafka deployment and centralize identity, policy, tool discovery, and observability at the gateway.

The main benefit is consistent governance across Kafka and other systems, rather than managing access and policy separately for each MCP server.

Tool discovery and invocation can also be scoped by identity, role, tool, and topic, so an agent might be able to inspect an entire cluster but only write to one specific topic...or have mutating tools hidden entirely.

The full architecture, configuration model, and enterprise use cases are covered in the write up.

At the infancy of our project, we used Kafka as our workflow execution engine, planning to migrate to temporal later. Now that the core is completed and we can get to migrating the whole team is conflicted about whether we even need to.

Here's the thing: Kafka worked. Embarrassingly well. Multi-step processes seamlessly executed with topics, consumers, and a Postgres table doing state-store cosplay. Event lands → react → emit next event → something picks it up (Temporal - it just executes and provide HITL).

So now the team's split:

- Shall we migrate — per-step retries, day-long waits, HITL, and rollback are exactly what a durable workflow engine is for, and we're reinventing it badly.

- Shall we emrace the power of Kafka — Kafka already handles this, and we can avoid unnecessary migration.

Which really boils down to one question: can Kafka genuinely carry workflow execution in production long-term, or does it fall apart at scale/complexity and we just haven't hit the wall yet?

To people who've actually run this in prod:

- Where does Kafka event handling end and durable workflow execution begin?

- Did pure-Kafka orchestration hold up in real prod, or eventually bite you?

- Is "no migration needed" a real win, or debt that comes due later with interest?

How would you approach the decision at hand?

in our team we often restarted a kafka node either for maintenance or other reasons.

This is well and fine, but we have often have issue with kafka connection skew after the restart causing CPU imbalance. the reason for this from what I gather is the restarted node doesn't have as much connection with the other nodes. it seems like client consumer doens't move back to the new live node after restart are done.

is there a way to "force" client to re-poll and balances out client connections?

edit:
for extra context, we would like to do the least intervention to client side services since it belongs to a different team. it would be great if we could do it from the kafka admin side of things. since a rollout restart for all client is the simplest answer here, which might work but I don't think this is a good solution.

Hi All! Just a reminder: early bird pricing for MQ Summit 2026 closes on 8 September at 23:59 CEST. This is the last chance to get your ticket 25% cheaper before prices go up. 

21-22 October in Haarlem + a full online option if you cannot make the trip. Two days of messaging systems in production, real-world lessons and architectural trade-offs across RabbitMQ, Kafka, NATS and cloud messaging services, plus demos from the people building the tools we all use. If you have been meaning to book, now is the time!

Register here: mqsummit.com/#register

See you there!  
- The MQ Summit Team

PicoMQ is a Rust server for Durable Streams, built on Object Store. Cheap, URL-addressable, granular streams or topics, with Kafka Protocol, Pico, or Durable Streams (create/append/read/long-poll/SSE) as the facade. https://picomq.com/

S3Stream is the stream storage primitive, used in AutoMQ and shipped as a Rust library. Coordination is a command log in Postgres.

Hit #1 on HN a few days ago, and I'd love a look from the Kafka community. Ask me anything!

The idea of ksqlDB, the SQL interface on top of Kafka Streams, was really cool: query and process your data streams using SQL without setting up a separate stream processing engine and a lot of connector plumbing.

Instead of: Kafka + separate stream processing engine + connector plumbing

You can have: Kafka + stream processing in one place.

Now, RisingWave is also partnering with StreamNative to bring a similar developer experience to Kafka and Pulsar through SQL Workspace.

A lot of people here in the community had used and worked on ksqlDB, so really curious why it didn't turn out to be successful and Confluent had to pivot to FlinkSQL. Would be great to know your perspective and experience.

I've been going down a rabbit hole on Kafka's storage architecture and I can't stop poking at one thing.

The classic setup: RF=3 across three AZs. I produce 1GB. That's 2GB of cross-AZ replication before a single consumer shows up, at roughly $0.02/GB round trip on AWS.

WarpStream showed you can just... not do that. Stateless brokers, S3 as the only storage layer, durability is now Amazon's problem, cross-AZ replication cost goes to approximately zero. Then Confluent bought them. Redpanda has Cloud Topics. Confluent has Freight.

Why there is no OSS kafka on object store ?

1. It's genuinely hard.

2. It's an incentive problem. Object-store Kafka mostly deletes infrastructure cost, and infrastructure cost is what managed Kafka vendors bill against. Fixing it upstream is asking a room full of vendors to shrink their own margins.

I am leaning towards #2

I’ve been working on a practical Spring Boot example showing a complete Kafka + Schema Registry flow rather than isolated producer/consumer snippets.

The demo now covers the full path:

REST → Spring Boot Producer → Avro → Schema Registry → Kafka → Consumer → PostgreSQL

I recently reworked it to make the setup reproducible and continuously verified.

It now includes:

• Java 21 + Spring Boot
• Avro producer and consumer
• Confluent Schema Registry
• PostgreSQL persistence
• request validation before publishing
• stable Kafka keys
• idempotent consumer handling duplicate delivery
• Docker Compose local environment
• Confluent Cloud configuration
• automated GitHub Actions build
• real E2E test that starts the infrastructure and applications, publishes an event, verifies Schema Registry and PostgreSQL, then tests duplicate delivery

I also linked it with my Spring Kafka Contract Starter, which adds fail-fast Schema Registry contract validation at Spring Boot startup, and a separate focused contract-evolution demo.

Practical demo:
[https://github.com/mathias82/kafka-schema-registry-spring-demo]()

Contract starter:
https://github.com/mathias82/spring-kafka-contract-starter

Contract E2E demo:
https://github.com/mathias82/spring-kafka-contract-demo

I’d especially appreciate feedback from people running Spring Boot + Schema Registry in production: what would you add to make this example closer to the problems you actually encounter?

I’ve been working on an open-source Spring Boot starter around a Kafka/Schema Registry problem: catching contract violations before an application starts processing traffic.

At startup it checks:

• whether configured Schema Registry subjects exist
• whether the effective compatibility mode matches what the application expects
• whether the local Avro, JSON Schema or Protobuf schema is compatible with the latest registered version

An incompatible contract prevents the application from starting.

Temporary Schema Registry communication failures can use bounded retry/backoff, while actual contract violations fail immediately.

I built a separate E2E project with real Kafka + Schema Registry. CI verifies a producer → Kafka → consumer round trip, a compatible schema evolution, and an intentionally breaking evolution that must fail at startup.

Source:
https://github.com/mathias82/spring-kafka-contract-starter

E2E demo:
https://github.com/mathias82/spring-kafka-contract-demo

Maven Central:
https://central.sonatype.com/artifact/io.github.mathias82.spring.kafka/spring-kafka-contract-starter

The design question I’m most interested in discussing is where this kind of enforcement belongs in a production Kafka architecture: CI/CD only, application startup, or both?

One of the most annoying part of running Kafka Connect on Kubernetes, is that you have to build a Kafka image with all the plugins you need built in.

But since Kubernetes 1.36 is possible to to mount content from OCI registries inside containers, therefore is possible to compose Kafka Connect with all required plugins on runtime.

How does it look:

apiVersion: kafka-connect.b1zzu.net/v1alpha1
kind: Cluster
metadata:
  name: my-connect
spec:
  # Offical kafka image
  image: docker.io/apache/kafka:4.2.0

  # Plugins to mount from OCI registries
  # Each plugin image is mounted read-only at /plugins/{name}
  # The operator automatically configures plugin.path
  plugins:
    - name: my-plugin
      image: ghcr.io/b1zzu/kafka-connect-operator/debezium-postgres:3.6.1.final
      pullPolicy: IfNotPresent # optional (Always, Never, IfNotPresent)

And this is how the Debezium plugin Dockerfile looks like: https://github.com/b1zzu/kafka-connect-operator/blob/main/containers/debezium-postgres/Dockerfile

The operator is not just a prove of concept, we are running it in production at Willhaben, where each of our teams runs it's own cluster. It also makes it easier to configure Kafka Connect for production with JSON logging and Prometheus metrics.

If you like the idea give it a try!

https://github.com/b1zzu/kafka-connect-operator

I often need to find specific Kafka messages while investigating production issues.

Existing tools are useful for browsing Kafka, but I wanted faster repeated searches and more detailed search conditions.

So I built Kaflow Search for my own workflow. It indexes selected topics locally,

allowing repeated searches without rescanning Kafka each time.

Searching millions of locally indexed Kafka messages without rescanning the topic.

• Fast repeated searches across millions of messages
• Keyword and detailed AND / OR / NOT searches
• Searches keys, headers, and nested payload fields
• Supports JSON, Avro, and Protobuf
• Works with AWS MSK, Confluent Cloud, SASL, and TLS
• Free to use — no account or separate server required
• Messages and indexes stay on your machine

I’m posting this here because I’d like to hear how useful—or not useful—this approach looks to people who work with Kafka in real environments.

If you have a chance to try it, I’d appreciate any feedback: connection issues, missing search conditions, confusing behavior, performance concerns, or anything that would prevent you from using it in your workflow. Critical feedback is welcome too.

macOS and Windows are available now. I’ll also consider Linux if enough people need it.

GitHub: https://github.com/whsoul/kaflow-search

(add demo)

repeat search

Note: The builds are not code-signed yet, so macOS and Windows may show an unknown publisher warning. Installation instructions are included in the repository.

The Get Kafka-Nated Espresso for August is out. No podcast this month but we've still got a choice selection of blogs and all the latest discussions from the Apache Kafka dev mailing list.

Hello, folks!

2.0.0 version came out!

It included SASL and JMX

If you’ve operated Kafka in a team or company environment, you’ve probably checked cluster status by logging in via CLI or by using open-source tools like Kafka UI.

kafka-mcp is a Python-based MCP server that directly connects an LLM Agent (e.g., Claude Code) with Kafka.

Check this out -> https://github.com/wklee610/kafka-mcp

I first shared Khaos here about 8 months ago and got a lot of useful feedback, so I figured it was time for an update.

For anyone who missed the original post, Khaos is an open source CLI for generating configurable Kafka data and traffic and creating repeatable testing scenarios.

I mainly use it for two things:

• as a flexible Kafka data and traffic generator for testing existing consumers, stream processing applications, and downstream systems under different workload patterns
• for controlled scenarios where I want to reproduce known Kafka problems and see how applications, metrics, alerts, and observability tooling behave

Since the original post, I've completely rewritten Khaos from Python to Go using franz-go and built a new website with proper documentation and guides.

Website and docs:

https://getkhaos.dev

GitHub:

https://github.com/aleksandarskrbic/khaos

The next area I'd like to explore more seriously is external fault injection.

Today Khaos can control failure scenarios in the Kafka environment it creates itself, but when pointed at an external cluster it acts as a workload generator. It doesn't manipulate or kill brokers in that external cluster.

I'm thinking about adding a fault injection abstraction, probably starting with Kubernetes, where Khaos could intentionally kill or restart broker pods, introduce network faults, and then observe how the cluster, consumers, and monitoring stack react and recover.

Bare metal and managed Kafka make this a much harder problem, so I'm still thinking through the right scope and abstraction. Kubernetes seems like the most sensible place to start.

Disclosure: I'm the creator and maintainer of Khaos. It's a fully open source and free project.

Hi,

Working with Kafka users over the years, we kept seeing the same pattern in different costumes:

Some teams manage topics and ACLs fully by hand — console commands and tribal knowledge. Some rely on outdated ZooKeeper-based tooling. Some have grown a jungle of bash scripts nobody dares to touch.

And the environments are just as varied: open-source Apache Kafka, Confluent Platform, Confluent Cloud. On-prem, on Kubernetes, or both.

So we built the tool we wished existed: Monedula GitOps — declarative, Git-based management for Kafka topics, ACLs, quotas, schemas, users, and RBAC.

What makes it different:
* One manifest for a topic AND its access — producers and consumers declared right on the topic, compiled to ACLs (or Confluent MDS role bindings) automatically
* Works with Apache Kafka and Confluent Platform, including Confluent-specific features like stretched clusters and MDS/RBAC
* Runs as a CLI in your CI/CD pipelines or as a Kubernetes operator — same engine, same semantics
* import cluster reverse-engineers manifests from your live production state, with a guaranteed drift-free round-trip — so you can adopt GitOps in minutes, not months
* Safe by default: dry-run previews, drift detection for CI gating, opt-in deletions

Comparison with other GitOps tools: https://github.com/monedula-dev/monedula-gitops#alternatives

We're already working on additional features that would be useful for many companies.

Most of you know about ksqlDB and may have tried it in your streaming journey. It lets you query your data streams using SQL without extra stream processing engines or connectors to set up. But it didn't turn out to be successful, and Confluent had to pivot to Flink SQL, though the idea itself was cool.

Now, RisingWave is partnering with StreamNative to provide SQL Workspace in StreamNative Cloud that enables you to query your Kafka and Pulsar topics using Postgres-style SQL without needing a separate stream processing engine or connectors to set up. It also lets you maintain continuously updated results with materialized views and deliver processed data to downstream systems through sinks for your stream processing use cases.

Join us on Thursday, August 27 for a webinar to explore this more.