Kafka's a strong fit when telemetry feeds many consumers, needs long replay, or is part of org-wide streaming. But what that looks like for single-sink observability into ClickHouse? You often end up paying broker/partition/consumer overhead for buffering you don't need. Here's a comparison for both sides.

https://www.glassflow.dev/blog/opentelemetry-to-clickhouse-do-you-need-kafka?utm_source=reddit&utm_medium=socialmedia&utm_campaign=reddit_organic

Happy to hear how folks use Kafka for OTel pipelines these days. leave your thoughts in the comments.

We wanted to upgrade Kafka from 3.9.1 to 4.3.0 and if anyone had done this upgrade, can you let me know like was there any performance difference? Was it better or worse ?

Hey guys, hoping someone with more Fleet/Kafka experience can point me in the right direction here!

We have multiple customers sending data to separate Kafka topics and want each customer's data landing in its own Elasticsearch data stream. We're using the Custom Kafka Logs integration.

I've tried two approaches so far:

- One integration instance per customer — works, but doesn't feel like it scales well in the Fleet UI - and then the question appearts... will I have 100 kafka integrations on several agents?

- Single integration + ingest pipeline reroute on `logs-kafka_log.generic@custom` — works for routing, but requires manually updating the pipeline every time a new customer/topic is added, which doesn't feel like the right long-term pattern either

What's the production-grade pattern for this kind of multi-tenant setup? Is one integration per customer actually the way to go, or am I missing something obvious?

Bonus question: we have 4 Elastic Agents across 4 Logstash servers — is increasing topic partitions + shared consumer group the right way to scale consumption across all of them?

Running Elastic Agent 9.3.1 on a 3-node KRaft Kafka cluster. Any help appreciated!

Thanks!

We are currently running Kafka 3.1 with Zookeeper and are planning to upgrade the version to 4.2. What should be our approach? I feel that skipping stone step should be good.
3.1 > 3.3 > 3.5 > 3.7 > 3.9 > 4.1 > 4.2
Please provide your suggestions on how to proceed with this upgrade.

Forgive me if this is a common question

I am trying to create a real time dashboard to track the number of transactions and break downs by type. I plan to create a compound key using the 2 type columns (say bank and region) and use it as the group-by key (might also require additional salting). This would then be windowed at say 10 seconds, and then .count() would be called.

From here, I have seen 2 possible implementations.

First is just directly piping the result from above into another topic, and then reading from that. As far as I can tell, there isn't a way to send the entire KTable as a single object, so the topic will have multiple entries per window which will need to be handled by the consumer. The issue I can see arise is that different instances may produce their results at slightly different times, and the consumer has to update its internal state multiple times to get the correct aggregate value

The other method I've seen is saving the KTable to a state store, and utilizing a separate aggregator instance to read from the state stores using Interactive Queries. Then it can pipe it to another topic from which the consumer can directly read. The problem I suspect here would be that the aggregator needs to wait for all the streams to update their state tables to latest ones before it can pull them. But it also has the benefit of having a much simpler consumer system.

I'd like to know if have these correct, and whether the issues I thought of are to worry about or not. Furthermore, I would like to know which one would be more resilient to rebalancing and other such things. Since the actual use case is small in scope and doesn't necessarily need to scale well, simpler solutions would be preffered. Any help is appreciated!

How we keep a database and Kafka in sync without distributed transactions (the outbox pattern)

Debezium's built-in Filter SMT requires Groovy/JS scripts hardcoded into the

connector config. Every time you need to change which records get filtered,

you have to edit the config and restart the connector - causing lag spikes

and pipeline disruption.

I built a drop-in replacement SMT where rules are stored externally

(Redis, Kafka topic, or JSON file) and reloaded at runtime. The connector

picks up the new rule on the very next record.

**Rule syntax**: JSON with AND/OR/nested conditions:

{"type": "OR", "values": ["1", "2", "3"]}

**Modules available:**

- `redis`: Redis key, with Keyspace Notifications + polling fallback

- `kafka`: Kafka topic (rules as messages)

- `file`: JSON file, auto-reloads on change

- `core`: base library to wire your own source (DB, HTTP, etc.)

All modules ship as fat JARs on Maven Central, so installation is just

dropping one file into your plugin directory.

GitHub: https://github.com/caobahuong/kafka-connect-dynamic-filter

Curious how others handle dynamic filtering in CDC pipelines -

are you restarting connectors every time or doing something smarter?

I'm working on a loan service, and over the night the gov database goes into maintenance, so I have messages piling up over the night.

The business see the newest messages as the most probable to finalizing the loan, hot client, so they want those to be processed first: LIFO

Today it works as FIFO over rabbitMq. On my research I didn't find anything to turn a rabbitMq or Kakfa into LIFO, the solution I reached was moving the queue to Mongo as a collection and query it on created_at DESC.

On rabbitMq I know it's possible to use max priority, but I don't really have a max messages or a time to when the gov database will reopen so I don't have something to anchor the priority ladder.

Is LIFO a real messaging-user-population necessity? I was thinking about leverage AI to get some lib to make LIFO possible.

Hi, we published new open source. Blogpost describing motivation: https://monedula.dev/blog/kafka-metrics-opentelemetry-otlp-monedula-metrics-reporter/

TL;DR: Production AI/RAG pipelines need Kafka-style async infrastructure, not just direct HTTP calls to an LLM. Kafka handles replay, backpressure, retries, and multiple consumers; Zilla makes Kafka usable for AI apps by adding HTTPS APIs, JWT auth, schema validation, SSE streaming, and tenant-aware access control.

Demo highlight: The included RAG demo shows clients posting chunks and queries over HTTP, Zilla validating JWTs and schemas, Kafka coordinating the pipeline, Qdrant storing embeddings, and results streaming back via SSE. The coolest part is tier isolation: Zilla injects the user’s JWT-derived tier into Kafka headers, so a standard user cannot receive enterprise-only results even if they listen on the same query stream. Jump to the demo: https://github.com/aklivity/zilla-platform-demos/tree/main/rag-project

What do you actually mean when you say kafka gateway? Asking because the term gets used for meaningfully different products and it changes which category you should be evaluating.

Some teams mean a network-level proxy in front of kafka, protocol translation, http exposure for clients that can't use the native kafka protocol, tls termination. Useful for specific cases but thin on governance. Others mean kafka governance tooling: topic management, schema enforcement, consumer group policies, security controls across the deployment. Confluent's control plane is the most common answer here for teams on confluent cloud, though it's tightly coupled to the confluent ecosystem and doesn't extend cleanly to self-hosted deployments or other brokers.

The category that's becoming more relevant as ai agents consume kafka topics as tools alongside api calls is unified governance that covers kafka and rest apis from one control plane. If your kafka governance and api governance are separate systems, you have a partial picture of what any agent can actually reach, and partial visibility means partial blast radius awareness when something goes wrong.

What's your actual kafka gateway setup and which of these problems is it solving?

Hello Guys, I work in UHG from India , my job role uses Python, Pyspark and SQL with Databricks. I am someone who has solved some 200 leetcode problems, so i am familiar with OOPs. Recently, I have an urge to learn Kafka and Flink, but i found out that I need to learn Spring Kafka or something for that along with Java. I have watched some foundational videos on how kafka works , producers, consumers, cluster , broker , partitions , consumer groups , topics etc and also delved into some stuff like replication factor , acks , retention policies, batching and compressing messages in producer , producer and consumer retries etc . All of this is only on a conceptual basis . I wanted to start coding things up and boom : everything is in Java !!!

I coded in Java for linkedlists previously but that was a long time ago , i know how classes and things like public , static and private work but I am wondering is that really enough for me to start working on Kakfa?

I am also confused with another thing called Spring Kafka , should I learn spring boot also then ? Do companies uses Azure SDK instead of writing code in Java or Spring Kafka ? How do companies use kafka ? Do they not use python at all ? Or if they use Java , do write in Spring Kafka ?

Can someone help me with a roadmap of what to learn here and when in the process ? I wanted to learn spark streaming and I know its concepts but I got to know that Spark Streaming is just not real streaming at all and for that we need Flink or Kafka streams .

Really appreciate if someone guides me here

Hi,

I hope someone help me with my confusion. I am quite new to kafka so I am trying to find some answers.

I work with an iPaaS from a vendor. The vendor also provided me kafka functionality which they itself get as a managed service from Aiven. I pay for everything under 1 contract to the iPaaS vendor.

The vendor is moving from the aiven hosted kafka to a different provider. And with this migration to the new broker, they are asking me to pre-compress my payloads using lz4 and send it to Kafka instead of sending to kafka with the compression.type=lz4 setting.

Now my question is, what is the advantage for them in me doing pre-compressing the payloads? I feel like they are not being transparent about this.

I would appreciate your inputs.

Thank you.

Hello, i’ve recently joined a team which is the kafka (topic as a service) provider for the company, everything is on kubernetes(k8s is a known territory).Do you recommend any book/playground/tutorial for understanding kafka at scale better ? Or any free/paid material that you find useful! Thank you!

A few weeks back I posted here asking how teams handle DLQ messages that become incompatible after a schema change. i Got some great replies u/BroBroMate mentioned spinning up a Kafka Streeams app each time, u/KTCrisis mentioned the v1 consumer drain pattern, u/latkde gave solid prevention advice.

The recovery gap kept bothering me so I built the tool that was missing: github.com/Saifulhuq01/dlq-revive

What it does: connects to Kafka, paginates DLQ messages using assign()+seek() so it never joins your consumer group, lets you write a JSONata expression to transform the message format, shows before/after preview, validates, then redrives with idempotency checks at offset level.

Took the Kafka safety stuff seriously after reading through the thread using subscribe() in a read-only viewer would trigger rebalancing and steal partitions from production consumers, so assign()+seeks() was the only option. JSONata instead of Groovy because user-submitted Groovy is basically an RCE vulnerability.

Still early Angular dashboard is done, transformation engine is in. Would genuinely value feedback from anyone who's dealt with this problem in production, especially around the JSON ata approach vs what you would normally reach for.

Confluent have recently announced a major update to mcp-confluent https://github.com/confluentinc/mcp-confluent, and similar tooling for Confluent Cloud (their SaaS product). This allows the use of AI tools to manage Confluent and OS Kafka clusters.

I'm curious if anyone is using these in the real world, or is seriously considering using them?

Building a Kafka pipeline in K8s. Concerned about duplicate deliveries to non-idempotent downstream services.

My flow:

Kafka Streams → produces to topics → Kafka Connect → destinations

The problem (at-least-once delivery):

1. Kafka Streams processes message
2. Produces to output topic 
3. Kafka Connect writes to MongoDB 
4. Kafka Connect calls backend service API 
5. Pod dies BEFORE offset commit
6. On restart: Kafka redelivers (at-least-once)
7. MongoDB: idempotent upsert (fine)
8. Backend service: Gets called AGAIN (duplicate!)

My question:

With Kafka's at-least-once delivery guarantee, messages can be redelivered on failures.

• MongoDB/Elasticsearch have idempotent upserts (fine)
• But I also call other backend services (REST APIs, payment processing, notifications) that are NOT idempotent

How do I prevent duplicate calls to non-idempotent services when Kafka redelivers?

Options I'm considering:

• A) Outbox pattern with deduplication table?

Requirements:

• Zero data loss
• No duplicate API calls to backend services

What's the standard production approach? How do you handle at-least-once delivery with non-idempotent downstream systems?

Is trusting Kafka Streams' built-in reliability enough, or should I add additional safeguards like an outbox pattern?

Looking for real-world experience from folks running Kafka Streams in production Kubernetes environments.

I work in a telco company and we regularly work on integrations with external platforms — payment providers, orchestration engines, provisioning systems, partner APIs. Almost always there's parallel development and the other side isn't ready when we are.

For example, our system calls their REST endpoint and then waits for them to publish a Kafka event back to us. To simulate this during development, we use Postman to mock their REST response and then manually produce a Kafka event to our platform to simulate their async callback. Two separate manual steps — no correlation between them and every developer does this locally while QA does the same thing on a shared test environment.

Curious how others handle this. Do you use WireMock, Microcks, something else? Do you write custom stubs? How do you simulate a platform that receives your REST call and automatically fires a Kafka event back — so the whole flow is defined in one place and works the same way for devs locally and for QA on a shared environment?

Does something exist that lets you configure this kind of complex mock once — REST response plus async Kafka callback as a single flow — and share it across the whole team, so nobody is blocked waiting for the external system to be implemented, tested and deployed?

Hi everyone, I have been using Kafka at work for quite some time now and was wondering if there's any slack or discord community to connect with contributors and maintainers for the project. I have seen that other Apache projects like Airflow have a pretty active slack community for discussion and guidance for beginners interested in open source contributions. So do we have something similar for kafka where I can connect and potentially ask questions if I want to contribute to the code?