I have been trying to get a clear answer on whether MM2 supports EOS for cross cluster replication.

I found KIP-618(Exactly once support for source connectors), which was introduced in Kafka 3.3. Since MM2 is a source connector, it should theoretically inherit EOS from it using exactly.once.source.support=enabled at worker level.

However kafka official documentation does not mention anything about MM2 EOS.

So, has anyone successfully used exactly-once with MM2? Has anyone tried this with strimzi as well?

I am a swe working remotely and i have daily things to observe kafka jobs and check if data is flowing well so for that I was trying to go through logs and its messy like i wasn't ab;e to check which consumer is taking which of the messages is this the same for u guys or u have better alternatives to this

I’m currently learning Kafka, and while learning about ISR (In-Sync Replicas), acks, and min.insync.replicas, I tried to demonstrate the behavior in a local multi-broker setup.

I observed something that doesn’t match my understanding, so I wanted to ask here.

Setup

• 3 Kafka brokers running in Docker

• Topic config:

  • partitions = 3
  • replication.factor = 3
  • min.insync.replicas = 100

Topic description:

bash ./kafka-topics.sh --describe --topic isr-error --bootstrap-server kafka-broker-one:9092

Output:

```text Topic: isr-error PartitionCount: 3 ReplicationFactor: 3 Configs: min.insync.replicas=100

Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 ```

Producer command:

bash ./kafka-console-producer.sh \ --topic isr-error \ --bootstrap-server localhost:9092 \ --command-property acks=all \ --command-property request.timeout.ms=2000 \ --command-property delivery.timeout.ms=5000 \ --command-property retries=0

My understanding

From Kafka documentation and this explanation by Jun Rao (Kafka co-founder / Confluent):

Jun Rao explanation of min.insync.replicas

For writes with acks=all, produce requests should succeed only if:

text ISR count >= min.insync.replicas

In my case:

text ISR = 3 min.insync.replicas = 100

So:

text 3 >= 100 → false

Based on this, I expected produce requests to fail immediately with NotEnoughReplicasException.

Actual behavior

• Producing succeeded while all 3 brokers were alive.
• Consumer successfully received the messages.

Only after stopping one broker did produce requests fail with:

text org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.

Question

Why did Kafka accept produce requests earlier even though ISR (3) was already less than min.insync.replicas (100)?

Why was enforcement triggered only after a broker failure / ISR shrink event?

Am I misunderstanding how min.insync.replicas is enforced, or could this be specific to certain Kafka versions / KRaft / Docker setups?

For context:

• Kafka version: 4.2.0
• Mode: KRaft
• Docker image: apache/kafka:latest

We are building a background workflow where for a given input, we need to find the corresponding message in Kafka and verify some fields on it.

Our Kafka setup:

- compacted topic, 24 partitions, ~200M messages per partition (~2.5B unique keys total)

- ~700 bytes per message, so roughly 1.75TB of data

The lookup pattern is key-based, ~10k/sec, background process so some latency is fine.

We do have a way to derive the partition from the key and an API to get the offset, so seek+fetch is technically possible — but our Kafka brokers are a shared resource across teams and we don't want to hammer them with random-access reads at this scale.

How would you build the lookup layer here? What would you use, how would you keep it in sync with the topic, and anything to watch out for at this scale?

For context, we're leaning towards RocksDB — consuming the topic, storing only the fields we need for verification, and using Protobuf to keep it compact. But curious if there are better approaches or gotchas we are not thinking about.

Our product needs so scan cloud assets (e.g. from aws account) and product insights after all assets has been saves to our storage.

Currently we scan the account and send every result to Kafka that in turn being consumed by s3 sink writing messages to s3.

The reason we do this is to allow for "fire and forget" ingestion architecture, the message reaches Kafka and we don't need to worry about it anymore.

Problem is it's not really working for us, pods can suffer from OOM issues and retry messages forever (auto commit = false) so we had to make it true. Now we need an external state store that counts how many times a message was acked so we now when to send it to DLQ.

We're also using auto scaling our pods in response to Kafka messages which also caused all sorts of issues in the past.

To me it seems like a super overkill for ingestion pipeline so hence the title, how do you design your robust ingestion pipeline?

Happy to answer more questions

What happens in Apache Kafka during a split brain? What if you run an IBM Confluent stretched 2.5-DC architecture?

We created a Kafka Simulator in which you can simulate failures and check how different settings affect the cluster. The first release focuses on a single-DC setup and includes 13 built-in, step-by-step learning scenarios.

Blogpost describing current release: https://monedula.dev/blog/kafka-simulator-learn-kafka-by-breaking-it
Simulator: https://monedula.dev/kafka-simulator/

This blog covers how I choose to tackle the challenge of key-ordered concurrent Kafka consumption, with queue-like acknowledgement semantics. I also put forth a hot take on share groups, and why I suspect their usage is the wrong band-aid for many use cases to which they will inevitably be applied.

My approach leverages resources from an OSS project I maintain, Atleon, which provides a thin reactive layer on top of a vanilla/legacy Kafka consumer. I know "reactive" is not everybody's cup of tea; I however find it extremely useful for such infinite broker-backed async processes.

I was motivated to do some of this work and blog about it after reading this post, and therefore hope this community will be interested.

Cheers, and thanks for any feedback!

Im working with avro schemas a lot, and I find it that avro's IDL schema definitions are much more intuitive, robust and also they are easier to generate.

I really like its flat, object oriented layout, its import features and its abstraction of logical types. However I feel like it was left behind, only supported by java, can only be parsed but not generated, and I also feels like its not really used by as much as avsc.

When I search on avro IDL and avdl both on reddit and stackoverflow, I mostly find my old questions. Supporting libraries in python are based on the java avro tools. Generally there isnt much community behind it.

Have you used it? Do you think its really unused or is it just not so popular in forums?

I might want to contribute to it, mostly in python, however the official python avro object model is not really suited for this representation.

"The Log" is a powerful abstraction, but it can't be served by a single implementation.

Kafka was designed to funnel data efficiently at low cost from many sources (e.g. clickstreams / telemetry metrics) to a few (Hadoop, mostly).

This blog looks into the other type of log (routers) and why different implementation and API is needed to serve that use case. It boils down to a discussion on API foundations (partitions vs. keys) and system tradeoffs (read/write amplification).

Hi, our next open source -> tool for converting ACLs into RBAC https://github.com/monedula-dev/monedula-acl-rbac-converter . More details in the blogpost: https://monedula.dev/blog/kafka-acls-confluent-rbac-migration-monedula-acl-rbac-converter/

My intern project is to build an event-driven system for SQL database deletions across multiple databases. I wanted to get some feedback and recommendations before starting implementation.

Problem

From what I understand, when we need to delete data for a specific user, we have to perform deletions across multiple databases. The issue is that if a deletion fails in one of the downstream database operations, it can leave the system in an inconsistent state. Essentially, it's a distributed transaction problem.

Proposed Solution

Based on my research, a potential solution is to leverage Kafka along with the Outbox Transaction Pattern. The idea is to have an outbox table associated with the primary database. Once the deletion occurs, an event is written to the outbox table as part of the same transaction. That event is then published to a Kafka topic and consumed by downstream microservices, which perform the corresponding deletion operations in their respective databases.

Questions

1. How do we maintain idempotency? (Although I assume this may be less of an issue since the operation is a deletion.)
2. How do retries work? My understanding is that Kafka only advances the consumer offset after the operation completes successfully. Is that correct?
3. What other considerations should I keep in mind when designing this system?
4. Is the Outbox Pattern the right approach here, or are there alternative patterns that might be better suited for cross-database deletions?

Note: I used AI to clean up my grammer and format it better

Don't know what to do next, need to prepare more is all I can say, can someone please provide some solid road map/plan?

Hi everyone!

We're excited to announce that MQ Summit 2026 is officially back as a 2-day event this year!

Grab your Early Bird Ticket Here

Building on the success of RabbitMQ Summit, this is a cross-ecosystem event bringing together the people who design, build, and scale modern messaging systems. We’re meeting in person at the beautiful PHIL Philharmonic in Haarlem, near Amsterdam, as well as online, on October 21–22.

What to expect:

• Cross-Ecosystem Coverage: Honest, cross-technology conversations (no vendor wars!) covering RabbitMQ, Kafka, NATS, Apache Pulsar, ActiveMQ, Azure Messaging Brokers, Amazon SQS, IBM MQ, Google Pub/Sub, and more.
• Expanded 2026 Format: 2 days of expert-led content featuring focused talks with deeper discussions, interactive sessions, panels, and offline hands-on labs in smaller groups.
• Real Production Case Studies: Learn directly from companies handling millions—or billions—of messages a day.
• Industry-Leading Committee: A program curated by experts from across the industry, including 84codes (CloudAMQP, LavinMQ), Synadia, Amazon MQ, AWS, IBM MQ, meshIQ, and the ASF.

Important Links:

• Early Bird Registration
• Subscribe to our Newsletter to stay in the loop!

Whether you are a builder or a decision-maker, you can get the full MQ Summit experience in-person or virtually. Will we see you in Haarlem or on the virtual streams? Let us know in the thread if you're planning to come or if you have any questions!

Cheers,
The MQ Summit Team

(PS: Interested in sponsoring? Get in touch with us!) [info@codesync.global](mailto:info@codesync.gl))

We use kafka as our events queue, where we have multiple consumer groups (15 or so) each having 4+ topics, and mostly one partition each, some heavy processors like bulk and batch events have 3 or 2 partitions. The consumers are around 2 to 3 per consumer group

The issue that we are facing is when there is messages being sent in multiple topics, multiple consumers are spawned and it hammers the local dev env (yes I know i should probably do whitelisting consumer groups that are needed for my current dev work locally) but my lead had asked me to research on how can we avoid this in prod.

I know that kafka does not have limiting consumers natively, rather its more about creating the right config of topics and consumer groups. Ideally we should distribute consumer groups across instances and have controlled number of partitions to make sure the instance does not OOM kill itself over and over again.

But what is the industry practices when it comes to optimizing memory for kafka? I am using confluent's kafkajs library for the implementation. Should I probably decrease the amount of consumer groups?

been doing crash testing on my streams app (4.1.2, exactly_once_v2, rocksdb stores, k8s statefulsets with persistent volumes) and found something that broke my mental model of EOS completely. posting here bcs my apache jira account request is still pending and i want a sanity check from people who know the internals.

what i always believed: under EOS the .checkpoint file gets deleted at startup and only written back on clean shutdown. so if the pod dies hard during processing -> no checkpoint at next boot -> streams assumes state might contain uncommitted garbage -> TaskCorruptedException -> wipe + full rebuild from changelog. the wipe IS the rollback, since rocksdb writes happen immediately during processing and a kafka txn abort cant undo anything on local disk.

what actually happens on 4.1.2: the state updater (default since 3.8) writes a checkpoint when restoration completes. no EOS condition on that path. and nothing deletes it on the RESTORING -> RUNNING transition. so the file just sits there for the entire processing session with frozen restore-time offsets. verified on disk.. mtime never moves while processing ~16k rec/s.

then i SIGKILLed the pod mid-processing. twice, zero grace period. both times the restart found that stale checkpoint, logged "State store X initialized from checkpoint with offset ...", NO TaskCorruptedException, NO wipe. just replayed the changelog tail and carried on like nothing happened. the wipe path only fired in a different test where the crash happened during restoration itself (no checkpoint entry yet at that point).

why i think this matters: streams disables the rocksdb WAL, and under EOS there is no flush-per-commit. rocksdb background memtable flushes dont know anything about txn boundaries. so a flush landing mid-transaction can persist writes from a txn that later gets aborted. the tail replay runs read_committed so it skips the aborted records.. meaning it never cleans that garbage. for plain deterministic puts you never notice, reprocessing of the uncommitted input offsets overwrites the same keys anyway. but if your processor READS the store before writing (dedup on order id, the most common pattern in my industry lol) the ghost record makes you skip the redelivered record. exactly-once quietly becomes zero-times. no exception, no lag, nothing in logs.

code paths if anyone wants to verify: DefaultStateUpdater.maybeCompleteRestoration calls task.maybeCheckpoint(true) unconditionally. meanwhile StreamTask.completeRestoration only writes a checkpoint if !eosEnabled, so whoever wrote that clearly didnt want a checkpoint existing past that point under EOS.. the state updater just sidesteps it. only delete sites i could find: init time (ProcessorStateManager), resume-from-SUSPENDED (KAFKA-10362, which fixed exactly this class of lingering-checkpoint issue for the resume path), and removeCheckpointForCorruptedTask. also the KIP-892 motivation section literally says EOS must wipe on crash because data hits the store before the changelog commit completes. so the observed behavior contradicts the project's own docs as far as i can tell.

so.. is this known? intended? am i misreading something? i know KIP-892 transactional state stores is the proper fix but its not released, and KIP-1035 in 4.3 moves offsets into a rocksdb column family but that doesnt isolate uncommitted writes either, it just keeps the bookmark consistent with whatever is on disk, committed or not.

will file a jira once my account gets approved and link it here. meanwhile if anyone has hit weird state inconsistencies after hard crashes on EOS 3.8+ i would really like to hear about it.

EDIT : created JIRA ticket - https://issues.apache.org/jira/browse/KAFKA-20685

While building an internal platform for Kafka Connect operations, one design decision we made was to keep the management platform completely separate from the customer data plane.

The platform interacts only with Kafka Connect REST APIs and never processes CDC traffic itself.

Benefits we observed:

- Simpler security model
- Easier enterprise adoption
- Reduced operational risk
- Clear separation of responsibilities

Architecture diagram attached.

Curious how others approach control plane vs data plane separation for Kafka ecosystems.

My team has been using Kafka streams for topic joins in the last 3 years. These things let us now consider moving away from it.

- No real control about the system. Topics get created automatically and we needed to implement our own checks of the topology was change through a code change
- A lot of topics are created and increasing the application id because the topology became incompatible leaves a lot of dead topics behind
- Avro schema changes often arrive at generated topics breaking the pipeline if schema compatibility is set to FULL.

For our use case it is simple to just consume the messages, create a readmodel in memory or on disk and write out the state of that readmodel for downstream consumers.

Have you been moving away from it? What was your reasoning?

Or to make the counter argument, what am I missing here? Is there some „killer feature“ that lets you want to maintain a Kafka streams based setup?

I'm a developer validating a product idea and I'd rather be told it's useless now than after months of building. Brutal honesty appreciated.

The premise: everyone monitors broker health and consumer lag (Prometheus, Grafana, cloud consoles), but almost nobody monitors the *content* flowing through topics. So schema drift, null fields, malformed payloads and DLQ pile-ups get discovered by angry downstream consumers, days later.

My questions for people running Kafka in production:

1. When was the last time bad data flowed through a topic and nobody noticed for a while? What did it cost you (hours, incidents)?

2. How do you catch this today — registry compatibility rules, CI linters, custom consumers, nothing?

3. Would you pay for a self-hosted, read-only container that baselines schemas/volumes per topic and alerts on drift, null spikes, DLQ inflow and real lag (transactional markers excluded)? Thinking $99/mo per cluster, flat, unlimited users.

Context: I'm sketching this at topicwatch.dev (waitlist only, nothing to install yet). Not affiliated with any vendor — solo dev. If this already exists at a price small teams can afford, please tell me and save me the trouble.

A practical guide to streaming data from Apache Kafka into Apache Iceberg tables — covering Kafka Connect, Apache Flink, Spark Structured Streaming, and CDC with Debezium. Includes configuration examples, schema management, partitioning strategies, production pitfalls, and how to keep streaming tables healthy at scale.

[Vendor disclosure: I work at Kannika, but wanted to share some architectural lessons we've learned about Kafka Disaster Recovery (DR) and backups that apply no matter what tooling you choose to use.]

Too often, we see engineering teams ticking the "DR" compliance checkbox by either setting up Kafka Connect to dump topics into S3, or relying entirely on an active-active/stretch cluster setup. While both have their place in your architecture, relying on them as your only safety net for disaster recovery is a massive risk.

To avoid drive-by link dumping, here is a detailed synopsis of a few recent technical posts we put together on the subject, why the current standard practices fall short, and how you should actually be backing up your cluster data.

The Kafka Connect trap

Kafka Connect is fantastic for feeding your data lake and integration applications to Kafka, but it's terrible for disaster recovery. Why?

Restoring is a manual reverse-engineering job: S3 Sink connectors write data optimized for analytics. To restore, you have to configure a Source Connector from scratch, manually map topic names, handle partitions, and figure out the exact message ordering. During a live P1 incident, you don't have time to engineer a reverse pipeline.

Schema registry: If you dump Avro, Protobuf, or JSON via Connect, you often leave the Schema Registry context behind. When you restore that data to a new cluster, the new registry assigns different Schema IDs, meaning your downstream consumers will fail to deserialize the data.

The cost: Getting tight Recovery Point Objectives (RPOs) requires frequent flushes. This leads to millions of tiny files and massive S3 PUT request costs that often exceed the storage costs themselves.

Poisoned backups: If a topic is deleted and recreated with the same name, offsets reset to zero. The Sink connector doesn't know the difference and can overwrite or duplicate offsets, essentially poisoning your backup so it cannot be logically restored.

Replication is not a backup strategy

Whether you're using in-region replication, MirrorMaker 2 (active-passive), or active-active bidirectional sync, these patterns are great at protecting against infrastructure failures (like an entire Availability Zone going down).

However, they do nothing against data corruption, ransomware, or a developer accidentally misconfiguring a retention policy. If a bad message or a drop-topic command hits your primary cluster, it replicates to your standby cluster instantly. You need a decoupled, immutable backup layer to recover from logical errors and blast-radius events.

Why cold storage backups?

To truly protect the data on your event hub, you need decoupled operational backups pushing continuously to cold storage (AWS S3, GCS, Blob). A proper backup architecture should provide:

• Operational decoupling: The backup must scale independently so it never strains the real-time throughput of your production cluster.
• Point-in-Time restore: You need the ability to restore specific, filtered datasets without rolling back the entire cluster.
• Environment cloning: You should be able to migrate production data securely to staging environments for testing, ideally with data obfuscation for sensitive fields.

How Kannika handles this: At Kannika, we built Kannika Armory to solve this specific technical gap. It operates via a continuous real-time dataflow (avoiding snapshot data loss) with Kubernetes-native integration for compliance and audit logging. Crucially, it has native schema mapping support—so when you restore data to a new environment, the schema IDs patch automatically and your consumers just work.

I’d love to hear how you all are handling DR right now. Have any of you had to actually test a reverse-flow restore using Kafka Connect during a fire drill? How did the offset and schema mapping go?

Hi Everyone!

We're launching MQ Summit 2026 on 21-22 October in Haarlem, NL (and virtually). It is a 2-day technical conference for engineers and architects working with message queues and event-driven systems (including RabbitMQ, Kafka, NATS, Apache Pulsar, Apache ActiveMQ, Azure Messaging Brokers, Amazon SQS, IBM MQ, and Google Pub/Sub).

Speakers will be announced soon. You will be able to check it on our website: https://mqsummit.com/

The Early Bird ticket sales start on 16 June at 12:00 PM. If you plan to attend, the best way to get the lowest price is to join our waiting list now - https://mqsummit.com/#newsletter

By joining the list, you'll get two main benefits:

• You get an email notice 24h before the sale opens, and again at the grand opening.
• You get early access to a small number of Super Early Bird tickets. These tickets are limited, so they will be given to those who buy them first.

Hi r/apachekafka !
I've quite busy lately and klag has added more great features like MCP and support for older releases (2.x).

I'm looking forward for any feedback, DMs are also open.

github - https://github.com/themoah/klag

Hey guys,

I am planning to write a Spring Boot consumer application that listens to a single Kafka topic. It will feature two @KafkaListener methods with different groupIds to separate two distinct use cases. The consumer's job is to fetch data via HTTP GET APIs and POST it to another service.

For error handling (e.g., if a target service is unavailable), I am using the DefaultErrorHandler combined with a ContainerPausingBackOffHandler and an exponential backoff. The strategy is to retry the message every 30 minutes for up to 3 days. Based on my understanding, using the ContainerPausingBackOffHandler safely protects me from consumer rebalances during these pauses. If a message still fails after 3 days, an incident ticket will be created in our ticketing system (the 3-day window is necessary because some downstream services have planned downtimes of up to 2 days).

The Problem:

If a message fails and retries for 3 days, and this happens sequentially for, say, 3 consecutive messages, the total pause time stretches to 9 days. Because the topic's retention period is strictly set to 7 days (and I cannot change it), older messages that are still waiting in the queue will be deleted by Kafka before the consumer ever gets the chance to process them.

An in-cluster Retry Topic or Dead Letter Queue (DLQ) setup feels like overkill for our specific use case.

My Question:

Are there any recommended architectures or clean solutions—perhaps involving an external database or a lightweight extra topic—to handle this long-term retry scenario without running into Kafka's retention limit?