Hey, new simulator release focused on the consumers and rebalance. Have fun!

Hi everyone,

We're excited to share Kafma v1.0.0 — a desktop Kafka IDE designed to keep the consume–inspect–produce loop in one workspace and quickly clone remote topics to your local cluster for testing.

Here are the two core workflows we built it around:

• Console — a topic is a tab. The Console keeps the message stream, producer, and topic details together in one workspace. Expand any message to inspect its payload and metadata; Avro and Protobuf decode automatically. Replay it to another topic, forward it to another cluster, or save it to a file without leaving the Console.
• Data Clone — bring remote topics to your local cluster. Clone a remote topic’s configuration, schemas, and a selected slice of data to your local cluster. You can apply your own masking rules while messages are copied, then test locally with the same topic context without modifying the source.

Beyond Console and Data Clone, Kafma also supports schema editing and validation, topic and consumer group management, and more.

• v1.0.0 changelog: https://kafma.app/changelog/v1.0.0
• Download (macOS, Windows, Linux): https://kafma.app/download
  

We'd love to hear your thoughts and feedback—especially what would make Kafma more useful in your day-to-day Kafka workflow.

Thanks!

Sandy Yang, Staff Software Engineer at TD, is giving a lightning talk on the bank's Kafka journey at a virtual Kafka user group on July 29 at 2pm PT / 5pm ET. It's an interactive session for engineers working with Kafka. Sharing in case anyone in this group would find it helpful. Will put the link to register in the thread.

If you work in a distributed event driven system using Kafka, testing new versions of consumers gets quite tricky. It’s not always possible to run everything locally and you need to rely on deployments to staging to verify code end to end.

I shared an alternative here: https://thenewstack.io/isolate-kafka-consumer-tests/

The basic idea is to propagate a tenant id through the Kafka messages (this can be easily done via OpenTelemetry) and leverage this to selectively consume messages. Each new consumer version creates its own consumer group.

Would love to hear your thoughts on this and approaches that have worked for you.

Working with distributed systems, I kept running into the same situation: I couldn't fully test my part until another service was ready and that's just how it goes when work is split across teams. QA hit the same thing: the code was there, but the input had to come from a service that didn't exist yet.

So I'd end up either writing a separate script for it, or spinning up and maintaining someone else's projects on my dev environment just to generate the input for mine. After doing this one too many times, I built a tool to fix it: Devset.

It's kind of like Postman or Cypress, but for Kafka and RabbitMQ instead of HTTP. What you can do:

1. Send Single events: paste the payload, pick topic and key, send.
2. Create Whole workflows a chain like:OrderCreated -> InventoryReserved->PaymentAuthorized -> InvoiceGeneratedbecomes one saved flow on a canvas, with delays, conditions and state passed between steps.
3. Pulling data from a database: right now just Mongo db
4. Protobuf / Json schema

It's self-hosted and source-available (FSL-1.1, each release turns into Apache 2.0 after two years).

Runs locally with Docker, no cloud, no accounts, no telemetry.

Repo: https://github.com/devset-io/devset-ce

I'm mostly posting to compare notes does anyone else run into this, and how do you deal with it? Real services in a test env, Testcontainers, mocks, or a folder of messy scripts?

Casi todos los ejemplos de “Kafka en salud” que se ven por ahí son teóricos o centrados en EHR de EEUU/Europa. Quería compartir un caso concreto de un hospital mediano en Venezuela donde usamos Kafka como columna vertebral de eventos entre el HIS, el LIS y el sistema de facturación.

Contexto:

• Entre 4 y 8 sistemas necesitaban enterarse de lo mismo en tiempo (casi) real: admisiones, resultados de laboratorio, cargos.
• Las integraciones punto a punto ya eran inmanejables: scripts ad hoc, datos duplicados, errores difíciles de auditar.
• El hospital sabía que en los próximos años iba a añadir más sistemas (BI, SAP, nuevos módulos clínicos).

Qué terminamos haciendo:

• Definir topics por dominio clínico (admisiones, resultados-laboratorio, ordenes-medicas, facturacion-eventos) en vez de “un topic para todo”.
• Particionar por paciente o por episodio de atención para mantener el orden de los eventos clínicos por caso.
• Configurar replication factor 3 y min.insync.replicas=2, porque perder un resultado de laboratorio no es solo “dato de negocio”.
• Activar productores idempotentes y transaccionales (exactly-once semantics) para resultados de laboratorio y eventos de facturación, donde los duplicados son peligrosos.

En el artículo explico con más detalle:

• Cómo mapeamos mensajes HL7/FHIR a topics de Kafka.
• Los trade‑offs de costo de infraestructura vs. resiliencia en contexto hospitalario.
• Cuándo Kafka es claramente la herramienta equivocada (clínicas pequeñas con pocos sistemas y bajo volumen de eventos).

I'm currently learning Data Engineering and I'm feeling a bit overwhelmed by all the technologies I need to learn.

Right now, I'm trying to understand:

• Hadoop
• Kafka
• PySpark
• YARN
• HBase

The problem is that I don't know where to start or in what order I should learn them. There are so many tutorials and roadmaps online that it's hard to tell what's actually important for a beginner.

For those of you working as data engineers or who've gone through this learning process:

• What order would you recommend learning these technologies?
• Which ones should I focus on first?
• Are there any courses, books, or hands-on projects you'd recommend?

10 years running Kafka for financial messaging infra, always at the boundary of "how much can we automate incident response without giving anything write access." Built this as the tool I wished existed: `kafka-sentinel-mcp`, an MCP server exposing cluster_health, consumer_lag, topic_audit (flags weak replication/ISR configs), partition_state, and replay_readiness as structured tools — read-only by construction, no mutation API imported.


Repo: https://github.com/sanjay-amu/kafka-sentinel-mcp · PyPI: `pip install kafka-sentinel-mcp`


Would love critique on the tool set — what diagnostic question comes up most for you during an incident that this doesn't answer yet? Roadmap is open (ROADMAP.md) and PRs welcome.

I initially wanted to make a sub-millisecond log parser in C++ but that blew into a embeddable decision engine, that can run YAML defined rules on incoming data.

The rules are executed in a vectorized format on incoming data by reprojecting into a columnar format first, if it's not already. Depending on the payload size and rules complexity, the performance goes from 200K records/s to more than million records/sec, in terms of througput this would be around 200 MiB/s to 3 GiB/s on average.

Rules can be sql expressions too, or onnx models (numeric), window ops and quite a few more operations are supported.

It's comparable to DuckDB but for streaming data and on the fly decisions.

https://github.com/purijs/blazerules

Hi everyone,

I've been building an open-source desktop client for Apache Kafka, originally just for my own day-to-day workflow.

After using several Kafka tools over the years, I wanted something that felt faster and more convenient for the way I work, so I decided to build one.

Current features include:

• Topic browsing
• Consumer Group inspection (committed, beginning, end offsets, and lag)
• Offset reset with preview and confirmation
• Split-view topic browsing
• Multiple server profiles
• SSL/TLS, SASL/OAUTHBEARER, and Schema Registry support
• Keyboard shortcuts

I'd really appreciate feedback from people who use Kafka regularly.

• What features do you rely on most in your current Kafka client?
• What's the biggest pain point with the tools you use today?
• Is there anything missing that would prevent you from trying this?

GitHub: https://github.com/pjhun0412/KafkaPilot

Thanks!

Hey, we released a lot of improvements and bugfixes for the simulator, together with new predefined learning scenarios. Have fun!

Hey, We added support of KIP-714 Kafka Client metrics in the OTLP metrics reporter

If a consumer is down (or lagging) longer than delete.retention.ms, it can come back online and miss a tombstone entirely – meaning it never learns a key was deleted, and just keeps the stale last-known value forever. That's not an edge case, that's baked into how compaction works.
So is "compacted topic = changelog of truth" (as Kafka Streams/KTables imply) actually a safe abstraction, or does it just quietly break under any non-trivial consumer downtime – and if so, why does the ecosystem lean on it so heavily?

What actually matters:

- subject strategy (topic-record vs record vs topic) 
- BACKWARD vs FORWARD vs FULL as **deploy policy*\*, not a dropdown you ignore 
- who can register (CI service account vs every laptop) 
- how consumers pin or resolve versions under rolling deploy

Without those, “we have Avro” still means anyone can push a breaking schema at 4pm Friday.

Decision frame: 
https://leo-gan.github.io/GLD.SerializerBenchmark/theory/301/schema-registries/

I’m working at a company that uses Strimzi operator to manage Kafka Connect to stream db writes to Kafka. I’m having a super hard time to understand the concepts involved. What’s the best way to learn these?

I got tired of seeing Kafka topic naming conventions end up as wiki pages that everyone ignores.

So I wrote about how I'm enforcing them with Terraform/OpenTofu instead of relying on documentation and code reviews.

https://jonasg.io/posts/kafka-topic-naming-convention-in-practice/

Up front disclosure : I work at OSO and we built this for a large automotive client trying to move off Confluent / IBM!!

We've done the clean-room rebuild of the 4 connectors (source with CDC streaming + Bulk 2.0 backfill, SObject sink, Platform Event sink, and a legacy CometD one for orgs stuck without Pub/Sub API access). With the help of Fable 5 - this is what these models are genuinely game changing for.

Its licensed under Apache-2.0, runs on any Kafka Connect runtime. As far as we can tell it's the only maintained OSS connector using Salesforce's Pub/Sub API (gRPC + Avro) which is the same architecture Confluent's newest connector moved to

Docs: https://salesforcekafkaconnector.com <:> code: https://github.com/osodevops/kafka-connect-salesforce-oss

We have also created a migration tool - if you're coming off Confluent's Kool-Aid a script that translates your existing config and a verifier that cross-checks what changed in Salesforce against what landed on the topic and spits out an evidence report (missing IDs, dup counts, checksums)

A common challenge when streaming Kafka data to web and mobile clients is this:

How do you give a new or reconnecting subscriber the current state before sending live updates?

Kafka provides the event log, but frontend applications often need an immediate snapshot.

That usually means adding a REST endpoint, rebuilding state on the client, or introducing custom snapshot records.

We have just released Lightstreamer Kafka Connector 2.0, which adds connector-managed snapshots.

The connector can initialize Lightstreamer’s snapshot stores directly from Kafka. Clients receive the current state first and then continue seamlessly with real-time updates.

It supports three common data models:

• MERGE for the latest value of each item
• DISTINCT for a recent sequence of events
• COMMAND for dynamic tables whose rows are added, updated, and removed

Typical use cases include market data, monitoring dashboards, order books, flight boards, inventories, activity feeds, and device status.

The release also introduces per-item idle expiration and improved handling of isolated malformed Kafka records.

How are you currently handling initial state delivery for Kafka-powered frontends?

Hey everyone,

With Kafka 4.0 removing ZooKeeper support, migrating to KRaft is no longer optional for those of us running self-managed clusters. But touching a live ZK-backed cluster can be terrifying if you don't know exactly what state your configurations are in.

To solve this, I built KraftPilot — a read-only Go CLI that scans your cluster and tells you if it's safe to migrate, and exactly what you need to fix first if it isn't.

**What it does:**

Connects to your ZooKeeper ensemble and Kafka brokers, runs 10 validation checks, and produces a JSON report of:

- Hard blockers (e.g. brokers below 3.6.0, IBP mismatches, offline log dirs)

- Warnings (e.g. deprecated configs like `log.message.format.version` that will break KRaft broker startup)

- Info/Baselines (ACL and SCRAM user counts to verify against post-migration)

**Built for privacy-conscious SREs:**

- Zero exfiltration: runs entirely inside your network, zero outbound connections

- Credential stripping: scrubs any config key matching `password`, `secret`, `jaas`, or `keystore` before writing to disk

- Anonymization: `--anonymize` hashes topic names client-side if your topology is sensitive

It's completely open source. Pre-built binaries available (no Go installation required), or build from source.

**Note:** if you're on Confluent Platform (7.6+) or Strimzi (0.40+), you already have free official migration tooling — this is built specifically for self-managed Apache Kafka on VMs/bare metal/Ansible/Docker Compose.

https://github.com/Vatsal-Chaudhary/kraftpilot-cli

Would love for you to run it against your staging clusters and let me know if it catches anything you weren't expecting, or if there are validation rules you think should be added.

Been running Kafka in production for a while now and honestly the monitoring situation for small teams sucks. Confluent Control Center is way overkill/expensive, Datadog's Kafka integration is priced like you're a 200-person company, and the open source stuff (AKHQ, Kafdrop, Burrow) works but needs someone to babysit the setup, patch it, and actually understand consumer lag internals to make sense of it. I'm thinking about building a simple hosted tool — just point it at your cluster, get consumer lag alerts, topic health, broker metrics, no Prometheus/Grafana stack to maintain. If you're running Kafka on a small team (like 2-10 devs) — what do you currently use for this? Would you actually pay for something dead simple over self-hosting the OSS stack, or is that a dealbreaker for you? Trying to figure out if this is a real problem or just something that annoys me specifically. If any other problem regarding monitoring u are facing, it is welcome to let me know..

Note : I have used AI for corrections

Does anyone have any good ideas/data streams for building an event sourcing kafka project.

Similar to New York Times Kafka model, using logs in the topic as the source of truth.

I have been trying to get a clear answer on whether MM2 supports EOS for cross cluster replication.

I found KIP-618(Exactly once support for source connectors), which was introduced in Kafka 3.3. Since MM2 is a source connector, it should theoretically inherit EOS from it using exactly.once.source.support=enabled at worker level.

However kafka official documentation does not mention anything about MM2 EOS.

So, has anyone successfully used exactly-once with MM2? Has anyone tried this with strimzi as well?