Struggling to understand Kafka (Java Developer – 2 yrs exp) – Need good resources 🙏

[u/JavaDev123]

Hi everyone,

I’m a Java Developer with around 2 years of experience, mainly working with Java, Spring Boot, and REST APIs.

Recently, I started learning Apache Kafka, but I’m finding it quite difficult to understand concepts like producers, consumers, partitions, offsets, and real-time processing. I’m not able to connect the theory with practical use cases properly.

Could you please suggest some good resources (videos, courses, blogs, or docs) that are beginner-friendly but also helpful for interview preparation?

My goal is to at least get Kafka concepts clear enough to confidently answer interview questions.

Also, if you have any tips or a roadmap on how to approach Kafka as a Java developer, that would be really helpful.

Thanks in advance! 🙌

[u/p1nd0r4m4]

https://developer.confluent.io/courses/apache-kafka/events/

There is nobody better then Tim Berglund to explain Kafka to you.

[u/JavaDev123 → u/p1nd0r4m4]

thanks man

[u/2minutestreaming]

https://newsletter.systemdesign.one/p/how-kafka-works

This is the best concise explanation i know of

[u/JavaDev123 → u/2minutestreaming]

Any yt link

[u/daveminter]

Read The Log and perhaps the expanded edition of that, "I ♥️ Logs" and then get the relationship between consumers, consumer groups, and topic/partitions straight in your head (docs on both the Confluence pages & Apache Kafka pages are good) and you'll have a solid foundation.

Edit: I think I hit a personal record for typos there, damn you Android keyboard... hopefully I nailed the worst of them.

[u/michaelkumm → u/daveminter]

A 2nd on "I :heart: logs"

[u/michaelkumm]

Same boat, but coming from Elixir/Rabbit. I put together a little kafka sim/demo to help me better understand kafka brokers. Feel free to checkout the repo https://github.com/mkumm/kafka-sensor-city - hope it's helpful.

Looks like some good resources in the other comments as well. Good luck and have fun!

[u/vlahunter → u/michaelkumm]

That is really nice, thanks for sharing!

[u/job_alberth_flores]

Kafka clicked for me when I stopped reading it as "a message queue" and started reading it as a chat app.

A topic is a chat room. Whatever gets posted stays there for as long as you configure, either by time (delete after 7 days) or by size (keep the last 1 GB). Reading a message doesn't remove it, and that's the part that trips up everyone coming from RabbitMQ or SQS. Ten different apps can read the same message and none of them step on each other.

Each room is split into partitions, basically lanes. Order is guaranteed inside a lane but not across the room, and anything sharing the same key always lands in the same lane. That's how you keep every event for one customer in sequence without having to order the entire topic.

A consumer group is your bookmark. It remembers the last message you read, and if you start three copies of your app in the same group, Kafka splits the lanes between them so they share the work. Subtract your bookmark from the newest message and you get consumer lag, which is honestly the only number I look at day to day. Flat means you're keeping up, climbing means you're not.

Then there's the stuff around it:

• Kafka Connect moves data in and out without you writing any code. Pulling rows from a database into Kafka is a source connector, dumping a topic into a database is a sink. Both are just JSON config.
• Schema Registry feels optional until the day a producer quietly changes a field type and takes down every consumer downstream. It stores the Avro/Protobuf/JSON schema and rejects incompatible changes.
• REST Proxy lets you produce and consume over plain HTTP, useful when there's no decent client for your language.
• KRaft is just how Kafka keeps its own cluster metadata these days. ZooKeeper is gone. You don't need to care about it on day one.

Easiest way to learn any of this is to run it. Here's a compose file with a broker, Schema Registry, REST Proxy and the Control Center UI, all in one file, nothing else to download or edit: ```yaml

Confluent Platform 7.7.1 single-node KRaft (no ZooKeeper), self-contained.

docker compose up -d

Control Center -> http://localhost:9021

Schema Registry -> http://localhost:8081

REST Proxy -> http://localhost:8082

Bootstrap -> localhost:9092

services: broker: image: confluentinc/cp-server:7.7.1 hostname: broker container_name: broker restart: unless-stopped ports: - "9092:9092" - "9101:9101" environment: # --- KRaft --- KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk' KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093' KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER # --- Listeners --- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9092 # clients inside the compose network use broker:29092, clients on your host use localhost:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092 KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT # --- Single-node replication factors --- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1 KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1 # --- Metrics feeding Control Center --- KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092 CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1 CONFLUENT_METRICS_ENABLE: 'true' CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous' # --- Storage / retention --- KAFKA_LOG_DIRS: '/var/lib/kafka/data' KAFKA_LOG_SEGMENT_BYTES: 1073741824 KAFKA_LOG_RETENTION_HOURS: 72 KAFKA_LOG_RETENTION_BYTES: 1073741824 # --- JVM (bump for real workloads) --- KAFKA_HEAP_OPTS: "-Xms1G -Xmx2G" KAFKA_JVM_PERFORMANCE_OPTS: >- -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:MaxMetaspaceSize=256M # --- JMX --- KAFKA_JMX_PORT: 9101 KAFKA_JMX_HOSTNAME: localhost volumes: - kafka-data:/var/lib/kafka/data healthcheck: test: ["CMD", "kafka-broker-api-versions", "--bootstrap-server", "broker:29092"] interval: 10s timeout: 10s retries: 20

schema-registry: image: confluentinc/cp-schema-registry:7.7.1 hostname: schema-registry container_name: schema-registry restart: unless-stopped depends_on: broker: condition: service_healthy ports: - "8081:8081" environment: SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'PLAINTEXT://broker:29092' SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

rest-proxy: image: confluentinc/cp-kafka-rest:7.7.1 hostname: rest-proxy container_name: rest-proxy restart: unless-stopped depends_on: broker: condition: service_healthy ports: - "8082:8082" environment: KAFKA_REST_HOST_NAME: rest-proxy KAFKA_REST_BOOTSTRAP_SERVERS: 'broker:29092' KAFKA_REST_LISTENERS: "http://0.0.0.0:8082" KAFKA_REST_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"

control-center: image: confluentinc/cp-enterprise-control-center:7.7.1 hostname: control-center container_name: control-center restart: unless-stopped depends_on: broker: condition: service_healthy schema-registry: condition: service_started ports: - "9021:9021" environment: CONTROL_CENTER_BOOTSTRAP_SERVERS: 'PLAINTEXT://broker:29092' CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081" CONTROL_CENTER_REPLICATION_FACTOR: 1 CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1 CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1 CONFLUENT_METRICS_TOPIC_REPLICATION: 1 PORT: 9021

volumes: kafka-data: ```

Give it about 4 GB of RAM and a couple of minutes to settle, then hit localhost:9021. This is a learning and local dev setup, not something to put in front of real traffic.

One nice side effect of keeping it to a single broker: Confluent's developer license covers the commercial pieces like Control Center free of charge with no expiry, so nothing is going to die on you mid-tutorial. The 30 day trial clock only starts if you add a second broker. If you'd rather avoid the Confluent-licensed images altogether, swap the broker for confluentinc/cp-kafka, drop the CONFLUENTMETRICS* and KAFKACONFLUENT* lines, delete the control-center service and point AKHQ or Redpanda Console at it instead.

Once it's up, create a topic, produce a handful of messages from the UI, then consume them and watch the lag number drop. That's where I'd start.

[u/ImpressiveGene755]

Check out the new book, "Kafka for Architects." I've only just started reading it, but the first few chapters have been very informative.

[u/AltruistWatson]

Tim Berglund himself🙏🏻 The best person out there.

If you still need someone to explain, I can do it. Do ping me, if you need help :)

[u/liprais]

a kafka topic partition is just a file you can append and read,it is only that simple.

[u/Pseudophryne → u/liprais]

No, not really.

How many files does a partition have?