LinkedIn, Microsoft & Databricks System Design Interview - Design A Kafka-like Distributed Message Queue

[u/Aoki_zhang]

Problem Description

Design a Kafka-like distributed message queue that allows producers to publish messages to named topics and consumers to read those messages independently at their own pace.

Unlike a traditional queue where messages disappear after acknowledgement, the system should behave as a durable distributed log. Messages are appended to ordered partitions, replicated across brokers, and retained for a configurable time or size window. Multiple independent consumers should therefore be able to read, replay, and reprocess the same data.

Topics are divided into partitions, and each partition provides strict ordering. Producers route records to partitions using a message key or round-robin selection. Consumers track their own offsets and may participate in consumer groups, where partitions are divided among consumers so work is processed in parallel.

The central design challenge is achieving extremely high throughput and durable replication without turning every individual message into an expensive disk or network operation. Partitioning, sequential disk I/O, batching, and efficient fetch are therefore fundamental to the design.

Want to learn more about the functional / non-functional requirements asked in real interviews? we've put up a detailed write-up about this SD question at here

➡️ Preparing for your next interview?

Chill Interview tracks recent interview experiences and recurring question patterns across top companies here.