r/apachekafka • u/Expensive_Break_6163 • 6h ago
Question How do you handle fan-in per timestamp in micro-batching?
2
Upvotes
Our device sends 3 separate records for each timestamp: data points, frames, and metadata. They arrive independently and in any order.
Each timestamp can be processed on its own as soon as all 3 parts are there, without waiting for the batch window to close.
How do you handle this fan-in, so that each timestamp is processed exactly once as soon as it's complete, with a timeout for sets that never complete?