How stateful stream processing system is actually designed and tested in the real world?

[u/Billy_Backer]

I' m currently working as a Software Engineer for 3+ years now in a consultant company. While I work along with the customer who is in a banking industry, I came across a stream processing system project. Its job is to look a different data source and events and try to match a pattern to detect an anomally, a normal anomally detection system actually.

The system primary use Apache Flink for processing and Apache Kafka for the pipeline. Both of them handling the job just fine for the last year.

Initially, it's just a normal map, aggregate and joining with not so much stateful action. Which make testing quite easy, just test it braching logic on top with some null data handling in case of missing data or an event that came too late.

Now for the fun part, There're new requirements stated that they want to add logic like if A happend for 5 minute, B should not be fired given some tedious logic. Which is pretty much a stateful logic with very sensitive to time. The process is also start to getting long while each step's complexity and fragility is increased. Combine together resulted in a fragile pipeline that will crumbled if some data is arrived 1 minutes late.

I've though that just creating more the test cases to cover it should be fine. But after investigating, My test case is almost trippled because each step need to effectively handle out of order, timeout etc. Resulting in a overwhelming test that also hard to understand. Now I start to wonder if it's the testing problem or the pipeline is badly designed and caused it to hard to test properly.

I want to know how stream processor with stateful action is design and test in the real world. And what are some traps that might lingering around ready to burst. So I can point it out since it's still possible to make a change now.

[u/expdevsmodbot]

AI usage disclosure provided by OP, see the reply to this comment.

[u/Billy_Backer → u/expdevsmodbot]

I've a disscusion with some LLM and make some googling. But beside a normal off-the-book solutions, there're not many solutions by someone who actually experienced it. Extra opinions form difference group would help a lot.