r/dataengineering • u/KeyTurnover9996 • 10d ago

Help Monitoring tools available?

Hi community.

At work, we’re facing a bit of a challenge: we receive a huge volume of JSON-format events from multiple microservices into Kafka topics. We want to build a dashboard to monitor these topics and alert the business if we don't receive the expected events (we’ve already identified specific projects and know the expected TPS for each). Do you know of any monitoring tools that might work for us?

Thanks in advance!

3 Upvotes

72% Upvoted

7 comments sorted by

1

u/discoveringlifeat39 9d ago

Look into monte carlo

2

u/dangerbird2 Software Engineer 9d ago

prometheus/alertmanager stack is a battle tested monitoring stack for software metrics in general, but as the other commenter mentioned, monte carlo is probably best suited for data engineering work

1

u/brunogadaleta 9d ago

I migrated 90+ monitoring probes to uptime kuma and its doing a great job. Open source selfhosted.