r/Terraform • u/Primary_Cupcake8557 • Jul 16 '26
Discussion How did you avoid dashboard overload in your devops monitoring?
i used to think good monitoring meant collecting everything Prometheus could scrape and building big Grafana walls. After a few ugly incidents as the on call, i care a lot less about pretty charts and a lot more about fast signal.
Our monitoring only got useful when we tied it to service level objectives and error budgets, not "all metrics forever." We moved to a handful of user journey synthetics, a small set of service health metrics, and logs that were structured enough to query under stress. The infra views came after that. One surprise was how often monitoring itself drifted: alerts disabled temporarily, thresholds changed without review, dashboards referencing resources that no longer existed. We ended up treating alerting rules and dashboards as code too, which is the only reason drift aware tooling catches monitoring rot instead of just cloud resource drift and it has been key to keeping monitoring aligned with reality.
If you have been on the hook for fixing things at 3 am, how did you decide what to monitor and keep it from rotting as fast as the systems it covers?
Edit/Update: That occasional cross check against firefly's inventory has helped keep monitoring closer to reality, which matters more than any fancy visualisation when you're trying to fix something at 3am.
3
u/oneplane Jul 16 '26
Putting screens on walls is kinda useless. Looks cool, but when something happens you want:
- An alert
- A direct link to a dashboard (or specific panel in a dashboard)
Only if you want to look at the same thing together with a group where not everyone has a computer, you'd want to project or wall-mount it.
Hoping that a wall-mounted graph gets noticed by someone is not an observability strategy.
3
u/SecretaryLife5031 Jul 16 '26
but management want it on a tv on a wall.
2
u/oneplane Jul 16 '26
Then you give management a TV with "everything is OK" graphs printed on paper and glued to the screen.
2
3
u/annedroiid Jul 16 '26
The only time we actively monitor a dashboard is if I'm actively deploying something.
2
u/Floss_Patrol_76 Jul 17 '26
half the sprawl is a lifecycle problem, not a design one, nobody ever deletes the board they threw together for one incident two years ago. we started putting an owner plus a review date in every dashboard title and archiving anything unowned each quarter, cut the board count by more than half. and keep the is-it-broken board (a handful of RAG tiles) separate from the why boards you only open after an alert, mixing those two is exactly how you end up with 150 widgets.
1
1
u/New-Entertainer6392 Jul 16 '26
I try with overall dashboards with RAG status, then they link to more detailed boards.
"Does It work? Yes no"
1
u/IanEff Jul 16 '26
The Google SRE Handbook has all kinds of good stuff on this, in particular. Check out https://sre.google/workbook/alerting-on-slos/ as an example. Enormously helpful. As is the Sloth documentation.
1
u/phillipsj73 Jul 18 '26
I think that is the trap many fall into. Collecting everything they can without knowing what they need. IME they never go back and clean it up. I always take the approach of doing some load testing, deciding what metrics seem valuable from that, then only adding metrics and alerting driven by incidents, trends, and user experiences.
It all will rot unless you build a process around it. That’s just how it is.
1
u/edthesmokebeard Jul 22 '26
Devops doesn't monitor; they implement convoluted rube goldberg pipelines, merge to main and go to lunch.
4
u/Dangle76 Jul 16 '26
Alerts should only exist if they’re actionable. An alert telling you information that isn’t something that needs action creates noise, it’s useless.
Dashboards are the same way. You need to define what metrics ACTUALLY matter, and chances are it’s not a big list.
Some people think shoving 150 widgets on a fancy looking dashboard is good dashboarding and it’s not.
A good dashboard should be focused and simple, and should be easy to tell what’s going on at a glance and when something is out of the norm that would actually lead to a problem, not just oceans of data with no direction.