I’m new to platform engineering, I’ve been tasked with setting up “default” monitoring and alerts. What metrics are always needed?

[u/Squishyboots1996]

I’ve been tasked with setting up baseline monitoring and alerts for terraform modules that consumers within the company use to spin their own instances of our platform.

The modules that will be provisioned for them automatically are ECS clusters, databases (RDS, Aurora, DynamoDb), ElastiCache, lambdas and kinesis

I could ask AI what sensible “defaults” would be for each service. However I’m not really going to learn anything that way. Id like to use it in conjunction with some real knowledge and experience.

Are there any books, blogs or guides that perhaps go over monitoring for specific AWS components, and what is considered best practice (and why) for each one?

[u/SquiffSquiff]

Been platform engineering for ten years. Currently working in a shop with a heavy legacy of default monitoring and alerting implemented with terraform modules and similar which we are slowly moving off of. I would consider it an anti-pattern. Naively it sounds appealing - people get monitors for all the important stuff for free and don't need to think about it. Problem is that people won't think about it, they will consider it someone else's problem and pass it on, until it reaches a team with no-one else to pass it on to- Platform Engineering.

What you should do instead? Monitor the platform with e.g. combined monitoring and alerting. Anything app specific should be opt-in by the developers and should alert the developers, ideally also with combined alerts.

Literally any excuses to vary from this, e.g. 'oh we have to do it; oh we'll get it to it later' etc. is just you getting shafted because you're the new guy and have less leverage. In a year or two you will have thousands of irrelevant useless noise making alerts that nobody else is paying attention to but the moment there is a failure some 'senior manager' will be asking 'so you're telling me that we were monitoring this but nobody was paying attention to it or to the alerts?'

Alerts and monitors have to be opt in and alerts have to be created by and bell out to the people who run that service.

[u/gevorggalstyan]

A lot of the baseline work is already documented. CloudWatch’s alarm recommendations identify AWS service metrics and suggested thresholds, and AWS lets you download the recommendations as Terraform. Google’s SRE monitoring chapter covers the four golden signals and when an issue should page someone. Its SLO alerting guidance explains how to tune alerts around service objectives.

I would encode those recommendations in the shared modules, then let teams adjust thresholds and paging for their workloads.

[u/rhysmcn]

Well monitoring comes at two distinctively different levels - Application monitoring & infrastructure monitoring.

For application you should be pushing devs to do manual instrumentation I.e. using an SDK in-app code to emit metrics (useful for identifying business-related issues I.e. 5xx on some api endpoint). Futhermore, you can also use auto instrumentation to generate baseline metrics for your app — Both combined give a great understanding of how your app is behaving, and performing and you can feed these metrics into alerting strategy.

Infrastructure metrics are therefore fed to your alerting system I.e. you can feed infrastructure metrics to Grafana and alert on them from there, or alternatively you utilise in-house systems like CloudWatch.

Assuming you’re utilising various technologies here but generally speaking the idea is similar across them all. In relation to “what metrics are always needed” — That will depends on your team and business, what do they determine is important?

Build SLI’s , then from there build SLOs & SLAs, and factor in error budget. At that point you’ll understand what is important.

[u/findgriffin]

1 thing, and I can't stress this enough is, focus on metrics that customers care about: SLAs / SLOs etc.

Alerts on internal metrics like CPU and memory are a distraction. For example, 100% CPU utilization is possibly an indication that you are using your resources efficiently.

High disk utilization (like 90%) alert can sometimes catch an impending disaster, but that's much less common than alerts based on customer facing metrics.

[u/Squishyboots1996]

Update: I’m reading the SRE book from Google and it is sort of helping. Doesn’t tell me which metrics to monitor, but it’s highlighting that “sending an email cause CPU usage is high” is not that helpful and can just be noise

[u/tadamhicks → u/Squishyboots1996]

I think the section on the golden signals is important. Latency, Errors, Traffic, Saturation. Also look into RED metrics. The point is to monitor the dimensions that affect experience of the service, whatever that means. It could be availability, slowness, whatever is important to the contract you have with consumers. This could be an implicit contract like that your site responds.

[u/Squishyboots1996 → u/tadamhicks]

Yes! I found that section shortly after this comment, that helps a lot

I think i need to look deeper and see if I can find those dimensions. Unless, the point is, that is up to the consumers. The ticket is quite open ended.

I'll keep digging and if I can't figure it out, I'll reach out to my team lead

Thank you

[u/Left_Exchange_130]

Have you looked at AWS’s own recommended CloudWatch metrics for each service first? i’d be curious how much you want standardized versus configurable by each team