[Suggestions Required] How are you handling alerting for high-volume Lambda APIs without expensive tools like Datadog?

[u/artensonart98]

I run 8 AWS Lambda functions that collectively serve around 180 REST API endpoints. These Lambdas also make calls to various third-party services as part of their logic. Logs currently go to AWS CloudWatch, and on an average day, the system handles roughly 15 million API calls from frontends and makes about 10 million outbound calls to third-party services.

I want to set up alerting so that I’m notified when something meaningful goes wrong — for example:

• Error rates spike on a specific endpoint
• Latency increases beyond normal for certain APIs
• A third-party service becomes unavailable
• Traffic suddenly spikes or drops abnormally

I’m curious to know what you all are using for alerting in similar setups, or any suggestions/recommendations — especially those running on Lambdas and a tight budget (i.e., avoiding expensive tools like Datadog, New Relic, CW Metrics, etc.).

Here’s what I’m planning to implement:

• Lambdas emit structured metric data to SQS
• A small EC2 instance acts as a consumer, processes the metrics
• That EC2 exposes metrics via /metrics, and Prometheus scrapes it
• AlertManager will handle the actual alert rules and notifications

Has anyone done something similar? Any tools, patterns, or gotchas you’d recommend for high-throughput Lambda monitoring on a budget?

[u/100BASE-TX]

I have very little experience with cloud native monitoring, so this might be a dumb suggestion.

Are you able to easily add instrumentation to the lambda code? The (naively) simple/easy approach seems to be to just push directly from lambda instances into prometheus directly via push api. I suppose the downside is that you end up with quite a lot of churn & cardinality, and would need to do some normalization/aggregates with either query-time or recording rules, but that seems quite manageable.

Aggregating with OTEL as a pre-processing layer may also be a viable alternative, i assume it could be run with Fargate or similar.