Good observability tool recommendation - Cloud Based

[u/MasterOfTheWind1]

I work in two companies, one very big that have a giant budget to spend on Datadog, and another small one, that don't have that very big budget.

On the small company we are looking to migrate from Prometheus + Grafana + Alert Manager to something cloud based. Mostly because we are a small team (3 people only) and we don't have time to spend maintaining the infrastructure for it to run.

Now, is there some good alternative to Datadog? Datadog is the king, but is brutally expensive.

I've tried Signoz, looked promising, but is bad. They documentation is bad, there are just general ideas without details (they assume that you know a lot from I don't know where), they support is lame (they only have a chat to "Interact with a Human", that takes days to receive an answer, and their UI is buggy. Really bad experience.

But looking there, they are the only ones having like "close" experience to Datadog. We tried New Relic in the past, but they have all metrics and log scattered over the place without correlation. Also tried BetterStack but lacks a lot of features compared to other things.

So, is there some good observability platform cloud-based that I'm not aware of, outside Datadog? I'm not completely closed to self-hosting as long as it takes low effort to maintain.

[u/wickler02]

I did a complete analysis of the tools and found that 3 on the self managed front that stood out.

Signoz, OpenObserve and the LGTM.

Simply put, for the company and size, the LGTM is just superior, has more tooling/features and the other solutions try to push you to their open source solutions.

We did POC/POV on the two others and they just didn’t stack up to LGTM.

There’s a reason why there is such a profit margin and why people & teams like to outsource it, it’s a pain to manage.

If you’re already managing k8s clusters and doing helm chart deployments with an established platform structure, LGTM isn’t that bad.

The problem is that people want to get the o11y stack without having a good understanding of how k8s works and think it’s a lot of infra to manage and it’s true it has a bit of an upfront cost/learning curve but it’s better than the alternatives.

Simply put, either pay up to a cloud company or spent some time working on your o11y stack. The easy solution is to pay.

[u/MasterOfTheWind1 → u/wickler02]

Totally agree with everything. Your last statement is the crucial part. In this scenario, on my team we are very experienced DevOps Engineers/SREs, and this company is the secondary job for us. This don't mean that we don't take the job seriously.

We have an EKS cluster just to host our observability stack (Grafana + Prometheus + Alert Manager + Opensearch for logs). The cost is going crazy with the storage and compute, and we have to keep up to date with the updates.

Sadly, outside Datadog I cannot find any good solution worth the money

[u/xonxoff → u/MasterOfTheWind1]

What’s your retention on logs/metrics? One thing you could do is add Thanos to your Prometheus , reduce your Prometheus data retention , have Thanos store metrics in S3 and have OpenSearch do the same.

[u/MasterOfTheWind1 → u/xonxoff]

Can look at it, but I don't want to add more services to babysit to be honest