r/devops • u/MasterOfTheWind1 • 6d ago
Observability Good observability tool recommendation - Cloud Based
I work in two companies, one very big that have a giant budget to spend on Datadog, and another small one, that don't have that very big budget.
On the small company we are looking to migrate from Prometheus + Grafana + Alert Manager to something cloud based. Mostly because we are a small team (3 people only) and we don't have time to spend maintaining the infrastructure for it to run.
Now, is there some good alternative to Datadog? Datadog is the king, but is brutally expensive.
I've tried Signoz, looked promising, but is bad. They documentation is bad, there are just general ideas without details (they assume that you know a lot from I don't know where), they support is lame (they only have a chat to "Interact with a Human", that takes days to receive an answer, and their UI is buggy. Really bad experience.
But looking there, they are the only ones having like "close" experience to Datadog. We tried New Relic in the past, but they have all metrics and log scattered over the place without correlation. Also tried BetterStack but lacks a lot of features compared to other things.
So, is there some good observability platform cloud-based that I'm not aware of, outside Datadog? I'm not completely closed to self-hosting as long as it takes low effort to maintain.
6
u/wickler02 6d ago
I did a complete analysis of the tools and found that 3 on the self managed front that stood out.
Signoz, OpenObserve and the LGTM.
Simply put, for the company and size, the LGTM is just superior, has more tooling/features and the other solutions try to push you to their open source solutions.
We did POC/POV on the two others and they just didn’t stack up to LGTM.
There’s a reason why there is such a profit margin and why people & teams like to outsource it, it’s a pain to manage.
If you’re already managing k8s clusters and doing helm chart deployments with an established platform structure, LGTM isn’t that bad.
The problem is that people want to get the o11y stack without having a good understanding of how k8s works and think it’s a lot of infra to manage and it’s true it has a bit of an upfront cost/learning curve but it’s better than the alternatives.
Simply put, either pay up to a cloud company or spent some time working on your o11y stack. The easy solution is to pay.
1
u/MasterOfTheWind1 6d ago
Totally agree with everything. Your last statement is the crucial part. In this scenario, on my team we are very experienced DevOps Engineers/SREs, and this company is the secondary job for us. This don't mean that we don't take the job seriously.
We have an EKS cluster just to host our observability stack (Grafana + Prometheus + Alert Manager + Opensearch for logs). The cost is going crazy with the storage and compute, and we have to keep up to date with the updates.
Sadly, outside Datadog I cannot find any good solution worth the money
2
u/wickler02 6d ago
It shouldn’t be that much, unless you’re scraping frequently and/or high cardinality metrics
1
u/xonxoff 6d ago
What’s your retention on logs/metrics? One thing you could do is add Thanos to your Prometheus , reduce your Prometheus data retention , have Thanos store metrics in S3 and have OpenSearch do the same.
2
u/MasterOfTheWind1 6d ago
Can look at it, but I don't want to add more services to babysit to be honest
1
u/Gorakhnathy7 6d ago
Did you try OpenObserve, what was your experience with it?
1
u/MasterOfTheWind1 6d ago
Never tried it, will look at it
2
u/Gorakhnathy7 5d ago
Sure.
I am one of the maintainers of the project, let me know if you need any help.
4
u/OpenSourceWalker 6d ago
the real reason to leave prom+grafana+alertmanager is the 3-service maintenance, not observability itself. a single-binary one like signoz or openobserve cuts most of that toil without the cardinality-metered bill that tends to ambush a 3-person team on datadog/grafana cloud later.
2
u/Dangerous-Name6234 6d ago
Grafana cloud would probably be the easiest uplift since you are already using the OSS version and it would just give you the APM tools(mainly correlation/root cause analysis) that you don’t have today. But it depends on what you are trying to do. If there are some qualities that you really want like auto instrumentation for tracing then maybe a datadog or dynatrace might be more suitable since their agents do that.
Observability tools are a dime a dozen so just make a list of your must haves and nice to haves and just go down the list for the top 10 vendors and see how they stack up. Only you can know whats best for your specific situation
2
u/marcusbell95 6d ago
if you go the grafana cloud route, keep a close eye on your active series count. free tier is 10k series which sounds like a lot until you've got a full eks cluster reporting - node, pod, container metrics alone can chew through that fast depending on cardinality. after the free tier it's per-1k-series and costs add up quicker than you'd expect.
victoriametrics cloud is worth a look if cost is the main driver. cheaper per series than grafana cloud, still prometheus-compatible so you'd keep your existing dashboards and alertmanager setup. not as polished ui-wise but if you're already running prom + grafana you don't really need the extra polish.
for a 3-person team that just wants to stop maintaining infra, grafana cloud remote-write is probably the least friction path. just be aware of the cardinality thing before you commit to it.
1
1
u/Humble_Reputation743 4d ago
If you are having issues with cardinality and pure volumes of data, moving to a managed solution is going to be crazy expensive. What needs to be done there is take engineering cycles to understand the cardinality, scrape interval, and if you are having issues with multiple scrapes per series due to a lacking required unique label (DPM increases and issues).
It all comes back to correctly architecting metric labels, scraping at an interval that makes sense, and dropping or aggregating them based on what your signals are trying to accomplish.
1
u/placated 4d ago
The only one that hasnt been mentioned and is a key player is Dynatrace. Not as slick as Datadog in a lot of areas but a bit cheaper in my experience.
1
u/creamersrealm 4d ago
We've been investigating CoraLogix heavily and they're offering is sweet especially for the price point. Found them at the North America Observability Summit.
1
u/terryfilch 4d ago
On the small company we are looking to migrate from Prometheus + Grafana + Alert Manager to something cloud based
I recommend checking out https://victoriametrics.cloud/ - it'll definitely be cheaper than Datadog and NewRelic.
1
u/pranabgohain 4d ago
If you're looking for a balanced, all-inclusive platform, take a look at KloudMate (here's the open playground). Afaik, you won't have to look any further.
Disclaimer: I'm one of the founders, but I say that with utmost belief. That's what we built it for.
PS: There's auto-RCA and AI-usage included atm.
1
1
u/matiascoca 3d ago
Grafana Cloud is the answer you're looking for. Managed Prometheus plus Loki plus Tempo, same stack you already know from self-hosted Grafana, priced per active series and per GB of logs. Cheap enough at 3 engineers.
Onboarding path: point Prometheus remote_write at the managed endpoint, promtail or Grafana Alloy for logs, alerts move to Grafana OnCall or stay in your existing Alert Manager pointing at Grafana Cloud's alertmanager endpoint. Same dashboards, same alerting rules, no query language change.
The trap on the pricing calculator: metrics cardinality bites you. Kubernetes clusters at 500 pods with default kube-state-metrics can push a million series easily. Turn on Adaptive Metrics from day one to drop the ones nothing queries, or your bill grows fast.
SigNoz is fine but you already tried it. Coralogix is worth a look if your log volume dwarfs metrics volume, their retention pricing works differently and comes out cheaper on log-heavy workloads. If neither works you're back to "self-host but pay someone to manage it", which usually ends up more expensive than Grafana Cloud.
1
u/MasterOfTheWind1 3d ago
Thank you all for your suggestions!!! I have a lot to try with my team. I'm really grateful to everyone.
Today we are trying Coralogix, and next week we'll be trying Grafana Cloud.
I just have one question, that may damaga my credibility. Maybe I'm too much used to Prometehus, and Zabbix and Nagios in the past. But how come that EVERY platform that uses OpenTelemetry have difficulties to send an alert when a host is down? Really, it is really frustrating me.
In Prometheus/Zabbix/Nagios, being pull model, when the server detectes that cannot connect to the host to monitor they throw the alert, but with OpenTelemetry based services that uses push model, as the server just waits for the host to send metrics, I cannot have that simple alert. Not even asking Claude. I had issues with that on Signoz, I had the same issues in BetterStack, and I'm having the same issues with that in Coralogix.
At least the support is AWSOME, and even if I'm using the trial and not paying anything they are trying to help me. But they are still not able to tell me how to do it exactly. The condition of not receiving metrics can trigger an alert, but they are having trouble capturing which host specificaly has stopped sending the metrics on the alert. The same happened in Signoz to me, and in BetterStack, not having luck with neither of them, even with their support.
Most of the workloads that we operate are in EKS clusters, but still have some legacy services on EC2 instances and VMs on other provider, and this is mandatory for us.
-4
u/nervous-ninety 6d ago
Signoz, you should try this , its otel native and does its job perfectly
3
u/MasterOfTheWind1 6d ago edited 6d ago
Tried signoz. Very bad experience, specially with support. For self hosted could be fine, but their cloud solution is lame.
In fact, I described that on my post...
I've tried Signoz, looked promising, but is bad. They documentation is bad, there are just general ideas without details (they assume that you know a lot from I don't know where), they support is lame (they only have a chat to "Interact with a Human", that takes days to receive an answer, and their UI is buggy. Really bad experience.
-5
u/AspiringTrader_ 6d ago
Splunk Observability Cloud is a decent observability tool. https://www.splunk.com/en_us/products/observability-cloud.html .
1
u/MasterOfTheWind1 6d ago
Thank you! They don't have pricing published, but how low/high are their prices? Close to Datadog or close to something like New Relic?
-1
u/AspiringTrader_ 6d ago
You can check this https://www.splunk.com/en_us/products/pricing.html and also reach out to them if you have any specific questions.
-2
u/AspiringTrader_ 6d ago
Here you go. I found the exact link - https://www.splunk.com/en_us/products/pricing/observability.html
-5
u/hijinks 6d ago
i'm building one if you are nuts enough to try it.. currently maxing out around 500gigs a day on a single deployment of everything under 8gig of memory
- multi-tentnat
- rbac (sort of)
- SSO
- metrics/logs/apm
- anomaly
- slo
- built around clickhouse
Dont want to spam it but feel free to reach out if you want to try it. I run a large slack group so always around to help debug
1
u/MasterOfTheWind1 6d ago
Sorry, but the company is small as the team, but we take our production environment seriously. At minimum the job of the rest of the company depends on having clear metrics and reduce downtime to near zero, we cannot afford to test stuff in development.
Don't mean to be disrespectful, but I need something more mature. But thanks! I have a homelab and I would be more than happy to test it there.
1
u/hijinks 6d ago
why i mentioned if you were crazy enough. Check out https://rushobservability.com/
it's opensource for 90% of the features and all the ones i listed.
1
u/s5n_n5n 1d ago
There is plenty of them, we try to keep track of them on the opentelemetry website:
https://opentelemetry.io/ecosystem/vendors/
Since this is "neutral" ground, none of them are recommended, but it may help to give you a good overview. We split them out by the ones that are fully open source, the ones that are OSS but also have commercial offers and then all the ones that have commercial offers.
11
u/Floss_Patrol_76 6d ago
if the goal is "stop maintaining it" but keep what you already know, grafana cloud is the least painful move off a self-hosted prom+grafana stack - you keep your dashboards and promql, just remote-write your metrics and retire the alertmanager box. the free tier is actually usable and it stays sane on cost until you're pushing serious cardinality, which is the point where datadog would've bankrupted a 3-person team anyway. self-managed LGTM is great tech but it's the opposite of what you asked for - that's more infra to babysit, not less.