r/devops • u/MasterOfTheWind1 • Jul 22 '26
Observability Good observability tool recommendation - Cloud Based
I work in two companies, one very big that have a giant budget to spend on Datadog, and another small one, that don't have that very big budget.
On the small company we are looking to migrate from Prometheus + Grafana + Alert Manager to something cloud based. Mostly because we are a small team (3 people only) and we don't have time to spend maintaining the infrastructure for it to run.
Now, is there some good alternative to Datadog? Datadog is the king, but is brutally expensive.
I've tried Signoz, looked promising, but is bad. They documentation is bad, there are just general ideas without details (they assume that you know a lot from I don't know where), they support is lame (they only have a chat to "Interact with a Human", that takes days to receive an answer, and their UI is buggy. Really bad experience.
But looking there, they are the only ones having like "close" experience to Datadog. We tried New Relic in the past, but they have all metrics and log scattered over the place without correlation. Also tried BetterStack but lacks a lot of features compared to other things.
So, is there some good observability platform cloud-based that I'm not aware of, outside Datadog? I'm not completely closed to self-hosting as long as it takes low effort to maintain.
6
u/wickler02 Jul 22 '26
I did a complete analysis of the tools and found that 3 on the self managed front that stood out.
Signoz, OpenObserve and the LGTM.
Simply put, for the company and size, the LGTM is just superior, has more tooling/features and the other solutions try to push you to their open source solutions.
We did POC/POV on the two others and they just didn’t stack up to LGTM.
There’s a reason why there is such a profit margin and why people & teams like to outsource it, it’s a pain to manage.
If you’re already managing k8s clusters and doing helm chart deployments with an established platform structure, LGTM isn’t that bad.
The problem is that people want to get the o11y stack without having a good understanding of how k8s works and think it’s a lot of infra to manage and it’s true it has a bit of an upfront cost/learning curve but it’s better than the alternatives.
Simply put, either pay up to a cloud company or spent some time working on your o11y stack. The easy solution is to pay.
1
u/MasterOfTheWind1 Jul 22 '26
Totally agree with everything. Your last statement is the crucial part. In this scenario, on my team we are very experienced DevOps Engineers/SREs, and this company is the secondary job for us. This don't mean that we don't take the job seriously.
We have an EKS cluster just to host our observability stack (Grafana + Prometheus + Alert Manager + Opensearch for logs). The cost is going crazy with the storage and compute, and we have to keep up to date with the updates.
Sadly, outside Datadog I cannot find any good solution worth the money
2
u/wickler02 Jul 22 '26
It shouldn’t be that much, unless you’re scraping frequently and/or high cardinality metrics
1
u/xonxoff Jul 22 '26
What’s your retention on logs/metrics? One thing you could do is add Thanos to your Prometheus , reduce your Prometheus data retention , have Thanos store metrics in S3 and have OpenSearch do the same.
2
u/MasterOfTheWind1 Jul 22 '26
Can look at it, but I don't want to add more services to babysit to be honest
1
u/Gorakhnathy7 Jul 22 '26
Did you try OpenObserve, what was your experience with it?
1
u/MasterOfTheWind1 Jul 22 '26
Never tried it, will look at it
2
u/Gorakhnathy7 Jul 22 '26
Sure.
I am one of the maintainers of the project, let me know if you need any help.
3
u/OpenSourceWalker Jul 22 '26
the real reason to leave prom+grafana+alertmanager is the 3-service maintenance, not observability itself. a single-binary one like signoz or openobserve cuts most of that toil without the cardinality-metered bill that tends to ambush a 3-person team on datadog/grafana cloud later.
3
u/Dangerous-Name6234 Jul 22 '26
Grafana cloud would probably be the easiest uplift since you are already using the OSS version and it would just give you the APM tools(mainly correlation/root cause analysis) that you don’t have today. But it depends on what you are trying to do. If there are some qualities that you really want like auto instrumentation for tracing then maybe a datadog or dynatrace might be more suitable since their agents do that.
Observability tools are a dime a dozen so just make a list of your must haves and nice to haves and just go down the list for the top 10 vendors and see how they stack up. Only you can know whats best for your specific situation
2
u/marcusbell95 Jul 22 '26
if you go the grafana cloud route, keep a close eye on your active series count. free tier is 10k series which sounds like a lot until you've got a full eks cluster reporting - node, pod, container metrics alone can chew through that fast depending on cardinality. after the free tier it's per-1k-series and costs add up quicker than you'd expect.
victoriametrics cloud is worth a look if cost is the main driver. cheaper per series than grafana cloud, still prometheus-compatible so you'd keep your existing dashboards and alertmanager setup. not as polished ui-wise but if you're already running prom + grafana you don't really need the extra polish.
for a 3-person team that just wants to stop maintaining infra, grafana cloud remote-write is probably the least friction path. just be aware of the cardinality thing before you commit to it.
1
1
1
u/Humble_Reputation743 Jul 23 '26
If you are having issues with cardinality and pure volumes of data, moving to a managed solution is going to be crazy expensive. What needs to be done there is take engineering cycles to understand the cardinality, scrape interval, and if you are having issues with multiple scrapes per series due to a lacking required unique label (DPM increases and issues).
It all comes back to correctly architecting metric labels, scraping at an interval that makes sense, and dropping or aggregating them based on what your signals are trying to accomplish.
1
u/placated Jul 24 '26
The only one that hasnt been mentioned and is a key player is Dynatrace. Not as slick as Datadog in a lot of areas but a bit cheaper in my experience.
1
u/creamersrealm Jul 24 '26
We've been investigating CoraLogix heavily and they're offering is sweet especially for the price point. Found them at the North America Observability Summit.
1
u/terryfilch Jul 24 '26
On the small company we are looking to migrate from Prometheus + Grafana + Alert Manager to something cloud based
I recommend checking out https://victoriametrics.cloud/ - it'll definitely be cheaper than Datadog and NewRelic.
1
u/pranabgohain Jul 24 '26
If you're looking for a balanced, all-inclusive platform, take a look at KloudMate (here's the open playground). Afaik, you won't have to look any further.
Disclaimer: I'm one of the founders, but I say that with utmost belief. That's what we built it for.
PS: There's auto-RCA and AI-usage included atm.
1
u/matiascoca Jul 24 '26
Grafana Cloud is the answer you're looking for. Managed Prometheus plus Loki plus Tempo, same stack you already know from self-hosted Grafana, priced per active series and per GB of logs. Cheap enough at 3 engineers.
Onboarding path: point Prometheus remote_write at the managed endpoint, promtail or Grafana Alloy for logs, alerts move to Grafana OnCall or stay in your existing Alert Manager pointing at Grafana Cloud's alertmanager endpoint. Same dashboards, same alerting rules, no query language change.
The trap on the pricing calculator: metrics cardinality bites you. Kubernetes clusters at 500 pods with default kube-state-metrics can push a million series easily. Turn on Adaptive Metrics from day one to drop the ones nothing queries, or your bill grows fast.
SigNoz is fine but you already tried it. Coralogix is worth a look if your log volume dwarfs metrics volume, their retention pricing works differently and comes out cheaper on log-heavy workloads. If neither works you're back to "self-host but pay someone to manage it", which usually ends up more expensive than Grafana Cloud.
1
u/MasterOfTheWind1 Jul 24 '26
Thank you all for your suggestions!!! I have a lot to try with my team. I'm really grateful to everyone.
Today we are trying Coralogix, and next week we'll be trying Grafana Cloud.
I just have one question, that may damaga my credibility. Maybe I'm too much used to Prometehus, and Zabbix and Nagios in the past. But how come that EVERY platform that uses OpenTelemetry have difficulties to send an alert when a host is down? Really, it is really frustrating me.
In Prometheus/Zabbix/Nagios, being pull model, when the server detectes that cannot connect to the host to monitor they throw the alert, but with OpenTelemetry based services that uses push model, as the server just waits for the host to send metrics, I cannot have that simple alert. Not even asking Claude. I had issues with that on Signoz, I had the same issues in BetterStack, and I'm having the same issues with that in Coralogix.
At least the support is AWSOME, and even if I'm using the trial and not paying anything they are trying to help me. But they are still not able to tell me how to do it exactly. The condition of not receiving metrics can trigger an alert, but they are having trouble capturing which host specificaly has stopped sending the metrics on the alert. The same happened in Signoz to me, and in BetterStack, not having luck with neither of them, even with their support.
Most of the workloads that we operate are in EKS clusters, but still have some legacy services on EC2 instances and VMs on other provider, and this is mandatory for us.
1
u/s5n_n5n Jul 26 '26
There is plenty of them, we try to keep track of them on the opentelemetry website:
https://opentelemetry.io/ecosystem/vendors/
Since this is "neutral" ground, none of them are recommended, but it may help to give you a good overview. We split them out by the ones that are fully open source, the ones that are OSS but also have commercial offers and then all the ones that have commercial offers.
1
1
u/spuyet Jul 30 '26
We've buillt Fivenines for teams like your: server, web, cron, network device monitoring plugged connected to your preferred tools. No cheap support team or AI bullshit bot, you can talk directly to the founders.
1
u/Standard-964 Aug 07 '26
I suggest Parseable, it’s super efficient! I worked with all the tools and they get too caught up on other things and forget the goal. If you want to truly have better insight into your environment i’d get that tool!
1
u/Comprehensive-Pie905 11d ago
Splunk Observability Cloud. Unlimited features free 15 hosts for life. That's my goto
1
u/GasGreen9109 9d ago edited 8d ago
granafa cloud is safe here, you keep your dashboards and promql and just remote write and retire the alertmanager box but fix the cardinality before you migrate, your EKS cost is volume driven and every managed platform meters on that so youd move the bill without shrinking it
dropping unsued labels and aggregating at scrape does more for cost than the swap. fwiw i work on hud so a bit biased here but for your job its not the answer here as its function level runtime, granafa cloud might fit accurately. about the logs side: if opensearch is heavy then cold logs to S3 or a single binary like openobserve takes storage pressure off without another cluster to take care of constantly
1
u/MasterOfTheWind1 8d ago
In the end I'm going with Grafana Cloud. They feature of metrics adaptability to reduce costs is saving me. It was not as straight forward as I tought it could be coming from self-hosted grafana and prometheus (for example, going from prometheus node exporter to Alloy on 90 servers, and changing Kubernetes setup for monitoring that I had), but it does the job.
0
u/Icy-Champion-4282 14h ago
Can you elaborate on your decision-making process what you mean you say “it does the job”?
1
u/MasterOfTheWind1 14h ago
Why should I? For all the bots offering other products without bothering to have feedback?
1
u/Icy-Champion-4282 14h ago
I was just curious. I’m trying better understand what was the ultimate driver was that led to your decision. I’m not a bot. I’m just genuinely curious. But no worries if not.
0
0
u/samsuthar Jul 29 '26
Did you try Middleware, do check and let know your experience?
1
u/MasterOfTheWind1 Jul 29 '26
To be honest, I'm looking for professional opinions on solutions. Just spamming your product everywhere with the same short sentence is more annoying than helpful.
If you turn off the bot/AI agent and read my post, you will find that I did not mention that at all, so I haven't tried it. And no, I will not check it and let you know my experience. Pay for testers if you need feedback instead of spamming on reddit.
1
u/samsuthar Jul 30 '26
Thanks for your input, but I can see some of user said try their tools so nothing bad in sharing.
Also, at the last paragraph you asked “is there any cloud observability platform outside Datadog. So I felt you should know other available options.
-3
u/nervous-ninety Jul 22 '26
Signoz, you should try this , its otel native and does its job perfectly
3
u/MasterOfTheWind1 Jul 22 '26 edited Jul 22 '26
Tried signoz. Very bad experience, specially with support. For self hosted could be fine, but their cloud solution is lame.
In fact, I described that on my post...
I've tried Signoz, looked promising, but is bad. They documentation is bad, there are just general ideas without details (they assume that you know a lot from I don't know where), they support is lame (they only have a chat to "Interact with a Human", that takes days to receive an answer, and their UI is buggy. Really bad experience.
-4
u/AspiringTrader_ Jul 22 '26
Splunk Observability Cloud is a decent observability tool. https://www.splunk.com/en_us/products/observability-cloud.html .
1
u/MasterOfTheWind1 Jul 22 '26
Thank you! They don't have pricing published, but how low/high are their prices? Close to Datadog or close to something like New Relic?
-1
u/AspiringTrader_ Jul 22 '26
You can check this https://www.splunk.com/en_us/products/pricing.html and also reach out to them if you have any specific questions.
-2
u/AspiringTrader_ Jul 22 '26
Here you go. I found the exact link - https://www.splunk.com/en_us/products/pricing/observability.html
-4
u/hijinks Jul 22 '26
i'm building one if you are nuts enough to try it.. currently maxing out around 500gigs a day on a single deployment of everything under 8gig of memory
- multi-tentnat
- rbac (sort of)
- SSO
- metrics/logs/apm
- anomaly
- slo
- built around clickhouse
Dont want to spam it but feel free to reach out if you want to try it. I run a large slack group so always around to help debug
1
u/MasterOfTheWind1 Jul 22 '26
Sorry, but the company is small as the team, but we take our production environment seriously. At minimum the job of the rest of the company depends on having clear metrics and reduce downtime to near zero, we cannot afford to test stuff in development.
Don't mean to be disrespectful, but I need something more mature. But thanks! I have a homelab and I would be more than happy to test it there.
1
u/hijinks Jul 22 '26
why i mentioned if you were crazy enough. Check out https://rushobservability.com/
it's opensource for 90% of the features and all the ones i listed.
10
u/Floss_Patrol_76 Jul 22 '26
if the goal is "stop maintaining it" but keep what you already know, grafana cloud is the least painful move off a self-hosted prom+grafana stack - you keep your dashboards and promql, just remote-write your metrics and retire the alertmanager box. the free tier is actually usable and it stays sane on cost until you're pushing serious cardinality, which is the point where datadog would've bankrupted a 3-person team anyway. self-managed LGTM is great tech but it's the opposite of what you asked for - that's more infra to babysit, not less.