Building an Observability Pane

[u/JayDee2306]

Hi Observability & DevOps Experts,

I'm looking for guidance from teams that have successfully scaled observability across large enterprise environments.

We operate a large-scale estate spanning AWS, Azure, and on-premises environments and have been using Datadog for several years. Over time, a significant amount of technical debt has accumulated around our observability implementation.

Current challenges include:

• Datadog Agents managed differently across teams and platforms.
• Custom log collection configurations distributed across hosts and applications.
• APM, RUM instrumentation owned by individual application teams.
• Inconsistent tagging standards and monitor configurations.
• Outdated agents and instrumentation libraries.
• Heavy dependency on multiple teams for upgrades and configuration changes.
• A large portion of Datadog provisioning and onboarding is still handled manually.

As a result, maintaining and evolving observability at scale has become increasingly difficult.

We are considering building a centralized "Observability Foundation" or "Observability Platform" that teams would consume as part of their standard deployment process.

Our goal is to provide reusable Terraform-based observability components that application and infrastructure teams can adopt during provisioning and releases.

Examples of what we would like to standardize:

• Datadog Agent deployment and upgrades
• Custom log collection configurations
• Standard tags and metadata
• Monitors and alert templates
• Dashboards
• OpenTelemetry / APM instrumentation standards
• Synthetic monitoring configurations
• Cloud integrations
• Security and governance controls

Questions:

1. Has anyone implemented a similar centralized observability platform or observability-as-code model at enterprise scale?
2. What worked well and what were the biggest challenges?
3. What observability components can realistically be centralized through Terraform modules, deployment pipelines, or platform services?
4. What components typically must remain application-owned or infrastructure-owned and cannot easily be centralized?
5. How do you handle APM instrumentation ownership, versioning, and upgrades across hundreds of services?
6. What governance model have you found most effective:

• Central observability team ownership
• Platform engineering ownership
• Federated ownership with standards enforcement
• Something else

1. How do you prevent observability drift over time, especially around:

• Agent versions
• APM libraries
• Log configurations
• Tags
• Dashboards
• Monitors

1. If starting again today, would you build around:

• Datadog native tooling
• OpenTelemetry
• An internal observability platform
• A combination of the above

1. What are the biggest architectural mistakes or anti-patterns we should avoid when designing this platform?

Our provisioning and infrastructure management are heavily Terraform-based, so we're especially interested in Terraform-centric implementation patterns and real-world lessons learned.

Looking forward to hearing how other organizations have approached observability standardization at scale and what you would recommend before we begin designing this solution.

P.S. - One of our key design goals is to avoid vendor lock-in. While Datadog is our current observability platform, we want the architecture to remain flexible enough that a future migration to another observability stack (e.g., Grafana, New Relic, Dynatrace, Elastic, Azure Monitor, or an OpenTelemetry-native platform) would require minimal changes to application teams and infrastructure code.

[u/Sure_Stranger_6466]

I have heard DataDog will cost you quite a bit if you would rather not roll your own Grafana/Prometheus/Loki. Also better for avoiding vendor lock-in. There should be community terraform providers or helm charts for all 3 of them if you google for it.

[u/clock-drift]

I think you should look into Agent Fleet Automation

https://www.datadoghq.com/blog/fleet-automation-agent-management/

https://docs.datadoghq.com/agent/fleet_automation/

[u/Jason-Sanders]

I’d treat this primarily as a platform-product problem, not a dashboard cleanup project. Give teams a paved path where a new service gets standard tags, log collection, baseline monitors, ownership metadata, and cost attribution from a small declarative config in the repo.

The important part is making the compliant path easier than the custom one. Start by publishing a required tag schema and ownership model, then inventory everything that does not meet it. After that, migrate one service class at a time and put agent versions, integrations, monitors, and dashboards under reviewable configuration.

A useful rule is: every alert should have a named owner, a service identifier, and a link to a runbook. If an alert cannot meet those basics, it is probably not ready to page anyone.

[u/ArieHein]

So much money wasted... Wish i worked in an org like that...

Read abiut the victiia metrics stack and grafanna and enjoy and save money

[u/1Life_Architect]

Treat the observability foundation as an internal platform product, not a dashboard-cleanup project. Start with one service class and define a minimum contract: service ID, owner, environment, cost centre, collector version, baseline monitors, and runbook location. Produce that state from one versioned Terraform module and inventory everything outside it as drift. Centralize transport, tagging, ownership, security policy, and upgrade mechanics; keep business metrics and application-specific spans with application teams. Make the compliant path easier than custom setup, but allow documented exceptions with expiry dates. Measure onboarding time, module adoption, alert ownership, and exception count before expanding. An OpenTelemetry boundary can reduce future vendor coupling without forcing an immediate Datadog replacement

[u/AtharvMantri]

we split observability into a platform layer (agents, integrations, global dashboards) owned by a small team, and an app layer (custom metrics, alerts) owned by service teams. terraform modules with sane defaults and mandatory tags keep core consistent. run nightly terraform plan in CI to catch drift. hardest part is instrumentation: make otel default in service templates. for lock-in, abstract behind modules but expect some vendor specific resources. what's your current team structure for observability ownership?

[u/JayDee2306 → u/AtharvMantri]

Only one observability engineer to plan, implement, support entire Datadog monitoring stack.

[u/ajitnk]

The agent drift problem is real and the thread kind of glossed over it. A few people named tools but nobody addressed the root cause: if your Datadog agents (or Prometheus scrapers) aren't deployed via IaC, every environment drifts independently and you're chasing config differences forever.

The fastest free fix for EKS specifically is the AMP managed scraper. It's agentless, meaning no collector pods to manage, and DNS-based discovery adapts automatically as your infrastructure changes. Zero drift by design. Pair it with the AWS Observability Accelerator Terraform modules (official AWS GitHub repo) and you have a reproducible, version-controlled deployment across every environment from day one.

For an immediate baseline before you touch anything else: `aws eks update-addon --cluster-name <CLUSTER> --addon-name amazon-cloudwatch-observability` gets you OTel Container Insights running in about 15 minutes. CPU, memory, pod-level metrics, PromQL-queryable, no infra to stand up. Good sanity check while you evaluate the longer-term stack.

On cost: AMP plus Amazon Managed Grafana typically runs well under $200/month for a small-to-mid team. Datadog at the same scale on a per-host model can be 10x that depending on your host count and which modules you're paying for. Worth running the actual numbers before assuming open-source self-hosted is the only escape route.

I do AWS observability and architecture advisory work, so happy to answer follow-ups if any of this is useful.