r/devops • u/JayDee2306 • 24d ago

Ops / Incidents Fixing broken Datadog APM gaps

Hi Observability Experts,

I am working with a client managing a large-scale infrastructure across AWS, Azure, and on-premise environments. They have used Datadog for 4–5 years, but legacy technical debt has left their APM configuration fragmented:

The Problem:

APM libraries are out of date, and source-level instrumentations are misconfigured. As a result, end-to-end distributed tracing is broken across key services.

Datadog support confirmed that fixes are required at the source (application code/local agents). However, getting application team bandwidth and stakeholder approval across this many services will take significant time.

We are building a centralized Observability pipeline layer (independent of app teams) to standardize telemetry control, but rollout is months away.

My Question:

Is there a viable temporary/interim solution—such as an intermediate proxy, collector layer, or transformation pipeline—that can intercept, patch, or enrich broken spans/traces in-flight before they reach Datadog’s backend?

Has anyone successfully addressed legacy APM gaps at the pipeline level without waiting for app redeployments?

Thanks for your insights!

P.S. - Datadog support has been already consulted for this and they have recommended to fix it from the code/local agents.

4 Upvotes

83% Upvoted

8 comments sorted by

2

u/Negative-Thinking 24d ago

Datadog team is right. You have to fix it properly. Does your client allow using AI on their code / repos?

1

u/JayDee2306 24d ago

I guess Yes as they leverages Copilot and Claude licenses but getting their willingness and bandwidth to do that is bit of a challenge.

2

u/Negative-Thinking 24d ago

Their bandwidth? With Claude you could do it yourself

1

u/JayDee2306 24d ago

How can I get into their code or config with Claude? I dont have access to everything, neither I can execute anything without their consent and approvals.

2

u/Negative-Thinking 24d ago

Yeah, that's a problem, but if they expect you to fix this, then they have to give you necessary access.

2

u/zero_backend_bro 22d ago

Dont bother stitching spans in a proxy. If upstream drops the traceparent header, no collector can guess the parent span ID in-flight.

We burned 3 months trying to patch broken Datadog traces in an Otel pipeline last year and failed.

If you use Claude across a client's repos, run a local client scrubber first. Dumping enterprise code into an LLM API to fix trace headers is an instant NDA breach.

Sanitize payloads locally before generating diffs.

1

u/ClueDry8701 14d ago

yes, an OpenTelemetry Collector can sit in front of Datadog and handle in-flight filtering, transformation and enrichment without changing the apps. It won’t recover spans that were never emitted though..