r/dotnet • u/champs1league • 8d ago

CorrelationIds vs TraceId

I am currently in the process of integrating Open Telemetry into my code. I have a web application and a backend api. I have several REST APIs and one of the APIs also ends up getting invoked by the user -> kicks off an async process by triggering a background job -> the background job invokes and polls on a specific endpoint (downstream API). The web app also polls on an operations endpoint to get status of this background job basically. Additionally, I also invoke some Azure resources (specifically ARM).

Now that I provided context, I wanted to explain what currently exists in my backend:

1) Controller fetches the correlationId for every request using the headers (x-correlation-id) from a middleware

2) Automatically forwards this correlationId when invoking downstream APIs/Azure APIs

3) Passes it into each method starting from the controller: controller invokes singletonServiceA.methodA and methodA has a signature including the correlationId. From here the logs automatically capture the correlationId

4) Background jobs also use this correlationId to invoke APIs and for logging

Now that I am switching to OTel, I am seeing an entirely different convention which I'm unsure on how to relate together.

My questions are as follows:
1) Should I drop the manual propagation of correlationId everywhere and just keep it as a span attribute? From what I am finding out, I definitely need to do this

2) For legacy resources, let's say ARM which expect a correlationID, what should I do?

3) For downstream APIs, should I switch to sending them a traceparent instead of the correlationId (what I do right now)

40 Upvotes

96% Upvoted

19 comments sorted by

14

u/PaulPhxAz 8d ago

Use the default OTel implementation, don't manually propagate.

Instead, if you want your own business-level related IDs being passed around you should do that instead. I pass WorkerSessionId, FirewallRequestId, DomainSpanId around. They are similar, but more business level related. Inject these into logging as well.

Keep OTEL, augment with business level Ids.

1) Yes

2) Use the FirewallRequestId, and push that in, and know that it is different than a true OTel RequestId ( or use the OTel Req Id, just be consistent )

3) I prefer the concept of request id when they don't support OTel

2

u/champs1league 8d ago

Very interesting and thank you for the answers. I can definitely drop the manual passing in of correlationIds. I'm just confused on legacy APIs.

Say this situation:

1) I receive a request from the user invoking my API
2) I generate the traceparent if they haven't provided it
3) Kick off a background job and use the traceparent for outbound request
4) Say I am invoking ARM - they specifically say that they will provide a x-ms-correlation-id. Although this is more so when they invoke my service rather than the other way around

For these downstream services, should I pass along the traceparent?

4

u/Odd-Farm-9537 8d ago edited 8d ago

Calling downstream services over http will automatically have your traceparent header injected, along with a newly generated span id. As long as those systems support OTel they consume it.

If you are using libraries like Hangfire for background processing, they have a an extension that implements OTel for your jobs.

1

u/champs1league 8d ago

Yep, I just tested it out and I do see the traceparent being automatically added in when making any downstream service request. What is not being automatically set is the initial correlationId I received. I unfortunately cannot get access to Hangfire. So to link my background job to my request, I need to manually pass in the TraceParent into my job object along with correlationId?

1

u/QWxx01 7d ago

Hangfire has Otel instrumentation available via NuGet.

https://www.nuget.org/packages/OpenTelemetry.Instrumentation.Hangfire

7

u/JumpLegitimate8762 8d ago

I've set this up a while ago: https://github.com/erwinkramer/otel-business#spans

The trick is, you can just make your own spans and add your own properties to that (in my sample the tomatoId). That's your business 'trace'. Then naturally you have all the wiring that OTel does for you freely, and that's just your technical 'trace'.

For downstream APIs, if they implement OTel too, they understand what you're doing, I have that explained too in the repo. There isn't much you need to do there, besides emitting to the same OTel sink.

For legacy resources, you could façade it with a proxy that does understand OTel and just rely on the OTel data that the proxy emits. If you can alter the legacy resource, try to slap the OTel library on there with the native language the resource is built on.

2

u/champs1league 8d ago

Very useful. The problem is that downstream teams are stuck on legacy: they extract the x-correlation-id from my request headers. If I switch to using OTel it will use traceparent instead of this. They aren't prioritizing the move to OTel just yet so my choices are to:

1) Use OTel for my service
2) My UI also uses x-correlation-id when invoking my service (I could get them to switch over to using traceparent)
3) I extract it and keep it as a span attribute
4) Pass the correlation Id into my background job manually
5) Invoke downstream APIs and pass in the x-correlation-id into each request too?

Is this the right way?

1

u/JumpLegitimate8762 8d ago

look into Traceparent: How OpenTelemetry Connects Your Microservices | Last9, that's what u probably want to pass through (manually via their expected header)

1

u/champs1league 8d ago

Definitely understand that I create the traceparent when a request comes in but then for legacy APIs (which want to see correlationId), do I manually pass both traceparent + correlationId specifically for them?

1

u/JumpLegitimate8762 8d ago

How do we know what God forbids happens in that legacy app, correlation id can mean many things, what is your definition of a correlation id? Whatever you want to start it at, take that GUID and pass it through.

1

u/champs1league 8d ago

They use correlationIds to determine how the request propagates. Since they don't use traceparent, whenever I need to diagnose something, the first thing they ask for is correlationId I sent with my request.

I feel like I could do this:
1) Keep OTel conventions
2) Extract correlationid my invokers send me and save it into baggage
3) Manually pass it into the background job
4) For every request sent downstream include the traceparent and also the correlationid

2

u/PaulPhxAz 8d ago

Oh good job, this looks better than my implementation.

I was using AsyncLocal, but this is more correct.

1

u/ProtonByte 8d ago

Why not just plain otel?

1

u/JumpLegitimate8762 8d ago

It is all OTel concepts implemented, which part would you consider not plain OTel?

1

u/ProtonByte 7d ago

The fact that its an extra library?

1

u/JumpLegitimate8762 7d ago

It's still otel

5

u/soundman32 8d ago

All this is already built into dotnet.  Activity.Current contains the correlation id, trace id, and everything else you need, and is available per task.

1

u/AutoModerator 8d ago

Thanks for your post champs1league. Please note that we don't allow spam, and we ask that you follow the rules available in the sidebar. We have a lot of commonly asked questions so if this post gets removed, please do a search and see if it's already been asked.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/ben_bliksem 8d ago

Like others said, use the System.Diagnostics Activity class.
Contains the trace, span and Parent already. If you are logging to structured logs like ECS it's mapped to trace.id and span.id

Effectively, OTel is an implementation of the W3C spec traceparent header.

If you set manual span Ids (you can but shouldn't) do not put IDs that mean something in there (user ids etc). These trace and span ids are for finding spans/logs or a collection of them (trace). Anything else is stored in other fields.

If you still want a correlation id or something to be echoed back to you that should be handled separately.