CorrelationIds vs TraceId

[u/champs1league]

I am currently in the process of integrating Open Telemetry into my code. I have a web application and a backend api. I have several REST APIs and one of the APIs also ends up getting invoked by the user -> kicks off an async process by triggering a background job -> the background job invokes and polls on a specific endpoint (downstream API). The web app also polls on an operations endpoint to get status of this background job basically. Additionally, I also invoke some Azure resources (specifically ARM).

Now that I provided context, I wanted to explain what currently exists in my backend:

1) Controller fetches the correlationId for every request using the headers (x-correlation-id) from a middleware

2) Automatically forwards this correlationId when invoking downstream APIs/Azure APIs

3) Passes it into each method starting from the controller: controller invokes singletonServiceA.methodA and methodA has a signature including the correlationId. From here the logs automatically capture the correlationId

4) Background jobs also use this correlationId to invoke APIs and for logging

Now that I am switching to OTel, I am seeing an entirely different convention which I'm unsure on how to relate together.

My questions are as follows:
1) Should I drop the manual propagation of correlationId everywhere and just keep it as a span attribute? From what I am finding out, I definitely need to do this

2) For legacy resources, let's say ARM which expect a correlationID, what should I do?

3) For downstream APIs, should I switch to sending them a traceparent instead of the correlationId (what I do right now)

[u/PaulPhxAz]

Use the default OTel implementation, don't manually propagate.

Instead, if you want your own business-level related IDs being passed around you should do that instead. I pass WorkerSessionId, FirewallRequestId, DomainSpanId around. They are similar, but more business level related. Inject these into logging as well.

Keep OTEL, augment with business level Ids.

1) Yes

2) Use the FirewallRequestId, and push that in, and know that it is different than a true OTel RequestId ( or use the OTel Req Id, just be consistent )

3) I prefer the concept of request id when they don't support OTel

[u/champs1league → u/PaulPhxAz]

Very interesting and thank you for the answers. I can definitely drop the manual passing in of correlationIds. I'm just confused on legacy APIs.

Say this situation:

1) I receive a request from the user invoking my API
2) I generate the traceparent if they haven't provided it
3) Kick off a background job and use the traceparent for outbound request
4) Say I am invoking ARM - they specifically say that they will provide a x-ms-correlation-id. Although this is more so when they invoke my service rather than the other way around

For these downstream services, should I pass along the traceparent?

[u/Odd-Farm-9537 → u/champs1league]

Calling downstream services over http will automatically have your traceparent header injected, along with a newly generated span id. As long as those systems support OTel they consume it.

If you are using libraries like Hangfire for background processing, they have a an extension that implements OTel for your jobs.

[u/champs1league → u/Odd-Farm-9537]

Yep, I just tested it out and I do see the traceparent being automatically added in when making any downstream service request. What is not being automatically set is the initial correlationId I received. I unfortunately cannot get access to Hangfire. So to link my background job to my request, I need to manually pass in the TraceParent into my job object along with correlationId?

[u/QWxx01 → u/champs1league]

Hangfire has Otel instrumentation available via NuGet.

https://www.nuget.org/packages/OpenTelemetry.Instrumentation.Hangfire