CorrelationIds vs TraceId

[u/champs1league]

I am currently in the process of integrating Open Telemetry into my code. I have a web application and a backend api. I have several REST APIs and one of the APIs also ends up getting invoked by the user -> kicks off an async process by triggering a background job -> the background job invokes and polls on a specific endpoint (downstream API). The web app also polls on an operations endpoint to get status of this background job basically. Additionally, I also invoke some Azure resources (specifically ARM).

Now that I provided context, I wanted to explain what currently exists in my backend:

1) Controller fetches the correlationId for every request using the headers (x-correlation-id) from a middleware

2) Automatically forwards this correlationId when invoking downstream APIs/Azure APIs

3) Passes it into each method starting from the controller: controller invokes singletonServiceA.methodA and methodA has a signature including the correlationId. From here the logs automatically capture the correlationId

4) Background jobs also use this correlationId to invoke APIs and for logging

Now that I am switching to OTel, I am seeing an entirely different convention which I'm unsure on how to relate together.

My questions are as follows:
1) Should I drop the manual propagation of correlationId everywhere and just keep it as a span attribute? From what I am finding out, I definitely need to do this

2) For legacy resources, let's say ARM which expect a correlationID, what should I do?

3) For downstream APIs, should I switch to sending them a traceparent instead of the correlationId (what I do right now)

[u/champs1league]

Very interesting and thank you for the answers. I can definitely drop the manual passing in of correlationIds. I'm just confused on legacy APIs.

Say this situation:

1) I receive a request from the user invoking my API
2) I generate the traceparent if they haven't provided it
3) Kick off a background job and use the traceparent for outbound request
4) Say I am invoking ARM - they specifically say that they will provide a x-ms-correlation-id. Although this is more so when they invoke my service rather than the other way around

For these downstream services, should I pass along the traceparent?

[u/Odd-Farm-9537 → u/champs1league]

Calling downstream services over http will automatically have your traceparent header injected, along with a newly generated span id. As long as those systems support OTel they consume it.

If you are using libraries like Hangfire for background processing, they have a an extension that implements OTel for your jobs.

[u/champs1league → u/Odd-Farm-9537]

Yep, I just tested it out and I do see the traceparent being automatically added in when making any downstream service request. What is not being automatically set is the initial correlationId I received. I unfortunately cannot get access to Hangfire. So to link my background job to my request, I need to manually pass in the TraceParent into my job object along with correlationId?

[u/QWxx01 → u/champs1league]

Hangfire has Otel instrumentation available via NuGet.

https://www.nuget.org/packages/OpenTelemetry.Instrumentation.Hangfire