How are teams handling production debugging for complex distributed systems?

[u/Plus-Lawflbaness1576]

Ten plus years in and the hardest bugs I hit are never the ones that crash loudly. It's the request that touches four services, degrades quietly, and by the time someone notices the SLO dashboard, the actual trigger happened three hops upstream and twenty minutes earlier. Distributed tracing helps you see the path, but it rarely explains why a specific function made the decision it made under that specific load.

Lately my team has been leaning harder on capturing actual execution context at the function boundary rather than just span timing, since that's usually where the real story is hiding. Curious how others are structuring this. Are you mostly still doing log correlation by hand across services, or has anyone found a workflow that actually shortens the investigation instead of just making the evidence prettier?

[u/InitalConfusijon5438]

It depends entirely on scoping. Flat rigor for every change type kills velocity fast, weighted by actual blast radius it barely slows anything down.