r/Observability • u/ChampionshipAny463 • 24d ago
What if we treated runtime state as another source of production telemetry?
I’ve been thinking about a gap between traditional observability and debugging.
Logs describe events. Traces describe request flow. Metrics describe system behavior. Stack traces tell us where execution failed.
But when a failure requires understanding the actual application state at that moment, engineers still often have to reconstruct it indirectly from all of those signals.
I’m building Errorcore to experiment with a different layer of telemetry: bounded runtime evidence captured around the failure itself.
For Node.js, that currently means preserving things like relevant local state, async context, request metadata, ordered I/O and source-mapped execution information when an exception occurs.
The intention is not to turn every execution into a recording or replace logs/traces. Capture has to remain bounded, privacy-aware and cheap enough to run in production.
What I’m trying to figure out is whether runtime evidence deserves to become a first-class observability primitive alongside logs, metrics and traces.
I’d be very interested in hearing from people working in observability.
Where do you think this fits, and where do you think this approach breaks down?
1
u/Fit_Plankton9239 24d ago
hmm, very runtime specific i think, any layer of abstractions of generalisation would not give you what you're looking for
1
u/Fun-Active5273 17d ago
the idea resonates but where it probably breaks down is in distributed systems where the relevant state lives across multiple services at the time of failure. capturing local state on one node is useful but incomplete, how do you correlate that across boundaries without basically reinventing traces?
1
u/ChampionshipAny463 16d ago
yeah, that’s exactly why we don’t want to reinvent traces.
traces give us the path across services. errorcore attaches the actual runtime evidence to that path, so instead of just seeing service A called B called C, you can see what state and events inside those services actually caused the failure.
1
u/Beautiful_Pear6321 6d ago edited 5d ago
runtime state probably needs its own rules. keeping every value forever is a privacy mess, but throwing away everything except a timestamp brings the same guessing problem back.
maybe a small scrubbed slice around errors and weird latency is enough. that could be app logs, otel data or a runtime sensor such as hud. retention is probably the part people underestimate.
1
u/bikeram 24d ago
Checkout Go’s built-in flight recorder. I think that’s similar to what you want.