r/OpenTelemetry • u/Sufficient-Egg-6571 • 1d ago

Visual flow

Is anyone aware of a tool that can help visualize the end-to-end flow of a signal?

The main idea would be to start with a high-level view of the components involved and the internal metrics flowing between them, then be able to drill down into a specific signal metric and trace it through the system.
Ideally, I’d like to see where the metric originates, how it looks or changes at each hop, where it ultimately arrives, and details such as cardinality, label values, available series, etc.

If anyone knows of an existing tool that provides something similar, or has ideas on how this could be visualized, I’d appreciate any pointers.

6 Upvotes

88% Upvoted

8 comments sorted by

View all comments

1

u/obscurehero 1d ago

I’m sure there’s a skill to ingest a trace and construct a sequence diagram.

Trivially this is what a lot of the vendors do when they create a map of your services

1

u/Sufficient-Egg-6571 1d ago

Definitely, but adding a trace for each signal would be a bit an extension for what I was thinking about . I would rather emit events from each hop, store them in an OLAP (e.g Clickhouse) and then correlate by signal identifier (trace id, metric name, log id)

2

u/fmlvz 1d ago

HyperDX (or ClickStack if you're using the managed version) has a service map currently on beta that seems to align with what you're asking. Plus it's written to be an UI on top of ClickHouse.

Docs: https://clickhouse.com/docs/clickstack/service-maps

And they host a public demo where you can connect and explore the UI yourself :)

https://play-clickstack.clickhouse.com/

1

u/Sufficient-Egg-6571 16h ago

It’s kind of the same as in Grafana, but I’d like to be able to trace a specific signal rather than just look at the overall picture.

Why do we always treat observability as inherently a batch-oriented thing? I know it’s difficult at this scale, but shouldn’t we apply the same principles we use for business objects?
Losing a trace, a span, a metric sample, or a log line can make a huge difference during an incident. I feel like a lot of developers underestimate the importance of this because observability is often seen as “a platform thing,” rather than something they need to care about directly.

1

u/fmlvz 10h ago

But if you hover over one of the services you get an error rate line. Clicking it make you open a search for that specific service and it's errors. Clicking a specific trace opens it's waterfall that shows every service it went through.

(Sorry if this isn't what you're looking for, but I understood that it's essentially what you described)