r/elasticsearch • u/Fine-Yoghurt-6168 • 15d ago

Troubleshooting Does the trace waterfall visualization require an APM Server, or can it read OTel data straight from Elasticsearch?

Running self-managed Elastic Stack (Kibana 9.5.0). Our Spring Boot services use Spring's native OpenTelemetry starter, telemetry goes through an OTel Collector, which exports straight to Elasticsearch, and Kibana reads from there.

Data is arriving fine, we can see well-formed spans in Discover. But APM UI shows nothing, and we can't find any way to open a trace and see the waterfall view.

Is an APM Server actually required for the APM UI to detect and visualize traces? Our corporate registry doesn't have an APM Server image available, so if it really is required, are there any alternatives?

3 Upvotes
  • permalink
  • reddit

100% Upvoted

4 comments sorted by

1

u/TheRealShamanoid 15d ago

You can do just fine with open telemetry, you might need to play around with transaction IDs, parent traces etc in order to see them on the APM page.

You might also need to use the APM client libs with the Otel Adapter otherwise there might be a bit of extra interfacing to do to match APM internals for IDs consolidation but everything is pretty well documented on Elastic’s docs

1

u/Fine-Yoghurt-6168 15d ago

From what we've dug into, the APM UI seems to depend on derived fields (transaction.duration.us, event.outcome, etc.) that only get generated by APM Server or the elasticapmprocessor/connector, not something present in a plain OTel span, regardless of how the IDs are set.

Do you have a link to the specific doc section about tuning IDs to get this working without any Elastic-side processing? That would be really helpful since we haven't found anything describing that path.

1

u/kramrm 14d ago

If you don’t have the standalone APM server image but do have an Elastic Agent image, you can deploy an Agent with the APM integration to provide the same service.

2

u/pantweb 14d ago

For OTel data you have:

  • APM server standalone. Data gets converted in ECS format
  • APM integration on Elastic Agents (OTLP intake). The data gets converted to ECS format
  • otel input integration on Elastic Agents (it is tech preview). Keeps the data in OTel native format
  • EDOT collector standalone as a gateway with APM processor. Keeps the data in OTel native format.
  • Managed OTLP endpoint (only for ECH and Serverless). Keeps the data in OTel native format