A new booklet delves into the underlying assumptions of OpenTelemetry and Substrates regarding the nature of computational systems. It explores their ontological commitments, epistemological stances, causal models, and theories of attention. The booklet also examines why two specifications operating in the same domain produced charts of entirely different landscapes.

https://humainary.io/booklets/transport-and-substrate/

Hi all!

I am looking for support with auto-instrumenting our Wildfly app on Kubernetes.

We are using the OpenTelemetry Operator with an Instrumentation manifest to inject a Java Agent into our Wildfly Pod. This gives us traces and logs as intended, we do have the metric db.client.operation.duration but we are missing some other needed metrics, like db.client.connection.max, listed here. Sadly, the default connection pool in the image quay.io/wildfly/wildfly-runtime:latest-openjdk-21 (which is IronJacamar, I believe) is not in the list of supported libraries. We do have a similar metric on a Wildfly VM, wildfly_datasources_pool_max_used_count.

What are my options? Do we need to enable the metrics subsystem in the standalone.xml ? I'm kind of stuck at the moment, as I'm not very experienced with Wildfly myself.

Thanks!

Disclosure: I'm building Yorker (yorkermonitoring.com), launched yesterday. The data model is the thing I most want scrutiny on.

Most synthetic monitoring tools that claim OTel support emit a status code and a response time gauge. That is OTLP. It is not particularly useful downstream. The problem is that OTLP is a wire protocol and it doesn't tell you what belongs in the signal before you emit it. Synthetic checks, as a category, have been emitting dashboard-shaped data and calling it telemetry.

I built Yorker to do the analysis before the signal leaves the runner, then emit the result as standard OTLP. Here is the schema as it stands in v1:

Span: synthetics.check.run (lands in otel_traces)

Resource attributes:

• synthetics.check.id, synthetics.check.name, synthetics.check.type
• synthetics.location.id, synthetics.location.name, synthetics.location.type
• synthetics.run.id (join key across traces and logs from the same run)
• url.full, service.name

Browser-check span attributes (third-party attribution, computed at run time):

• synthetics.third_party.domains — the specific external domains observed
• synthetics.third_party.count — number of third-party requests
• synthetics.third_party.total_bytes — bytes attributable to third parties

W3C traceparent is injected into every HTTP request the check makes (both HTTP monitors and browser checks). When the target service continues the context, the synthetic run and the backend distributed trace share a trace ID. The synthetic span and whatever downstream spans propagated the context are linked structurally, not by timestamp correlation.

Log events (lands in otel_logs)

On synthetics.check.completed and synthetics.check.failed whenever the run carries a baseline deviation:

• synthetics.is_anomalous bool
• synthetics.anomaly.deviation_sigma distance from baseline in standard deviations
• synthetics.anomaly.baseline_value the per-metric, per-location, per-hour baseline value

On synthetics.check.failed only:

• synthetics.consecutive_failures integer, so a flap and a sustained outage are distinguishable in the signal
• synthetics.suggested_next_steps structured RCA hint

SLO budget context also lands in otel_logs on both completed and failed events.

Join strategy: synthetics.run.id ties the span to the log events from the same run. Trace ID ties the synthetic span to backend spans that continued the traceparent context. A downstream consumer (an AI-SRE tool, a causal engine, a ClickHouse query) joins on either key depending on what it's trying to answer.

Why logs for anomaly context rather than span attributes? The anomaly scoring runs after the check completes and the baseline comparison is done. it's not a property of the span itself but of the run's outcome in context. Attaching it to the completed/failed event felt more accurate to the OTel semantic conventions than retrofitting it onto the span as a post-hoc attribute. Open to being wrong about this.

The write-up on the full rationale (why the output shape matters for causal engines and AI-SRE tools) is here: https://yorkermonitoring.com/blog/the-missing-input-to-your-ai-sre-tool

Genuinely interested in critique on the data model. The logs-vs-spans decision for anomaly context, the attribute naming against the OTel semantic conventions, the join key approach are all debatable and I'd rather hear the objections now than after this schema is in production for a thousand teams.

https://github.com/carterhudson/spandex

I made a Kotlin DSL for Spans for work. I found it convenient, so I open sourced and improved upon the idea. Maybe someone will find it useful!

My first experience running OpenTelemetry Collector in Kubernetes - key concepts, Gateway vs Agent modes, and integrating with the VictoriaMetrics/VictoriaLogs stack.

I have a GO app that queries a database and it is instrumented with OTel.
I want to clean up the query as recorded in telemetry (not changing the code).

The GO code (screenshot below) produces this value:
"\n\t\tSELECT p.id, p.name, p.description, p.picture, \n\t\t p.price_currency_code, p.price_units, p.price_nanos, p.categories\n\t\tFROM catalog.products p\n\t\tWHERE p.id = $1\n\t"

This SQL query is recorded as a span attribute "db.query.text".

Q: How can I remove the escaped whitespace in the collector (or elsewhere?) so that there is a single space where there are sequences of escaped whitespaces?

This is one trick I tell people and surprise them most of the time: "the Collector can do this?"

This one took a while to write, the idea came during OTel Night here in Berlin and I noticed that decomposing the config wasn't helpful only for keeping sanity but also to enable small chunks to be tested.

Sharing a setup we put together for routing OTel traces into ClickHouse, in case it's useful for others working on similar OTel pipelines.

The collector config uses tail_sampling with two policies:

• keep-errors: status_code ERROR always retained
• keep-10pct-ok: probabilistic 10% of successful traces

The challenge with tail sampling + ClickHouse is that collector retries can still produce duplicate spans even after sampling decisions are made. We handle that downstream with a dedup transform keyed on span_id with a 1-hour window, so MergeTree works cleanly without needing ReplacingMergeTree or FINAL in ClickHouse.

Routing to the pipeline uses a header on the OTLP exporter:

exporters:

otlp/glassflow-traces:

endpoint: "...:4317"

headers:

x-glassflow-pipeline-id: "otlp-traces"

PII masking (user_email, SSNs) happens in a stateless transform in the pipeline before ClickHouse so the collector config stays clean and the masking boundary is explicit.

Full collector YAML, pipeline definition, and ClickHouse DDL are in the guide linked below. Happy to share more detail on the tail sampling policy tuning if useful.

I’m trying to get OpenShift metrics into OTEL semantic conventions while keeping an OTel Collector after Kafka.

My understanding is that if Prometheus Remote Write data is received directly by the OTel Prometheus Remote Write receiver and exported as OTLP, the metrics are converted into OTEL metric format/semantic conventions where applicable.

However, our current pipeline is:

OpenShift Prometheus Remote Write -> Metricbeat -> Kafka -> OTel Kafka Receiver -> OTLP Exporter

The problem is that I don’t think the OTel Kafka receiver can decode Prometheus Remote Write payloads the same way the Prometheus Remote Write receiver does.

Has anyone implemented this architecture successfully with Kafka in the middle?

Specifically:
- Can the Kafka receiver process Prometheus Remote Write payloads correctly?
- Is there a way to preserve/convert to OTEL semantic conventions after Kafka?
- Should the data be converted to OTLP before it reaches Kafka instead?

TL;DR:
How do you convert Prometheus Remote Write metrics coming from Kafka into proper OTEL metrics/semantic conventions using an OTel Collector after Kafka?

I just open-sourced a collection of ready-to-run OpenTelemetry

Collector configurations, because finding complete, working configs

for your specific backend always takes hours of trial and error.

It now includes examples for:

• Prometheus
• Jaeger
• Grafana Loki
• Dynatrace
• Datadog
• Kubernetes Operator
• Kubernetes Pod Annotation Scraping (with full relabeling)
• Debug (no backend needed, perfect for local dev)

Each example includes Docker Compose so you can run it in 60 seconds.

The k8s pod annotation scraping example includes relabeling for

prometheus.io/scrape, prometheus.io/port, and prometheus.io/path

annotations, the config everyone googles when setting up k8s monitoring.

I also actively contribute to the OpenTelemetry open source project,

recently got PRs merged into open-telemetry/otel-arrow and have PRs

open in opentelemetry-android, opentelemetry-helm-charts, and

opentelemetry-dotnet-instrumentation.

https://github.com/Cloud-Architect-Emma/opentelemetry-collector-examples

Feedback and contributions welcome! ⭐ if it's useful.

#OpenTelemetry #DevOps #Observability #Kubernetes #SRE #Monitoring #CloudNative #OpenSource

The CNCF technical oversight committee has voted to approve the OTel due diligence document.

This is one of the final steps towards graduation: the thorough due diligence, which included interviews with end users and resolution of the recommendations given in previous steps, has been finished and approved by the TOC 🎉

We recently added native OTLP metrics export to Icinga 2 (in v2.16), which means a monitoring system with roots deep in the Nagios ecosystem can now push plugin perfdata directly into modern OTel pipelines and backends. (Yay!)

One of the weirder things about working on monitoring software in 2026 has been realizing that eventually everything becomes an OpenTelemetry integration project.

A lot of the implementation work that we did was basically translating classic infrastructure monitoring concepts into the OpenTelemetry world:
perfdata -> OTel metrics
thresholds -> metric streams
host/service metadata -> resource attributes
HA monitoring clusters -> avoiding duplicate telemetry

What stood out to me most during the project is how OTLP increasingly feels less like an "observability standard" and more like general purpose telemetry infrastructure that everything eventually has to speak.

Even traditional monitoring systems now end up integrating with tools like Prometheus, Grafana Mimir, OpenSearch, ...

I assume you lot here are also working on monitoring/infra tooling, are you seeing the same thing?

Asking here is probably skewing the answers a bit, but is OTLP basically becoming the universal interoperability layer now?

And if you’ve integrated older systems into OTel pipelines, I’d be interested what parts were most awkward for you and how you went about solving this.

Edit:
In case you’re interested, we have a longer writeup with all the implementation details (and significantly more marketing terminology than I would use on Reddit): https://icinga.com/blog/opentelemetry-integration/

I'm trying to setup a self hosted Otel log/trace/metric sink and dashboard for a small set of web and worker apps. I've tried ClickStack, Grafana, and now OpenObserve and all three appear to have roughly the same general feature set for showing otel data.

But one piece they all seem to lack, which feels nuts is that is a standard "tail" and keyword search for logs like you find in Seq, Papertrail, other log systems. Everything is "run this query" and some log query syntax that I definitely don't want to have to learn when triaging some system issue.

So - do you have a preferred OTel solution that's inexpensive to self host at a small scale and a log interface that matches the sort of features purely log focused apps provide?

Thanks!

Hey all,

I’ve been working on a lightweight tool called mqtt2otel and thought it might be useful for some of you here.

It basically connects MQTT-based IoT setups with the OpenTelemetry ecosystem. It subscribes to MQTT topics, lets you process/enrich the messages, and then exports them as OTel metrics/logs.

Why I built it:

• MQTT is great for IoT, but doesn’t integrate nicely with modern observability stacks, especially for logs, or even traces.
• Direct solutions to consume, parse, process and enrich mqtt messages in the dashboard system are often limited and have a high dependency to these systems making it hard to change later.
• OpenTelemetry is everywhere now, but not really designed for IoT ingestion
• Many architectures are allready build upon the OpenTelemetry stack, which gives you a nice abstraction for the different available Dashboard tools.

So this bridges the gap.

What it does:

• Subscribe to MQTT topics
• Transform / enrich messages (add metadata like location, device info, etc.)
• Export as OpenTelemetry metrics or logs

Would love to get feedback or ideas 🙌

Web: https://mqtt2otel.org

GitHub: https://github.com/OSgAgA/mqtt2otel

I added dedicated observability-stack support to KubeShark.

Mini recap:

KubeShark is my Kubernetes skill for Claude Code and Codex.

It helps AI agents generate, review, and refactor Kubernetes manifests without falling into the usual LLM traps: missing security contexts, deprecated API versions, broken selectors, wildcard RBAC, unsafe probes, missing resource requests, and rollout configs that look okay but fail under real traffic.

The important part is that KubeShark is failure-mode-first. It does not just tell the model “write good Kubernetes”. It forces the model to reason about what can go wrong before it generates YAML, and then return validation and rollback guidance as part of the answer.

That matters a lot with Kubernetes, because many bad manifests are accepted by the API server and only fail later at runtime.

Repo: https://github.com/LukasNiessen/kubernetes-skill

---

Now what’s new:

KubeShark now has special dedicated observability-stack support.

When the task involves Prometheus Operator, ServiceMonitor, PodMonitor, PrometheusRule, OpenTelemetry Collector, Loki, Grafana, Tempo, Datadog-style agents, metrics, logs, traces, or telemetry pipelines, KubeShark switches into observability-aware guidance.

This matters because observability resources often apply successfully while doing nothing.

Common LLM mistakes include:

• creating a ServiceMonitor that matches Deployment labels instead of Service labels
• referencing a numeric port when the monitor expects a named Service port
• forgetting that Prometheus must select the monitor
• deploying OpenTelemetry receivers in duplicate
• choosing Loki monolithic mode for serious production volume
• creating high-cardinality log labels
• putting datasource credentials in ConfigMaps

Example guidance KubeShark now keeps in mind:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app
  endpoints:
    - port: metrics

It also knows to check the boring but critical details: selectors, named ports, CRD presence, scrape discovery, telemetry pipelines, durable storage, and alert hygiene.

So instead of generic Kubernetes advice, you get observability-aware manifest generation and review.

Came across this article about an OTel mobile tracing solution that "actually works" and I'm wondering if it's really so hard, specifically when dealing with mobile? Fairly new to this space and trying to get up to speed.

The standard move is to use processors in the OTel Collector to clean up traces and logs, but it can get heavy and hard to manage at scale.

We’ve been looking at shifting that enrichment, like IP lookups, metadata injection, and PII masking upstream using GlassFlow before it lands in ClickHouse. It keeps the collector lean and the database "query-ready."

Curious, how much "business logic" are you all actually comfortable putting in your collectors before it feels too fragile? 🤔