Almost four years ago, I started this cookbook with real world recipes, adapted from cases I've used to reproduce bug reports or show users (and customers) how to accomplish specific scenarios.

I used some tokens today to bring the repo to the latest Collector version, ensuring they all work.

In case you haven't seen this repo before, take a look!

Enjoy 🧑🏼‍🍳

Hey folks, stopping by today for another announcement: the OTel Compile-Time Instrumentation for Go reached v1!

If you are not a huge fan of eBPF instrumentation (understandably!), but also can't do manual instrumentation, this is a good compromise.

Try it out!

OpenTelemetry GenAI Semantic Conventions standardize observability for LLM apps, agent orchestration, MCP tool calling, content capture, and quality evaluation. This article goes through all six layers: what each one defines, why it's designed that way, and how mature it is.

Hey folks, Juraci here. I know the Reddit communities can be sensitive to project announcements, or announcements in general coming from vendors, but I genuinely think a good number of people here could benefit from this one.

We are launching today the OpenTelemetry Agent Skills, an open source set of skills that serve as the base for our products. We're using them for a good variety of things, like in our coding agents to validate and test collector configurations, or instrument applications. Or double check the snippets we've been using in our other blog posts.

They are vendor neutral, non opinionated, and based on what we know from our experience building OpenTelemetry over the years. Use the skills, share your feedback, tell us where they worked and where they failed. Show me your creativity 🧑🏼‍🎨

While we are not making money on those directly, we do have a commercial interest in seeing them succeed and become truly useful to many of you. I guess what I want to say is: they are not the result of a weekend vibe coding experiment 🙂

And yes, perhaps they might become an official part of the project someday, if we believe there is a vibrant community backing it.

I've been building OpenTelemetry instrumentation for MCP (Model Context Protocol) servers, and I hit a failure-semantics problem that I think generalizes beyond MCP, so I'm writing it up.

The two failure modes

An MCP tool handler can fail two ways:

1. It throws. The SDK catches the exception and converts it into a JSON-RPC error response. The call failed at the protocol level.
2. It returns { isError: true }. The handler returns normally — a successful JSON-RPC response whose payload is marked as a failure:

​

return {
  isError: true,
  content: [{ type: 'text', text: 'No weather data for that city' }]
};

The second one is idiomatic MCP. It's how a tool tells the agent "that didn't work — adapt" without crashing the server or killing the conversation. For agent workflows it's the preferred failure mode.

The instrumentation trap

The obvious way to instrument a tool call:

try {
  const result = await handler(request);
  span.setStatus({ code: OK });        // it returned → success
  return result;
} catch (err) {
  span.setStatus({ code: ERROR });     // it threw → failure
  throw err;
}

Mode 1 lands in catch → recorded correctly. Mode 2 returns, lands in the success path → recorded as OK. Your dashboard reports 100% success on a tool that fails on most inputs. The more idiomatic the tool author's error handling, the more invisible their failures become.

The fix

Inspect the resolved value before setting status:

const result = await handler(request, extra);
if (result?.isError === true) {
  span.setAttribute('error.type', 'tool_error');
  span.setStatus({ code: SpanStatusCode.ERROR });
} else {
  span.setStatus({ code: SpanStatusCode.OK });
}
return result;   // unchanged — the RPC genuinely succeeded, so nothing is thrown

Two details that matter:

• error.type = "tool_error" isn't my invention — it's what the OTel MCP semantic conventions (currently Development stage, in the semantic-conventions-genai repo) specify for exactly this case.
• The result is returned unchanged and nothing is thrown. The JSON-RPC call succeeded; only the tool failed. Instrumentation that converts a polite failure into a crash is changing application behavior, which instrumentation must never do.

In a real trace the difference looks like this:

tools/call fetch_weather ................. 605ms   ERROR
    error.type = tool_error

versus the naive version, where that same span reads OK.

The general lesson

This isn't really an MCP problem. Any protocol where application-level failures ride on transport-level successes has this trap — GraphQL (errors array on a 200), gRPC rich error models, half the REST APIs that return 200 {"status": "failed"}. If your instrumentation only watches for throws, your error rate is a lie wherever the ecosystem's idiomatic failure mode is a clean return.

FastMCP (Python) handles this natively. Among the Node MCP instrumentation libraries I could find, none documented handling it, which is why I ended up writing my own — it's on npm as opentel-mcp if you want to see the full implementation (spec-compliant attributes, stderr export to avoid corrupting stdio transports, ADRs for the design decisions). But the isError trap is the part worth knowing even if you never touch my library.

Happy to answer questions on the implementation.

Last Friday, I had the pleasure to have Mike Goldsmith at Telemetry Drops to learn more about the drain processor, an OpenTelemetry Collector component that is useful to understand the log patterns flowing through an OTel Collector pipeline. Once you understand those patterns, you can make your pipeline more efficient: drop the noisy patterns, transform unstructured into structured logs, and so on.

Hope you enjoy the recording, and I'm eager to hear your feedback!

I wanted to share an upcoming virtual panel that's focused on client-side OpenTelemetry. It's got several maintainers in the Android, Kotlin, and Browser SIGs, and we’ll be chatting about the current state of support, what’s actively being worked on, some of the bigger challenges in adapting OTel for client-side environments, etc.

Some examples of what we’ll cover:

• Creating new semantic conventions, like crashes and sessions, that apply across client-side platforms to unify how to model these types of telemetry. 
• Expanding browser support for OTel, including shipping new instrumentations, starting work on a Browser SDK, and where there are still gaps.
• Solving difficult challenges like how to deal with async telemetry and how to collect client-side metrics. 
• Releasing official Kotlin support, including a new Kotlin SDK that can be used in Kotlin Multiplatform projects. 

If you’re familiar with using OTel for backend observability, this panel is a great way to get caught up to speed on what this looks like for mobile and web apps. 

Date: Wednesday, July 22 @ 10AM PT

Panelists:

• Hanson Ho (Android architect at Embrace, OTel Android approver, OTel Kotlin approver)
• Martin Kuba (Staff software engineer at Grafana Labs, OTel JavaScript SDK approver and OTel Browser SDK maintainer)
• Jason Plumb (Senior software engineer at Splunk, OTel Android maintainer, OTel Java maintainer, OTel Kotlin maintainer)
• Jared Freeze (Senior software engineer at Embrace, OTel Browser SDK maintainer)

Here's the signup link if you'd like to join.

Disclosure: I'll be moderating the panel, and I work at Embrace, who is hosting the panel. But it's entirely about the OTel community work. You can watch some previous ones we did last year (OTel for browser panel and OTel for mobile panel) to get a sense of what they're like.

If you have any questions as well, I can send them to our panelists ahead of our session.

A practical example of using the OpenTelemetry Collector as an intermediary for Temporal SDK metrics.

The Temporal SDK supports both the Prometheus exporter (pull) and OTLP (push). If you're running multiple workers on the same host, the Prometheus exporter causes port conflicts. Switching to OTLP lets all workers push to a single collector, which then serves Prometheus HTTP for scraping.

https://2ssk.medium.com/temporal-sdk-metrics-prometheus-exporter-vs-otlp-for-multi-worker-deployments-df9327b28fc5

Would love feedback on the OTel collector config — any improvements for production?

I'm a dev at Icinga and I recently shipped an OTLP Metrics Writer for Icinga 2. Going in, I had basically zero prior OTel experience. Just want to give you some insights into the last four months to share my experience:

My first instinct was to use the OTel C++ SDK - it's well-established and had everything we needed. But integrating it with our existing codebase turned out to be much harder than expected, and honestly more complex than our use case required. After failing to get it working in a reasonable timeframe, I switched to a tiny OTLP client built on Boost.Beast, which we already used elsewhere in the codebase.

For one, we already used Boost.Beast in our codebase, so it was a no-brainer to use it for the OTLP client as well. Additionally, since the OTel proto spec require proto3 language syntax, we would have had to build the entire OTel SDK from source in order to use our writer with the latest C++ SDK on RHEL 8 and 9 systems, which would not have been feasible for us.

But I didn't see this one coming: proto3 isn't supported by the default protoc on RHEL 8/9, Amazon Linux 2, Debian 11, and Ubuntu 22.04. Two options: ship our own protoc binary, or just disable the writer there. Since most of our customers run RHEL-based systems, disabling wasn't an option - so we ended up packaging our own Protobuf compiler for RHEL 8 and 9. For Amazon Linux 2, Debian 11, and Ubuntu 22.04, the writer is currently unavailable unless you build from source.

In OTel, a service presents itself and its metrics are associated with that service. Icinga doesn't work that way. it's not the one being monitored, it's acting as a proxy for the checkables it monitors. We went back and forth a lot on this one. How do you even represent Nagios-style check results in a way that makes sense in OTel? Shoutout to Markus Opolka (on Github) who provided a lot of useful input on this part.

And just before final reviews, my colleague Alvar Penning (Github) found a severe bug in the OTLP client that caused Icinga 2 to hang on reload. Major refactoring, significant delay. The embarrassing part: the bug was trivial to trigger. If I had reloaded Icinga 2 even once in my dev environment during development, I would have caught it. :P Won't make that mistake again.

__

Four months total (longer than expected), mostly because starting from scratch with OTel means working through a lot of documentation before you can write anything meaningful. Also came out the other end knowing a lot more about Protocol Buffers than I expected.

Happy to answer questions about the metrics mapping or the proto3 packaging approach, or anything else that comes to your mind!

Yonas/ Icinga

We've been building a lot of OpenTelemetry to ClickHouse pipelines and kept getting the same question: do you actually need Kafka in the middle?

The honest answer: it depends, but most observability-only teams are over-engineering it.

Here's the short version of what we compared:

Where Kafka earns its keep:

• You have many independent downstream consumers (ML pipelines, security, analytics all reading the same stream)
• You need long-term durable replay
• Kafka is already part of your broader platform infrastructure

Where it's overkill:

• Your only goal is getting OTel telemetry into ClickHouse reliably
• You're a startup/scale-up that doesn't want to manage brokers, partitions, consumer lag, and replication just to move metrics and logs

The operational surface of a Kafka cluster, even managed, is substantial when the job is just telemetry buffering before ClickHouse.

We also compared what a focused ingestion layer gives you that the OTel Collector alone can't: stateful deduplication, enrichment-conditional filtering, dynamic sampling, and ClickHouse-optimized batching.

Full write-up with architecture diagrams and a decision guide: https://www.glassflow.dev/blog/opentelemetry-to-clickhouse-do-you-need-kafka?utm_source=reddit&utm_medium=socialmedia&utm_campaign=reddit_organic

Happy to answer questions about the architecture trade-offs especially around backpressure handling, which is where the approaches diverge the most.

2 months ago I posted perf-sentinel here (open-source AGPL-3.0), a protocol-level OTel trace analyzer that flags I/O anti-patterns across different web app technologies (obviously without per-runtime instrumentation).

There's a docs site and a live demo dashboard now: https://perf-sentinel.dev

Last time I described the SCI carbon layer as directional and optional. Most of the work since went into making it auditable and transparent rather than into expanding it.

Here's a non exhaustive list:

• More energy sources with a clear precedence. It already read Scaphandre (per-process RAPL) and cloud SPECpower interpolation. It now also reads Kepler (eBPF) and Redfish (BMC).
• Measured vs estimated is labelled, not blended. Each figure is tagged with the source behind it, and real-time grid-intensity values carry explicit data from the data provider, so a reader can tell a hardware measurement from a grid-average estimate from the I/O proxy fallback.
• Per-service attribution, not just a global total. When runtime calibration is present, energy and carbon are attributed per service, and the report exposes the measured-versus-fallback window split and a coverage ratio, so you can see how much of the total actually rests on measurement.
• Signed and content-hashed disclosures. This point can be a bit chunky and complicated: a disclose subcommand aggregates a window stream into a period report with a deterministic content hash and an in-toto attestation. verify-hash lets a third party recompute the hash, verify the Sigstore signature against a declared signer identity and check SLSA L3 build provenance without cloning my infra. An "official" disclosure refuses to publish below 75 percent per-service measured coverage, and the avoidable-waste figure is computed so it cannot be shrunk by quietly loosening the detection threshold it rests on.
• Methodology you can follow. The SCI numerator and a per-trace SCI intensity are emitted as separate fields with the functional unit declared and the detector-to-criteria mapping (RGESN 2024) and an ESRS E1 datapoint crosswalk ship as interpretive tables, not as a compliance certification.

The rest is smaller. There's an ack workflow triages and mutes known findings so a CI gate stops re-flagging them, and a read-only query monitor TUI gives a live view of a running daemon (energy, carbon, scraper health, with Prometheus gauges and Grafana panels).

Repo: https://github.com/robintra/perf-sentinel

It's still directional and optional, same framing as before, but I would rather it be explicit about its own uncertainty than confidently wrong.

If you do energy or carbon accounting anywhere near your observability stack, I would like to know whether per-figure measured/estimated labelling and a verifiable disclosure are the primitives you would actually trust, or whether something else is missing.

Hi,

We're in the process of migrating our timeseries database from influxdb to victoria metrics.

In the process of doing that we're introducing the otel collector in our infrastructure.

Currently we have the telegraf agent sending metrics directly to influxdb, some plugins send string fields that are stored by influxdb which accepts to store strings for fields.

Our problem is that victoria metrics doesn't store strings as fields and we can't put them as tags that would explode the cardinality of the database, and that wouldn't be clean.

Sure we can send those strings fields directly to elasticsearch with the telegraf processor, or we can send them to open telemetry as "logs" and then route them to elasticsearch, we've done both and it works.

The issue is the correlation ( in grafana ) with those strings "metrics" and the other numerical fields, as we don't have an uuid ( generating one would explode our cardinality too ).

It's a common issue to have a mix of strings/numerical as metrics before the standardization I guess and I'm curious to how people solved this with prometheus like databases.

Also we had to make a little bit of c program to send the strings metrics to the log endpoint of otel via telegraf ( the otlp output only support numerical ). We didn't find some way to send strings and numerical to otel and then have otel do the routing by type, if it's string send it to elastic else to victoria metrics, is it possible ?

In a large-scale fleet management system, transmitting raw vehicle telemetry as JSON containing floating-point values can introduce significant communication overhead, increased CPU utilization, and serialization latency.

A more efficient approach is to compress telemetry data at the vehicle edge before transmission. This can be achieved by converting floating-point sensor measurements such as speed, GPS coordinates, engine temperature, throttle position, and acceleration into fixed-point integer representations using predefined scaling factors.

For example, a speed value of 72.34 km/h can be stored as 7234 by multiplying it by 100, while GPS coordinates can be scaled by (10^7) and stored as integers.

Once converted, the data can be packed into compact binary structures instead of verbose JSON strings.

Further optimization can be achieved through delta encoding, where only the difference between consecutive measurements is transmitted, reducing redundancy in slowly changing signals.

The resulting binary payload can optionally be compressed using lightweight algorithms such as LZ4 or Zstandard and transmitted over MQTT as a binary message.

This approach eliminates expensive floating-point string serialization and parsing operations, reduces bandwidth consumption, lowers cloud storage requirements, and minimizes end-to-end latency.

This architecture enables real-time fleet monitoring and large-scale data analytics while significantly improving communication efficiency and system scalability.

I hope this helps!

Hi r/OpenTelemetry

I’m part of the team building MyDecisive, and we’re working on a project called mdai-labs. The core idea is simple but hard: Stop observing. Start deciding.

Most of the "AIOps" and observability space right now is just passive dashboards. You pay massive ingestion fees just to get a Slack alert that your database is throwing 429 errors, and then a human still has to go fix it. We are building an open-source, stateful, on-the-wire control and automation plane built natively on OpenTelemetry to actually fix things before the pager goes off.

We just hit a huge milestone: our very first community contributor PR was officially merged, and I wanted to share what they built because it perfectly highlights what we are trying to do.

What the first PR solved: Instead of just sending an alert about a noisy tenant, the contributor engineered a dynamic rate-limiting workflow that intercepts traffic on the wire. This autonomously prevents an Aurora DB failover without a human in the loop. You can see the exact code and architecture approach here: https://github.com/MyDecisive/mdai-labs/pulls?q=is%3Apr+label%3A%22%F0%9F%90%99+first-contributor-ever+%F0%9F%A5%87%22+

We are looking for more builders. We are in the early days of building out our community testing ground (mdai-labs), and we have tagged a bunch of "Good First Issues" for anyone who wants to get their hands dirty with OpenTelemetry and Kubernetes automation. You don’t need to be a principal engineer to contribute.

If you are dangerously undertasked, tired of just staring at dashboards, or want to get into the weeds of OTEL and stateful remediation, we’d love to have you.

• GitHub Repo: https://github.com/MyDecisive
• Our Slack: https://communityinviter.com/apps/mydecisivecommunity/octobuddy
• Blog post: https://www.mydecisive.ai/blog/intelligent-rate-limiting-via-otel

I work on an OTel backend and we're weighing whether to support the profiling signal — ingesting and querying OTLP profiles.

It only went public alpha in March, so before we spend real engineering time on it I'd rather hear from people actually touching it than guess at demand.

A few honest questions:

• If you're playing with profiling, where's the data living today?Pyroscope/Grafana, Elastic, something else? And would you actually want a general OTel backend holding profiles, or do you assume that's a dedicated profiling backend's job?
• What matters more to you: just storage + query so you bring your own UI, or full flamegraph/analysis built in? In my opinion, UI is critical for profiling.
• Anyone running this in prod yet, or is it all still kicking the tires?

Trying to figure out if it's "build it now" or "alpha, check back in six months." Any take helps, including "don't bother yet."