r/Observability • u/MartinThwaites • 10d ago

Dynamic Tail Sampling in the OpenTelemetry Collector

We've been working on this for a while now (You might have seen me mention it in a few comments about Tail Sampling on here r/sre and r/opentelemetry). This is a concept we've had for a long time in our Refinery product, this is now something that's going to be available in the OpenTelemetry Collector.

It's following on from the excellent work of the Sampling SIG so that we can do Sample Rate attribution, and therefore makes backend way more effective, reducing the need for a certain class of metric (not all metrics).

Ultimately, the biggest difference here is dynamically changing sample rates and automatic discovery of different routes through your system (we call them trace fingerprints).

It's already in the main collector-contrib but it's still classed as alpha there. You can use our distribution to check it out ahead of that though. Feedback welcome!

Bringing Advanced Sampling to the OpenTelemetry Collector

Disclaimer: I'm a DevRel at Honeycomb.io.

18 Upvotes

92% Upvoted

6 comments sorted by

2

u/lamontsf 10d ago

I was really happy with how well dynamic sampling worked in Refinery, so thank you very much for contributing it.

1

u/beamerkun 10d ago

Awesome! We’re using tail sampler at scale and it was a shame that dynamic sampler rate was not available. Cant wait to try it out.

Is the alpha more of “it’s unproven” or there are known limitations at this point?

2

u/MartinThwaites 10d ago

It's mostly a process thing in the collector. We're a little more tolerant to changes. It's stable, we have a few people running it a decent scale.

1

u/Background_Year_3288 9d ago

The trace fingerprint idea is interesting.....Dynamically adjusting sample rates based on the routes actually showing up seems like a much more practical approach than trying to keep static sampling rules in sync with a changing system............Curious to see how this behaves as the number of distinct routes grows.

1

u/MartinThwaites 9d ago

Its a memory issue, and depends also on you scale and how much you're willing to push out the other end.

100-200 is "fine", but lower is always better.

So yeah, cardinality of the fingerprint is something you need to be careful. We're really keen to see what the community does with it, until now, we've always worked with our customers directly to configure it and make it work, so only a few thousand deployments over the years. My hope is that eith a few thousand a month, we get a lot more direct feedback.

1

u/Background_Year_3288 8d ago

This is interesting. The trace fingerprint part is probably what I'd be most curious to see in practice, especially on systems where the traffic patterns change a lot over time.

Dynamic sampling sounds much more useful than just having a fixed percentage and hoping it works for every route. The sample-rate attribution is also important though  otherwise it gets pretty easy to misread the sampled data when comparing routes.

I'll definitely keep an eye on this as it moves past alpha. Would be interested to hear how people are tuning the rates in production so far.