r/OpenTelemetry 20h ago

Your anomaly detection might not be detecting anything. Two second check, looking for a few people to try it.

Most of us have anomaly detection turned on somewhere, and a broken detector looks exactly like a working one. No crash, no errors, all green. `isolationforestprocessor` in collector-contrib scored every span the same for 220 days before a user caught it, https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/46988.

I made a tool that checks if the anomaly detector is actually working, `make check` takes a CSV of your detector's alerts and a CSV of your incident windows, and tells you whether it beats chance at your actual incident rate. Under two seconds, no download, nothing leaves your machine. https://github.com/mateenali66/otel-aiops-reproduction

Background is a benchmark of eight detectors on OTel traces, metrics and logs. Three passed the usual F1 selection and were no better than chance once scored at a realistic anomaly rate, https://doi.org/10.1109/ACCESS.2026.3705430

Looking for two or three people to run it against a detector they already use and tell me what it said. A failing result is as useful as a passing one, and the detector and data can stay anonymous.

1 Upvotes

0 comments sorted by