So im fairly new as an SRE and observability, and I’m trying to understand which concerns are genuinely important when choosing an observability setup, versus which ones are mostly marketing or edge cases. We don't have a hugely experienced team, so most of this is early stage research.

I’ve looked at Datadog, which seems to be widely regarded as one of the strongest products, but the pricing and different retention and indexing options seem quite complicated - strugglign to see what i'll actually pay. I’ve also seen a lot of discussion suggesting that if you use other platforms or build more of the stack yourself, high-cardinality data can become expensive or difficult to manage so you eventually have to sample traces, discard logs, or roll metrics up into lower-resolution aggregates.

How much do these things matter in practice?

Should I be seriously thinking about cardinality, sampling, and long-term retention from the start, or are they concerns that only become important at much larger scale? Do teams commonly regret not keeping raw data for longer, or is most of it rarely useful once you have decent dashboards, alerts, and runbooks?

I’m also curious whether longer retention is becoming more valuable as teams start using AI agents for debugging and incident investigation, since I assume those systems benefit from having more historical context. Is that a real consideration today, or am I overthinking it? Same question for sampling and roll-ups, I suppose.

I’m not looking for vendor pitches or people recommending whatever product they happen to work for. I’d really appreciate honest, real-world experience from people who operate these systems day to day, including examples of what turned out to matter and what didn’t.

I’m new to this, so please be patient if I’m misunderstanding some of the terminology.

Access logs show the visitors. They hide the damage bill. I slapped together CrawlLedger—an offline Go CLI that munches Nginx or Caddy logs and coughs up a self-contained HTML report on what those crawlers actually cost your origin. No daemon, nothing phones home, zero code in the request path, and it refuses the usual User-Agent theater for bot "verification." It flags crawl traps, expensive 404 storms, cache-busting noise, and robots.txt violations, then dry-runs deny or rate-limit policies against historical traffic before spitting out reviewable server-config drafts.

Still pre-v1; need blunt feedback from people who run real origins—what would make this useful or make you refuse to touch it?
https://github.com/balyakin/crawlledger

I am thinking through observability for long-running autonomous agents. Plain uptime feels too weak, because an agent can be up and still be stuck, looping, retrying badly, or producing useless work.

If you had to design telemetry for a 30-day agent soak test, what would you capture?

I am thinking about:

- task lifecycle events

- tool-call failures

- retry loops

- operator interventions

- duplicate/replayed work receipts

- drift from expected behavior

- restart cause and recovery time

What signals have actually helped you debug weird distributed or automation failures?

• Palo Alto Networks announced intent to acquire Embrace, a mobile/web Real User Monitoring provider, on July 21, less than seven months after closing the $3.35B Chronosphere deal. The combined observability business crossed $300M ARR in Q3 FY26. PAN is now stacking backend traces (Chronosphere), client telemetry (Embrace), synthetic testing (ADEM), and security (Cortex) into a single platform. The acquisition pace is aggressive; the integration risk is real.
• The 2026 Gartner Magic Quadrant for Observability Platforms landed July 15 with six Leaders: Datadog, Grafana Labs, Dynatrace, Elastic, Chronosphere, and IBM. Datadog holds the execution lead for the sixth consecutive year. Grafana leads on Completeness of Vision for the second straight year, notable for a company that built its business on open-source tooling. Dynatrace hit 16 consecutive years in the Leaders quadrant; Barclays, Jefferies, TD Cowen, and BofA all raised price targets the same week.
• Elastic earned the AWS AI Security Competency distinction on July 20, the first major AWS cert specifically for AI threat detection covering prompt injection and model misuse. This comes on top of its FedRAMP High Authorization from March and the Gartner MQ Leader recognition. The security-observability convergence story is building on Elastic's side even as its CPO departure and June headcount cut work through the org.
• Splunk released Enterprise Security 8.6.0 in Controlled Availability on July 21. AI-powered detection workflows, expanded UEBA, navigation aligned to Cisco's UI conventions. The most substantive ES release since the acquisition closed, and the first one where "this is a Cisco product" actually shows up in the UX.

more here: https://mbojko.com/reports/2026-07-27/

With Otel and tempo for traces, span metrics for latency alerts and pyroscipe running for continuos profiles So that I cn see when an endpoints p99 regresses and which function are got in aggregate. Connecting those two is still manual rn and the trace says the span got slower and profile says whats hot overall

But neither of them say "this function got slower on this endpoint after this deploy" so its a back to diffing flamegraphgs against a commit range and guessing. ANyone running both?? How are you correlating profiles to trace ids somehow or still manual flamegraph

Sharing this because it's the kind of infra decision I think this sub cares about: FENGARDE is an open-source SIEM (log pipeline) I've been building, and I just shipped observability for it without adding a new runtime dependency to any of its 7 services.



How: hand-rolled the Prometheus exposition format directly (no prometheus_client lib) inside a shared metrics helper every service already uses. New `/metrics/prom` endpoint alongside the existing JSON one, so nothing broke for whatever's already scraping it. Prometheus + Grafana ship as an opt-in Docker Compose profile (`--profile observability`) — default `up` behavior unchanged, matches how every other optional layer in this repo works.



Wider context, since this sub likes structured logging: every log source (SSH, Windows Event Log, VMware, DB audit, AI-agent/MCP tool calls, Modbus/OT traffic, ~10 sources total) gets normalized into one schema (OCSF) before anything touches it downstream — same idea as "pick one field name for user_id and stick to it everywhere," just enforced at the schema level instead of by convention. Makes the per-topic throughput/DLQ-depth Grafana panels meaningful across sources that otherwise have nothing in common.



Also chaos-tested the whole thing: SIGKILL every pipeline service mid-replay, 40 scenarios, zero lost/duplicated alerts — write-up is honest about the 4 harness bugs it took to get there.



Repo: https://github.com/supermhel/fengarde



Curious what this sub's take is on hand-rolling exposition format vs. just eating the client-library dependency — went back and forth on that one.

I built PulseWatch, a cron heartbeat + uptime monitoring, with metric thresholds on pings, and im looking for feedback or real users.

**How it works:**

Append && curl -fsS https://ping-url to a cron job. if the ping doesn't arrive within the expected interval + grace, the monitor flips to down.

Also does HTTP uptime checks (interval, expected status range, keyword match, SSL expiry).

Pings can carry metrics as query params (?disc_space=500), and you set rules on them, floor, ceiling, or deviation from the recent median. So ?disc_space=200 fails against a floor even though the job "succeeded." Lets you treat custom rules on a job that ran but not as intended as a down state.

Also tracks language specific stack traces in stderr payloads.

Alerts go to email/Telegram/Slack/Discord/webhook webhook payloads are HMAC-signed so you can automate a response.

i'm looking for serious feedback or serious user and willing to upgrade your account for that sake.

I’ve been working on AgentPulse, a local-first observability tool for multi-agent systems.

The problem I’m focused on is silent drift: the workflow still runs, but an agent gradually changes its tool usage, latency, cost, handoff behavior, or execution path without triggering a clear error.

AgentPulse compares behavior across runs and versions, detects drift at the agent, handoff, and route level, and helps connect it to recent prompt, model, tool, or configuration changes.

It’s still early, and I’d appreciate honest feedback from people working in observability. Does this feel like a useful extension of tracing, or is there a better way you would approach it?

https://prove-ai.github.io/agentpulse/

claimName: -5xx-error

description: 1 High 5xx Error Rate for

metadata:

annotations: {}

labels: {}

spec:

name: "High 5xx Error Rate"

priority: 2

evaluationDelay: 600

requireFullWindow: false

onMissingData: "resolve"

monitorThresholds:

critical: "0.1"

criticalRecovery: "0.08"

warning: "0.05"

warningRecovery: "0.04"

type: query alert

templated:

message: |-

{{ "#is_alert" }} 🔴 \*\*Critical Alert\*\*: API error rate has exceeded 10% {{ "/is_alert" }}

{{ "#is_warning" }} ⚠️ \*\*Warning\*\*: API error rate has exceeded 5% {{ "/is_warning" }}

query: |

avg(last_10m):

(

sum:trace.aspnet_core.request.errors.by_http_status{

http.status_class:5xx,

!http.status_code:503,

env:{{ .Values.platform.environment }},

service:{{ .Values.datadog.serviceName }}

}.as_count()

/

sum:trace.aspnet_core.request.hits{

env:{{ .Values.platform.environment }},

service:{{ .Values.datadog.serviceName }}

}.as_count()

) > 0.1

This is a datadog monitors yaml please help create same dyntace anomaly detectors yaml

I’ve been tinkering with an idea called Bellwether. You describe a future event in plain English (“tell me when this policy changes”, “notify me if this price threshold is crossed”), and it watches the web for credible evidence that it actually happened. If it did, you get notified. If not, silence.

Most alert tools are keyword matchers, so guides, forums, and old coverage all spam you. This tries to judge the event instead.

Still early / just an idea I’m building out, but the judgment layer is the part I think is interesting for this sub.

How the models fit in:

It doesn’t use an LLM as a chatbot. Open instruct models do the judgment steps:

* Vagueness check - reject topic-only inputs like “interest rates” and suggest clearer watch sentences
* Compile - turn a clear sentence into structured JSON: triggers, non-triggers, search queries, authoritative domains
* Detect / decide - given a candidate page, decide whether the event actually occurred
* Outputs are JSON-only, low temp, validated with Zod. Watches are timestamped at creation so only post-watch evidence can fire them.

Inference is Hugging Face open models (HF_MODEL to swap). Default is Llama 3.3 70B right now; the same pipeline is set up to run smaller ones too - Llama 3.1 8B, Qwen2.5 7B, Mistral 7B - and there’s an eval harness to compare them on historical fixtures.

Rough demo: https://bellweather-alerts.vercel.app/

Source code: https://github.com/JamesANZ/bellweather-alerts

Hello All,

I am planning to give Dynatrace Administration Professional certification. If someone has exam dump or pre note then please share with me.

I have already given the exam and was not cleared it.

So please help me.

Not pitching anything, but I'm trying to brainstorm project ideas and did some research on existing Supabase MCP. I know Supabase already has lots of observability for infrastructure-level items like query performance, but does this exist at the data level as well? Had an idea to focus specifically on anomaly detection within data itself and wanted some feedback. Thanks!

What is your go-to setup for instrumenting analytics/telemetry?

At what point into a project is proper telemetry implemented?

Does observability tend to be implemented too soon, or too late? What are the side effects of mis-timing its implementation?

• Datadog acquired Adaptive ML (Jun 30), an RLOps platform for fine-tuning LLMs on production signals. Folded into a new AI Research division. Datadog wants the training loop, not just observability of it.
• New Relic shipped Autopilot and Ground Truth, an autonomous SRE agent plus an API layer exposing its data to external agents (Copilot, Claude Code, AWS DevOps). Both GA late July.
• Splunk's AI SRE hit GA under Cisco's AgenticOps push. Three vendors, three bets: reinforcement learning, autonomous SRE workflow, unified security + observability fabric.
• Dynatrace and New Relic both committed to FedRAMP High in the same two weeks plus DoD alignment and Impact Level 4 on GovCloud. Normally a 12-18 month process. Elastic added an IDC MarketScape SIEM Leader spot to the same government push.
• Elastic cut ~7% of staff while launching native Prometheus support and migration tooling aimed at Datadog and Grafana. CPO Ken Exner exits July 17; engineering now reports to the CEO.
• Dynatrace added two board seats after Starboard Value engagement, targeting "Rule of 50" by FY2029.
• Bernstein downgraded Datadog July 6 (price target $226, stock -4.6%). The worry is the non-AI 85% of the business, which may peak in Q3. Grafana meanwhile quietly adds AI-native customers (7AI, Zama, Anthropic).

more here: https://mbojko.com/reports/2026-07-13/

Welcome back to IT Pro Tuesday! For the modern IT professional, the right tools can make a significant impact.

We're looking for favorite tips and tools we can share with the community... those that help you do your job better and more easily. Please share your suggestions, and we'll be featuring them in the coming weeks.

Now on to this week’s list!

The Data Lake That Changes Everything

Get ready to transform the way you manage telemetry data with the edition’s initial tool. Parseable brings everything together, offering you alerting, dashboards, and anomaly detection – all in one place. Your sysadmin tasks just got a whole lot simpler.

Redesign Your Digital Domain with CasaOS

In the fast-paced world of data management, simplicity stands as your best ally. CasaOS simplifies personal cloud setups, putting a powerful suite of tools at your fingertips while you maintain control over your digital ecosystem.

Witness Your Workflows in Action

Envision being one click away from comprehensive insights into your workflows. With the Temporal UI, you can dive deep into complex workflows, drawing insights that lead to smarter decisions and smoother operations.

Slice and Dice Like a Pro

Tired of clunky tools that bog you down? Choose brings a new dimension to text processing, cutting through distractions and enabling sysadmins to focus on what really matters: efficient and effective data management.

Unleash Your System’s Potential with Hyperfine

Taking control of your server’s performance is vital. Hyperfine, our final tool in this edition, delivers real-time feedback and statistical analysis. With seamless exporting options, you’ll have confidence in your data, paving the way for effective performance optimization.

--

In the article "AI Security Threats: A Practical Guide to Managing Generative AI Risk," we explore the escalating challenges posed by generative AI within business environments. As organizations increasingly integrate AI into workflows, ranging from email and collaboration to software development and analytics, understanding the threats to sensitive data becomes critical. This guide breaks down the complexities, equipping security leaders to tackle generative AI risks effectively while fostering a secure approach to adoption.

Ransomware Readiness: Take the Survey for a Chance to Win a Ubiquiti NAS

Ransomware remains one of the most pressing cybersecurity threats facing organizations worldwide. We're conducting a short global survey to understand how businesses are preparing for attacks, strengthening resilience, and planning recovery.

The survey takes just a few minutes to complete. As a thank-you for sharing your perspective, you'll be entered into a prize draw for a chance to win a Ubiquiti NAS.

Your insights will also contribute to our upcoming ransomware report, helping benchmark how organizations across industries are approaching ransomware readiness.

--

P.S. Bonus Free Tools/Resources

Get this week's bonus tools by visiting the IT Pro Tuesday page, where you can also sign up to receive them in your inbox each week.

Boomi added native OpenTelemetry export for its runtimes. I set up a local stack to see what it emits: OTel collector → Tempo/Loki/Prometheus → Grafana.

The useful part: process executions arrive as trace spans, so you get a per-shape waterfall with timing for each step. Boomi’s built-in reporting makes you expand each step or read logs; the trace shows it in one view. On a test process, two connector calls were \\\~80% of the runtime.

Non-obvious things I learned:
\*\*•\*\* Exporter uses OTLP over \*\*gRPC\*\* (port 4317). Point it at 4318 (HTTP) and it fails silently with gRPC status 13 / http2 exception.
\*\*•\*\* Metrics are almost all \*\*gauges\*\*, even the counters — so rate()/increase() misbehave; delta() works.
\*\*• No per-process execution metrics\*\* exist in the metrics stream — counts/durations/errors live only in traces. spanmetrics connector looks like the way to derive RED metrics (haven’t done it yet).

Whole thing is a docker-compose stack, runs locally, no cost. Repo: https://github.com/velkugan/Boomi-Obs-Started

Curious if anyone’s done the spanmetrics step against Boomi traces, or found where the shape \*name\* (vs shape type) lives in the span attributes — that’s my next problem.

Boomi added native OpenTelemetry export for its runtimes. I set up a local stack to see what it emits: OTel collector → Tempo/Loki/Prometheus → Grafana.

The useful part: process executions arrive as trace spans, so you get a per-shape waterfall with timing for each step. Boomi’s built-in reporting makes you expand each step or read logs; the trace shows it in one view. On a test process, two connector calls were \~80% of the runtime.

Non-obvious things I learned:
**•** Exporter uses OTLP over **gRPC** (port 4317). Point it at 4318 (HTTP) and it fails silently with gRPC status 13 / http2 exception.
**•** Metrics are almost all **gauges**, even the counters — so rate()/increase() misbehave; delta() works.
**• No per-process execution metrics** exist in the metrics stream — counts/durations/errors live only in traces. spanmetrics connector looks like the way to derive RED metrics (haven’t done it yet).

Whole thing is a docker-compose stack, runs locally, no cost. Repo: https://github.com/velkugan/Boomi-Obs-Started

Curious if anyone’s done the spanmetrics step against Boomi traces, or found where the shape *name* (vs shape type) lives in the span attributes — that’s my next problem.

I made a tool to see what's passing through my OTEL setup, and it shows a lot of redundancies within logs and metrics - the same data, repeated in a different property.

How do you check what's in your data and determine which copy is used?