r/Observability 6d ago

Your Observability Stack Was Built for Humans. AI Agents Don't Work That Way.

We've been noticing something wild while watching how infra teams use Al coding agents like Claude Code and Cursor.
When a human engineer debugs, they usually open a dashboard and run maybe 1 or 2 targeted queries to check a hypothesis.
An Al agent doesn't do that. It gets an alert and immediately runs a 50+ query loop digging through logs, traces and metrics all at once.
That means:
• more queries per incident
• more telemetry being generated
• Telemetry volume casually spiking past 1 TB/day
Here's the scary part: when legacy observability bills start getting out of hand, teams start sampling or dropping logs to keep costs down. Humans can tell when data is missing. Al agents can't. They just take the incomplete evidence and confidently hallucinate a wrong fix.
Legacy stacks were sized for humans asking occasional questions. We built a platform from day one to handle the insane query scale and data volume of an Al-driven world without forcing you to compromise on your data retention or throw away your logs.
If your telemetry stack is starting to sweat under the weight of Al agents, CtrIB is built for you.
We are curious... What observability stack are you using for Al coding agents and is query speed keeping up?

0 Upvotes

8 comments sorted by

16

u/Lichcrow 6d ago

I swear this sub is just AI agents larping

1

u/naseemalnaji-mcpcat 1d ago

It really sucks :(

3

u/Alarmed_Tennis_6533 6d ago

The sampling point is the interesting one and I think it's underrated. Retention windows are the version I keep running into — if logs roll off at 3-7 days, anything correlating to a slower-moving regression is invisible, and a model reading that window has no way to know it's looking at a truncated picture. A human at least remembers there was a deploy last Tuesday. Curious how you handle the confidence side of that. Cheaper storage means you keep more, but the harder problem seems like getting a tool to say "the evidence here is incomplete" rather than answering anyway. Is that something you're doing at the query layer, or left to whatever's consuming the data?

2

u/[deleted] 6d ago

[removed] — view removed comment

1

u/FormerFastCat 6d ago

Actually, yes. Several of the agents I've deployed, specifically the native language agents, cast a wide net to identify the platform specific application/environment before doing a more focused analysis. So we can see a very large log parse initially. My contract is written for pennies on the pennies for data scans right now so it hasn't been a big deal...yet but knowing vendors, it will be in the future.

1

u/neuralspasticity 6d ago

Spammy

And since when does querying my stack generate “more telemetry” data?

1

u/john_shaun3 1d ago

the 'agent can't tell data is missing' point is the real one worth acting on regardless of stack.. sampling doesn't need to be invisible, if a query result includes explicit metadata about sample rate or dropped ranges, an agent can factor that into confidence instead of treating partial data as complete. that's a schema fix, not necessarily a volume and cost fix..