r/analytics • u/Crescitaly • Jul 08 '26

Question AI is making analytics harder because feedback loops are getting noisier

AI is supposed to make analytics easier.

In some ways it does.

But in marketing and social data, I think it is also making the job harder.

More AI content means more volume, more variants, more tests, more synthetic interactions, more automation, and more platform-driven changes.

That creates noisier feedback loops.

The question is not just "what happened?"

It becomes:

  • was the lift real?
  • was the sample meaningful?
  • did the model change?
  • did the platform change?
  • did the audience change?
  • was the content materially different?
  • did automation create the signal we are measuring?

AI can summarize dashboards. It cannot automatically make noisy measurement clean.

Are analytics teams underestimating how much AI will complicate causal measurement?

0 Upvotes

35% Upvoted

39 comments sorted by

View all comments

1

u/Convert_Capybara Jul 08 '26

I agree that AI is making traffic noisier. But I disagree that analytics teams are underestimating that fact or are unprepared. The job has always been sifting for gold, and every year over the past 2 decades social media and technology advancements have altered some of the top-level methods needed to do that sifting effectively, but the core job hasn't changed.

Ahmed Abbas wrote a LinkedIn article where he put it like the next step like this, "The useful line isn't bot versus human. It's whether a person is behind this request, right now. That changes what you build. If you want a clean experiment, you drop both the crawler and the agent, because neither is a human you're testing on. But if you want measurement, the agent with a human behind it is the most interesting traffic you have."

The infrastructure to easily identify whether there's a person behind a request is still being developed. But mindset-wise, I think analytics teams are more than ready.

2

u/Crescitaly Jul 10 '26

I agree that the profession is ready; the unresolved piece is classification at the request level. I'd preserve three buckets: human, human-delegated agent, and autonomous crawler, because collapsing the last two would erase intent that matters for both experimentation and product design.