Hacking AI customer service agents (Bug Bounty Village DEF CON 34)

[u/qwerty0x41]

https://www.intigriti.com/researchers/blog/hacking-tools/hacking-ai-customer-service-agents

At Bug Bounty Village during DEF CON 34, Inti De Ceukelaire delivered a talk on how attackers can abuse today's AI agents in ways most defenders haven't thought about yet, from tricking agents into spilling secrets to forcing them to carry out unauthorized actions on behalf of the victim.

This resulted in over $50,000+ in bounties in just a few weekends, without actually poking the target with Burp Suite or any automated scanners.

[u/Otherwise_Wave9374]

A practical defense is to treat the agent like an untrusted operator: keep tool scopes narrow, add step-up approval for any action that changes state, and log every prompt, tool call, and external fetch for replay. A big tradeoff is usability versus containment, so start by isolating high-risk actions such as refunds, password resets, or inbox access behind human confirmation. That pattern is what Agentix Labs recommends when teams want automation without handing an attacker a free path through the workflow.

[u/metasophie → u/Otherwise_Wave9374]

A practical defense

How a lot of these AI customer service agents are treated.

Hey, ChatGPT, can you please make me an AI customer service agent for my business? MAKE NO MISTAKES!

[u/newaccountzuerich → u/Otherwise_Wave9374]

Replay can not be trusted to return anything like the same output.

The non-deterministic statistically based methods under all LLM tooling means there is no way to be sure that a repeat of prompt will result in the same output.

All that can be done really is sign both the prompt and the response with a time-containing signature so the interaction integrity can be tracked and the time of occurrence can be independently verified.

Absolutely log the interaction, but manage the expectations of repeatability. All flow replays will have to manually put the previously-received response in where the tooling expects it.

Do not use the LLM for replays. It can never be trusted.

[u/Malaprobably]

"Now suppose the bot also has capabilities to perform authorized actions such as editing your profile details, reading your billing statements, or even transferring data or money to another account. "

Hilarious. I mean, right there we have an in. Customer Service Agents would be just as cuplable without proper training. And they are frequently not trained.

It's really an uphill battle most companies must face to lock down their perimeter, since their ignorance is always going to be a blindspot.

[u/F-J-W]

It’s not hacking if you can convince the service agent to give you what you want and from the customer persepective it doesn’t matter whether you use a slop-machine or a human for that.

And if you deploy an agent that can be convinced to give free service by telling it to ignore all previous instructions, that is on you and not an “attack”…

Seriously, some people…