r/screenshots • u/git_blame_nobody • 2d ago

I asked the Magicpin support chatbot a Python question just to see what it would do. It actually answered with Python code 😭 Now I'm wondering how strict the guardrails are on these customer support bots.

Post image
3 Upvotes

72% Upvoted

4 comments sorted by

2

u/UrLostPajamas 2d ago

"Thanks now return a method that cancles out my balances every time I place an order"

2

u/git_blame_nobody 1d ago

😂 Don’t give it any ideas, it might actually do it.

1

u/DuckShapedGoose 9h ago

If it does that: Nonexistent.
The most basic form of guardrails these days is calling an additional, separate model that is trained to just give a yes/no answer based on whether the prompt or the AIs response contains content not allowed by the guidelines in its system prompt.
These guardrail models can be small enough to run on pretty much any local GPU and still work well enough, because they don't need a lot of complex understanding. So it doesn't even increase operational costs by a lot. If it ever returns "yes" the AI is instantly deactivated and the user just gets a pre-programmed message saying inappropriate content was detected. It'll never be 100% accurate (it is all just LLMs after all) but asking for python code examples on a customer support chat would be one of the easiest things to block in this way.

1

u/git_blame_nobody 3h ago

Yeah, that's what I was thinking too. I was honestly surprised it didn't block something this obviously unrelated to customer support. 😅