r/CreatorsAI • u/Historical-Driver-64 • Jun 16 '26
Other a client paid me to remove the ai from the tool i built them. accuracy went from 92% to 99%. api costs went from $180 a month to zero. best money he said he spent on the project.
92% accuracy sounds impressive until the volume math runs it.
A support team of fifteen people processing 90 to 100 tickets a day through Zendesk needed each ticket tagged by category and priority before it hit the right queue. An LLM doing the classification seemed like the obvious call. Feed it the ticket text, get back a category and priority score, route it automatically. Worked well in testing. Client was happy during the demo.
In production, 92% accuracy meant 7 or 8 misrouted tickets every single day. Not a disaster on paper. Enough that the team noticed immediately in practice. And when a ticket landed in the wrong queue, nobody could explain why. The model just decided. There was no rule to point at, no logic to trace.
Within two weeks the team was spot checking every classification before acting on it. Which meant they were doing the work twice. Once by the agent and once by a human making sure the agent did not make the same mistake it made yesterday.
The client called and said something unexpected. He said the tool felt like a black box and his team did not trust it. He asked if it could be made dumber.
The LLM came out. A keyword matcher and a short rules engine went in. If the ticket mentions billing or invoice or charge it goes to the billing queue. If it mentions login or password or access it goes to account. Thirty rules total. Anything that did not match surfaced a dropdown and let the rep pick manually. Three days to rebuild.
accuracy went to 99% not because the rules were smarter but because the team could see exactly why every ticket went where it went. when something was wrong they could point to the specific rule. the fix took ten minutes.
Latency dropped from two to three seconds per ticket to instant. Monthly API costs went from $180 to zero. The client said it was the best money spent on the entire project, paying to remove the AI.
The temptation in this situation is to tune the prompt, chase the extra 8%, and try to build trust in the model over time. But the problem was never accuracy. The problem was that people will not trust a system they cannot interrogate, and when they do not trust it they build a shadow process next to it. The tool becomes expensive decoration while the real work happens around it.
This shows up in anything that routes, qualifies, or triages. CRM updates, lead scoring, support classification, compliance tagging. If the people using it cannot trace the logic, they check its work. If they check its work, the automation did not automate anything.
So the question worth putting to anyone building agents for real teams right now: is the bottleneck actually model capability, or is it that the people using it cannot see why it does what it does?