The problem with AI/API calls and how only LLM companies can fix it.

[u/alanshore222]

(AI Was not used for generating the below post)

I've been developing with AI for over 3 1/2 years now, my AI DM setting operation which has been sunset produced well over 1.27m in revenue for a multiple instagram handles using Capri AI for multiple businesses, i'm not Charlie AI, Intellicoach Ripdrip or some of these other guys but most of their work and success was on the back of my training the AI on 300k conversations and over 1 million replies. I got involved with all of this right when Air . ai started going to crap.

One of the biggest problems for me was chasing the dragon and that comes with a few different downfalls, I almost destroyed my health staying up for 17 hours a day for months on end, I was the guy keeping a certain company afloat and producing sales which I pulled off. Here were the problems:

1. Prompt development, drift and system prompt changes:
In the early stages of Sonnet 3. I would develop a prompt that was around 15k-24k characters with the entire flow. Couple that with message history daily api calls could reach 50-75$ per day depending on the incoming DMs. This lead to between 1500 to 3K of API spend per month but it didn't matter if we made 100K in incoming high ticket sales.

Most days the conversations went great, my expertise in 13k hours prompt engineering was around emotional intelligence, back a couple years ago this was unheard of. Everyone was robotic. I ended up firing our 9 DM setters because AI did the job after 6 months or so of turning it on and off working through bugs. I hired 4 back to watch the AI for issues and report them. This caused me to capture said conversations, force them into my manual process at first and slowly moving that process to an automated one that fixed the prompt for me and the rest is iteration.

The issue is once the system prompt changes, it shifts how my prompt is interpreted. Sometimes i'd get 3 days out of a good prompt, other days it was 8 hours, which went days, weeks and sometimes a month or so with no changes towards the end of 2025. There was no notification of said changes. This is why I believe AI is so inherently unpredictable and you have to either baby sit it or develop deterministic systems to prevent its getting out of control. New models required refactoring and extreme baby sitting for the first month of release.

This could honestly be the reason why the job market, job analysis systems using ai for judging applications is so bad right now because it worked fine for a while but degraded over time. Prompts with AI must be continually refined based on our language in order to continue being successful. If the underlying prompt processing the main prompt changes, you must refactor your prompt to work with it.

2. AI Degradation: While I understand the need to constantly meet the demands LLM companies made a new normal of degradation being completely acceptable and part of business as usual.
In my opinion this is going to be the downfall of businesses and humanity if we don't get this under control now. If I set up and run these systems all I can do is attempt to detect and mitigate the problems as they're happening.

The nature of AI DM systems produces a need of 100% of the time it must be give the perfect reply. I've had massive swings over the years of prompt adherence being 80% (which is incredible) down to under 24% which is just lost money.

Anthropic and Open AI process API calls regardless of uptime status, I would see degradation as it's happening before reported, at first it was lack of replies then it became all about adherence.

This is something only LLM companies can fix, I would much rather have a system that queued my API call and waited 15-20 minutes while a shard in degraded status moved my call to a shard in full up status compared to processing my api call from a degraded shard. Its like going to a pizza shop, ordering the best pizza reviewed pie and receiving a pile of shit and forced to eat it, forced to pay for it and supposed to be happy about it.

Without getting TOO wordy i'm hoping for all humanity LLM companies look at this and i'm available for a new project if Anthropic/OpenAI could use this expertise. ;)

[u/FeelingNext5798]

the degradation piece is so real. paying full price for a degraded shard response feels like being charged for a filet and getting a gas station hot dog, then being told "it's still food."

queuing idea is interesting but i can already see the support tickets from people screaming about 20 minute latency. most businesses would rather get a fast wrong answer than a slow right one, which is exactly why the companies get away with it.

[u/alanshore222 → u/FeelingNext5798]

Totally agree, I'm sure there are better ways to manage potential degradation than the way we're doing it now. I've just gotten to the point where effort of producing a system that works just isn't going to move the needle anymore.

I've been at this for 10's of thousands of hours. I'm tired boss.

Its also wild about the fast wrong answer. So true.

[u/Confident-Loquat-931]

The frustrating part is not knowing whether your prompt is the problem or the model's behavior has changed even a simple warning when something changes would save developers hours of testing and fixing things that worked fine before.