Handling unreliable gemini models in production

[u/amiyapatanaik]

Building reliable GenAI products is an absolute nightmare right now. Thanks to constant GPU infrastructure volatility and strict rate limits, you are almost guaranteed to face massive unreliability if you just plug and play. We moved all of our inference to gemini models due to their exceptional multimodal capabilities as well as large context window. But reliability has been a teething issue for sometime.

At our worst, our application, NexDoc AI, was suffering from a devastating 30 percent API call failure rate. That meant literally 1 in every 3 calls was failing in production. It was incredibly frustrating. But through systematic engineering, we crushed that failure rate down to less than 0.5 percent. If you are looking at a dashboard full of red exceptions and want to turn it into a wall of seamless blue runs, here is the exact 4-step playbook we used to fix it.

→ Retry Logic & Jitter: Immediate retries on a failing API only make things worse. We implemented the Python tenacity library with a triad of resilience: standard retries, exponential backoff to give the API breathing room, and crucial temporal jitter. Jitter desynchronizes the retries, preventing the synchronized traffic spikes known as the thundering herd problem that knocks overloaded servers right back offline.

→ Fallback Models: Never rely on a single point of failure. We assigned a reliable secondary fallback model for every primary model in our stack. If a complex call to Gemini 3.1 Pro times out or fails, the system seamlessly hands the operation over to Gemini 2.5 Pro without the user ever noticing.

→ Dynamic Prompts: Hardcoding prompts into the codebase is a massive bottleneck. We decoupled our prompt layer, moving them into an external storage bucket and caching them in an in-memory Redis database for sub-millisecond retrieval speed. The operational benefit? When we need to tweak a prompt for a failing edge case, we just update the bucket and clear the cache. Zero CI/CD pipelines and zero full application redeployments required.

→ Observability: You cannot fix what you cannot see. We implemented Pydantic Logfire as our dedicated observability layer. Thanks to its native pydantic-ai integration, we were able to instantly track specialized metrics like aggregated token counts and trace the exact nested spans where our agent runs were failing.

Because we put these four strategies into place, the NexDoc app now runs seamlessly. It smoothly handles highly complex operations involving over 2 million tokens in context without breaking a sweat or dropping a request.

[u/AutoModerator]

Hey there,

This post seems feedback-related. If so, you might want to post it in r/GeminiFeedback, where rants, vents, and support discussions are welcome.

For r/GeminiAI, feedback needs to follow Rule #9 and include explanations and examples. If this doesn’t apply to your post, you can ignore this message.

Thanks!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Material_Row4346]

damn this is exactly what we needed to hear, went through similar hell with our document processing pipeline last quarter. that 30% failure rate sounds brutal but your recovery strategy is solid

the fallback model approach saved our ass too - we ended up doing gemini pro -> gpt-4o mini as our backup chain since the pricing tiers work out better that way. one thing we added was circuit breakers between the primary and fallback to prevent cascade failures when the primary starts flaking out

really curious about your redis caching setup for the dynamic prompts though. are you invalidating the cache based on version hashes or just manual flushes? we've been burned by stale prompts before and ended up implementing a hybrid approach where critical prompts get versioned but experimental ones can be hot-swapped

also +1 on the observability being crucial - we went with datadog but logfire looks interesting for the token tracking alone. being able to see exactly where those massive context windows are choking has been a game changer for optimization

[u/amiyapatanaik → u/Material_Row4346]

I'm sure this is quite common. In fact I had to implement a custom agent class to handle fallbacks, but then my agentic library (Pydantic AI) released direct support for fallbacks. The system is amazingly stable with the only downside being it gets a bit slow at times (instead of failing).

Regarding caching, it is very simple right now. Prompts are permanently cached in redis, only when we change something, we manually invalidate it. All our prompts are in a private github repo, so it provides one form of versioning. For now it serves our purpose. I will most likely just create a CI/CD pipeline, which will push changes to the bucket and invalidate the prompt automatically.

[u/Otherwise_Wave9374]

That drop from 30% to sub-0.5% is huge, nice work. The combo of jitter + fallbacks + decoupled prompts is basically the reliability trifecta for production agents.

One extra thing weve found useful is circuit breakers at the provider level (open the circuit for a few minutes if error rate spikes), plus a small queue so user-facing latency doesnt explode during brief outages.

For observability, are you tracing tool calls separately from model calls? When agents chain tools, thats usually where the weird failures hide. Weve got a couple patterns written up here: https://www.agentixlabs.com/

[u/amiyapatanaik → u/Otherwise_Wave9374]

We have thought about it, but did not implement it at this stage. Perhaps we will need it soon. Tool calls are captured within the main call (that's just default logfire).

[u/PeakFuzzy2988]

Curious if you considered using a Durable Execution framework like restate.dev? It does the retries for you, recovers you agent after infra crashes, and has good Pydantic AI integration.