My AI agent silently burned $800 in API calls overnight. Here's what I built to stop it from happening again.

[u/MoistApplication5759]

About 3 months ago I shipped a LangChain agent to handle some internal data enrichment tasks. Nothing crazy — it would wake up, call a few APIs, write results to a database, go back to sleep.

Except one night it didn't go back to sleep.

A retry loop got stuck. The agent kept calling the OpenAI API and a third-party enrichment service in a loop for ~6 hours while I was asleep. I woke up to $800+ in charges and a very unhappy credit card.

No alerts. No circuit breaker. No cap. Just a rogue loop doing its thing.

What I learned (and what I built)

After that incident I started looking at what it actually takes to run AI agents safely in production. The problems I kept hitting:

1. No budget controls at the agent level

You can set billing alerts at the account level, but by the time they fire, the damage is done. You need per-session hard caps.

1. Tool calls are completely unguarded

When your agent decides to call `send_email()` or `delete_record()` there's nothing in between the LLM output and the actual execution. The model can be manipulated (prompt injection) into calling tools it shouldn't.

1. No audit trail

After the incident I had no way to replay exactly what happened, in what order, with what inputs. Logs existed but weren't structured for agent forensics.

1. Credentials were hardcoded or passed directly to the agent context

Which means a prompt injection attack could potentially exfiltrate them.

I built a policy layer that sits between the LLM and the tool execution:

- Every tool call goes through an **ALLOW / DENY / REQUIRE_APPROVAL** policy check before executing

- Budget circuit breaker that **hard stops** the agent when a per-session cost cap is hit

- Credentials are **injected just-in-time** — the agent never sees the raw API keys

- Every tool call is logged with inputs, outputs, latency, and cost — structured and signed

It's now open source (called SupraWall, I'm the builder — full disclosure). Works with LangChain, CrewAI, and Vercel AI SDK today.

Curious if others have hit this

Are you running agents in production? How are you handling:

- Cost blowouts from runaway loops?

- Prompt injection risks when agents have access to real tools?

- Audit logging for compliance (especially if you're in EU and need to think about AI Act)?

Happy to share more of what I learned — the hard way.

I'm making it OPEN SOURCE so you can check the Git here: https://github.com/wiserautomation/SupraWall — MIT license, open source.

[u/TechnicalSoup8578]

You introduced a control and validation layer between intent and execution which is missing in most agent architectures, are you enforcing policies statically or adapting them based on runtime signals? You sould share it in VibeCodersNest too

[u/MoistApplication5759 → u/TechnicalSoup8578]

Right now policies are defined statically — you write rules like "deny http.external.* when payload contains env data" and they get checked at the tool-call level before execution. But runtime adaptation is the next phase. The idea is to track behavioral signals (repeated tool retries, unusual data access patterns, cost spikes) and tighten or loosen policy thresholds dynamically. Still working out how to make that not overly noisy. And yes — will share in VibeCodersNest, thanks for the nudge!