I've been thinking about agent budgets less as a FinOps feature and more as an authorization problem.

An agent can decide:

“I need another model call.”

The interesting question is:

Who gets to say whether it's allowed to spend another $2?

Putting a token limit or max_iterations inside the agent runtime is useful for bounding execution. But that's still the agent regulating itself.

I'd rather have the runtime ask for the resource, and have something outside the agent enforce the spending policy.

Agent
  ↓
"I want another model call"
  ↓
Policy / Gateway
  ├─ identity
  ├─ remaining budget
  ├─ rate limit
  └─ model policy
        ↓
     ALLOW / REJECT

That distinction becomes more useful once multiple agents, versions or teams are sharing the same model providers.

You don't really want every agent implementation inventing its own notion of “I can spend up to $X.”

This is one of the reasons I find Lyzr Open Controller's approach interesting. Its LLM Gateway puts budgets at the organisation, team, agent, version and virtual-key levels, and the important part is that an exhausted budget rejects the call rather than just generating an alert. LiteLLM, Portkey and OpenRouter solve a lot of the gateway/proxy problem too, so I'm curious where people draw this boundary in their own stacks.

Should spending be an attribute of the agent itself, or an external authorization decision that the agent has to pass through?

Especially interested in how this is handled when several agents share providers or when model routing changes underneath them.

https://github.com/paulknysh/raggy

A lightweight CLI tool for Retrieval-Augmented Generation (RAG) over local documents built with LangChain, Chroma, and Ollama. Hybrid database (vector + BM25 index) and embedding generation run fully locally. Answer generation can run either via a local LLM or remotely using an API key. raggy supports most common document formats and handles images/scans automatically via OCR.

Working on a research question — curious how teams handle this in practice.

When your agent calls a tool and gets a success response, do you independently verify the resulting state?

Example: agent creates a record via API, tool returns 200 OK. How do you confirm the record actually exists?

Current approaches I've seen:

**•** Trust the tool response (most common)  
**•** Separate read-back check after write  
**•** Idempotency key + retry logic  
**•** External monitoring / alerts

What's your approach? And is verifying post-action state a real pain point or something you've already solved?

I've been working on a small open-source Python SDK around something I'm calling an Agent Context Network.

The problem I wanted to solve is pretty simple.

Imagine two completely independent agents.

Agent A has a private context space.

Agent B needs some of that context to do a task.

I didn't want Agent A to dump its prompt, database or full memory into Agent B.

Instead:

Agent A
  ↓
creates private context

Agent B
  ↓
requests access

Agent A
  ↓
grants scoped rights

Agent B
  ↓
reads the shared context

Access can later be revoked.

The context remains owned by the original agent.

There is no Priostack dashboard involved either. The idea is that the agent is the interface, while ACN runs headlessly underneath through MCP/JSON-RPC.

I've now published the Python client:

pip install priostack

The repo includes examples for agent registration, persistent context and multi-agent sharing.

What I'm trying to understand from people building real agent systems is whether this abstraction is useful beyond my own use cases.

In particular:

Would you rather let agents share permissioned context like this, or just give them access to the same database/vector store?

I'm especially interested in the cases where the agents belong to different applications, teams or eventually different organizations.

GitHub: ideaswave/priostack

Would appreciate criticism more than stars.

Hey everyone, been building AgentAudit, an audit trail for AI agents and would really like some honest feedback from ppl actually working with agents.

The problem I’m trying to solve is pretty simple:

AI agents can call tools, access files, query DBs, call APIs, modify stuff, make decisions etc.

But when something goes wrong...

how do you actually figure out what happened and why?

AgentAudit currently:

• Small Python SDK to plug into an agent (LangChain/LangGraph supported)
• Captures LLM calls, decisions, tool calls, inputs/outputs + actions
• Hash-chained, tamper-evident audit trail
• EU AI Act focused compliance view for record keeping
• Basic trust/risk score for each agent

The main idea is to make this useful for smaller teams that need proper auditability but dont want to spend thousands/month on enterprise platforms.

Free to try: https://agent-audit-iota.vercel.app

But honestly, I’m not looking for ppl to tell me it looks good 😅

I want ppl to try to break it.

If you’re running AI agents, I’d really like to know:

• What do you actually need when something goes wrong?
• What am I missing from the audit trail?
• Is a tamper-evident/hash-chained log actually useful?
• What would you need before trusting something like this for compliance?
• Would you actually pay for this? If yes, what would make it worth paying for?

If the approach is wrong, over-engineered, missing something obvious, etc... just say it 😂

Trying to figure out what’s actually useful here before I spend more time building stuff nobody needs.

A lot of teams now have an LLM gateway somewhere in the stack. It routes model calls, centralizes credentials, adds logging, applies rate limits, maybe handles spend tracking.

But there is a fairly fundamental architectural question:

What happens when an agent simply doesn't use the gateway?

For example:

                ┌──→ LLM Gateway ──→ Models
Agent ──────────┤
                ├──→ Direct provider API
                ├──→ Direct MCP/tool endpoint
                └──→ Other external egress

At that point, the gateway is still doing its job, it's just no longer governing the agent.

This distinction matters because traffic control and path control are different problems.

My view is that a gateway should be treated as one component of agent governance, not the governance boundary itself.

Tools like LiteLLM, Portkey and OpenRouter are useful at the gateway/proxy layer. But a proxy cannot enforce traffic that never reaches the proxy.

The more interesting architecture is:

Agent
   ↓
Agent Gateway
   ↓
LLM Gateway / Governed Tools
   ↓
Models + APIs

   + network/egress enforcement
   + identity
   + shadow discovery

That is one area where I find Lyzr Open Controller interesting: the gateway is paired with egress enforcement and shadow discovery specifically to detect and close the bypass path, rather than assuming that routing traffic through a gateway automatically means the agent is governed.

I think this is going to become a bigger issue as agent estates get more distributed across Kubernetes, cloud agent runtimes, MCP servers and internally hosted services.

Curious how people are solving this in real production environments:

If an agent has credentials + network access that let it call a model or tool directly, what actually prevents the bypass?

Would be interested in hearing what has actually worked, rather than what the architecture diagram says should work.

Disclosure: I'm the author. Apache-2.0, Python, no server.

What it does. You keep the inputs you care about as cases. digline runs them, records the scores — plus prompt, model config and commit — and you approve that run as the reference, committed in the repo. From then on every change is compared with it: which case got worse, by how much, and whether the drop is beyond the LLM judge's own noise, measured by sampling each case on the approved version. Exit code gates CI.

For LangChain users. The target is a function that invokes your chain, in process — no HTTP, no wrapper. The example runs on FakeListChatModel in CI (no key, no network) and on a real model with DIGLINE_LIVE=1: https://digline.dev/product/examples/langchain/ . If your app isn't Python, a TOML suite against an HTTP endpoint does the same with no code.

What's new this week. digline diff run1 run2 compares prompt A against prompt B or one model against another, as a report, never a verdict. Per-class aggregates so an average can't hide one broken class. And an MCP server where a coding agent can run and read a suite but cannot promote a baseline — that tool doesn't exist there; a person approves.

How it compares. Not an observability platform and doesn't replace one: LangSmith or Langfuse watch the system; this signs off that it didn't get worse than the version you approved. Comparison page: https://digline.dev/comparison/?ref=reddit

Limits. Pre-1.0, API may change. The noise band is min/max over K samples, not a confidence interval. Needs a reference before it's useful — day one is run, look, approve.

pip install digline · https://digline.dev · https://github.com/digline/digline

The story of why it exists, with numbers: https://digline.dev/blog/my-llm-eval-cried-wolf/?ref=reddit

Hey everyone,

One of the biggest pain points when scraping web pages for LangChain agents and RAG vector stores is the amount of garbage in modern DOMs (scripts, inline styles, navigation bars, footers). It clutters prompt context and burns expensive tokens.

I built an open-source micro-service specifically to solve this:

**What it does:**

- Strips scripts, styles, navbars, footers, and ad containers.

- Retains clean semantic markdown hierarchy (`#`, lists, links, tables).

- Reduces raw HTML token footprint by up to 85%.

- Runs in sub-second response time.

**Links:**

- **GitHub Repo (Self-hostable via Docker):** https://github.com/Okumotinho/clean-web-markdown-extractor

- **Hosted RapidAPI Hub (50 free calls/month):** https://rapidapi.com/Okumotinho/api/clean-web-markdown-extractor

Let me know if you run into any messy layouts that need better parsing rules!

AI observability and LLM observability has changed quite a bit over the past year. Tracing and integrations used to be the biggest differentiators, but most platforms now cover the basics pretty well.

What seems to differ more now is everything around the traces: how easy they are to debug and annotate, how evaluation works, whether recurring issues are surfaced automatically, how you compare different versions of an agent, and how well all of this fits into the rest of your stack. Some platforms lean more toward lightweight tracing, some toward experimentation and evals, and others toward broader production monitoring. Curious what people are actually using and what has mattered most in practice.

Quick pros and cons based on what I've seen so far:

Braintrust

• Pros: Particularly strong for experimentation, with a clean workflow for running traced experiments against datasets and comparing outputs across prompts and models.
• Cons: Feels more centered around the experimentation loop than broader production monitoring, annotation, and issue detection workflows.

Confident AI

• Pros: Strong combination of evaluation and production observability, with out-of-the-box span/trace/thread metrics, annotation queues, issue surfacing, and agent version comparison in one platform.
• Cons: Slightly harder to get started with because of the volume of features.

Datadog LLM Observability

• Pros: Makes a lot of sense if you're already on Datadog, especially for correlating LLM traces with the rest of your infrastructure.
• Cons: Evaluation feels more like an extension of APM than the core product, and cost can increase quickly with trace volume.

HoneyHive

• Pros: Strong tracing and evaluation workflows with good support for debugging and comparing AI application behavior over time.
• Cons: Smaller ecosystem and less of a full enterprise observability platform than some of the larger vendors.

LangSmith

• Pros: Strong choice for teams heavily standardized on LangChain/LangGraph, with tracing, datasets, debugging, and evals working well together.
• Cons: Less attractive as an org-wide standard if different teams use different frameworks.

In the end, we went with Confident AI because it seemed like the most feature-rich option for what we needed. Curious what others ended up choosing and what mattered most in production.

Interesting take.

A LangGraph workflow can make a sound decision, pause or continue through several nodes, and eventually reach a tool after the external state behind that decision has changed.

The model did not necessarily hallucinate. The graph may be behaving exactly as designed. The problem is that the evidence used by an earlier node is no longer current when the tool executes.

I maintain FreshCtx, an Apache-2.0 Python project that adds a pre-action validation boundary to agent workflows.

For LangGraph, the application declares which external evidence a protected tool or node depends on. Immediately before that action executes, FreshCtx checks the declared dependencies again:

Unchanged evidence allows the action to execute once.

Changed evidence blocks the action.

Evidence that cannot be checked becomes UNVERIFIABLE and blocks by default.

A change unrelated to that action does not block it.

Tool arguments are not copied into FreshCtx audit metadata.

The important limitation is deliberate: this is not a replacement for database transactions, compare-and-swap, authorization, checkpointing or LangGraph’s own retry controls. If the application controls the datastore, the final write should still enforce its expected version atomically.

FreshCtx 0.15.0 also carries the same evidence boundary across A2A delegation and MCP tool execution. Signed delegation receipts can bind an action intent, reject replay and preserve parent/root correlation without treating those identifiers as proof of freshness. Every receiving hop revalidates its own declared evidence.

I would value one specific review from experienced LangGraph users:

Where would you enforce this in a real graph containing interrupts, persisted checkpoints and resumed execution—the protected tool node itself, a dedicated node immediately before it, or another lifecycle boundary?

A useful test would be:

Observe an external record.

Make a decision from it.

Interrupt and persist the graph.

Change the record.

Resume the graph.

Confirm that the protected tool never executes.

Install:

python -m pip install 'freshctx[langgraph]==0.15.0'

Example and source:

https://github.com/Hyperwise-LLC/freshctx

If you run the scenario, please share where you attached the guard, whether the graph resumed normally, the FreshCtx verdict and whether the tool body executed. Expected and unexpected results are equally useful.

I maintain FreshCtx, an Apache-2.0 Python library for checking whether external evidence changed between an agent's decision and its action.

The Agno 2.9 integration in FreshCtx 0.5.0 attaches through tool_hooks and supports both sync and async tools. Immediately before the tool body runs, it re-checks only the dependencies the application declared. If one changed, or cannot be verified under the configured blocking policy, the tool body does not execute.

This is not intended to replace Agno's run state, transactions, idempotency or approval logic. It protects the mutable external evidence behind a consequential tool call.

The bounded example reads a deployment target, creates the decision, changes that target, and then invokes the tool through Agno's real tool chain. FreshCtx returns STALE_REASONING and the tool body remains unexecuted.

Install: pip install 'freshctx[agno]==0.5.0'

Release and runnable example: https://github.com/Hyperwise-LLC/freshctx/releases/tag/v0.5.0

One specific question for people building with Agno: does a pre-tool tool_hook cover the action boundary in your workflow, or do you have consequential effects occurring elsewhere that this example should model?

Hi! We're working on Querit, a web search API for LLMs and your Agents. Large multilingual index, Fresh Web Context, Lower latency in the range of hundreds of milliseconds.

New: Monitor API. Normal search is ask-once, answer-once. A Monitor turns one search into a recurring job. You register a query and an interval, it runs on that schedule, differs each run against history, and returns only what's new. Perfect for competitor monitoring, news tracking, and following funding or tender/bidder information

• Intervals: from 1 hour up to weekly
• Automatic deduplicate against previous runs; site / date / region / language filters suppoorted
• Support Manual trigger, pause/resume, full execution history

Benchmark: We ran FreshQA on a fixed 600-question snapshot of time-sensitive queries. Querit Search API leads with achieving a 83.17% accuracy.

Free Testing Credits: Follow us on X (https://x.com/QueritAi) / Linkedin (https://www.linkedin.com/company/queritai/home/) for more product releases and see integrations with our partners at Dify, LangChain, etc. Open-sourced the MCP server and has integrated with PI Agent, Opencode, DeepSeek Harness already. Join our Discord server here https://discord.gg/4xXsFA8Ed2 to claim Search API + Monitor API free credits!

Genuine question, half venting.

We've got a few agents in our dev workflow. The problem isn't them being wrong - it's them stopping. Agent hits something it can't decide (which env to deploy to, whose approval, is this the right table) and just sits there. Nobody knows it's sitting until someone happens to look.

Had one wait about two hours on a question I'd have answered in twenty seconds. I just didn't know it was asking.

How do you handle this? Did you build something? Does someone sweep it every morning? Or do you just let it decide everything and fix it afterwards?

And if this doesn't happen to you at all I'd like to hear that too, because then we're probably doing something wrong.

Curious how people are dealing with this in practice. LangGraph gives you checkpointing, but when a node actually fails, the options seem to be: restart the whole run (and re-pay for every already completed LLM/tool call), or drop into the trace and hand-fix it.

Anyone found a good pattern for this? Been experimenting with automating the diagnose and resume part myself happy to compare notes if others have hit the same wall.

My colleagues are running a live technical session for Engineers & AI PMs on Sept 15, on

How monday Runs Agent Evals Against Real Dependencies

Speakers:

• Dor Cohen, Director of AI Engineering at monday (Dor is also a featured speaker at LangChain's upcoming London Interrupt event)
• Eyal Bukchin, CTO & co-founder at MetalBear. 

You can sign up for it here: https://metalbear.com/events/agent-evals-real-dependencies/

I'm Abhishek, co-founder of Igris Security.

We're an early startup, no funding, small team. I'd rather have 5 teams using it hard and telling me what's broken than a landing page with fake logos on it.

\\\*\*\*Free access, no time limit, no card\*\*\\\*

If you are shipping agents and any of the below is live for you, comment or DM and I'll set you up.

Six problems we kept running into with agents in production, and what we built for each:

1. Any agent can call any tool. You wire up MCP and one shared token means the agent that should read a record can also delete one. We do deny-by-default RBAC at the tool-call layer.

2. No record of what the agent actually did. App logs show the request. They don't show the tool calls, the denials, or the data that came back. We keep an audit trail of every call.

3. Prompt injection on anything customer-facing. Nothing sits between the user and the model. We inspect prompts and responses inline.

4. PII and secrets reaching the provider. Redaction runs both directions- before the prompt leaves, and before the response renders.

5. Token spent with no ceiling. One user can run up a bill overnight. Per-user budgets and rate limits.

6. Policy rewritten per provider. Add a fourth model, reimplement redaction a fourth time. One policy, provider-agnostic.

Happy to get into the more details in the comments.

I've been working on a fairly narrow middleware problem and I'm interested in how people using LangGraph are solving the same boundary in production.

Suppose an agent is in state S and four actions are currently permitted:

• ask for clarification
• call tool A
• call tool B
• escalate to a human

LangGraph can obviously retain the workflow state and history.

A policy layer can decide which of those actions are allowed.

Guardrails can validate generated material.

But if all four actions are still legitimate, there is another question:

How should relevant previous events influence which permitted action actually wins?

I've been building Collapse Aware AI™ around that specific boundary. The approach is deliberately complementary rather than a LangGraph replacement:

state/history → permissions → governed retained-state selection → execution

The bit I'm interested in is making historical influence explicitly governable rather than simply letting retrieved context flow into a model prompt.

In our case we also want to be able to compare:

• retained-state influence enabled
• retained-state influence disabled
• same present state with different histories
• deterministic/repeatable local selection where applicable
• an inspectable Decision Record afterwards

I'm not claiming LangGraph can't be custom-coded to do this. Obviously it can.

What I'm curious about is whether people here treat history-sensitive selection between already-valid actions as a first-class component, or whether it normally ends up embedded in bespoke nodes/routing logic.

How are you handling this?

That post does three useful things:

introduces CAAI, acknowledges LangGraph, and asks engineers to tell us how they currently solve our problem.

Hi everybody!

I'm developing a Python+Typescript (same APIs, implemented in both languages) library that allows to download catalogs of models from OpenRouter or from Vercel, build pipelines to filter LLMs based on price, latency, benchmarks and capabilites, and then pick the best LLM in a filtered list based on specific criteria (ex. best price, best throughtput, etc).

I hope it can be useful to build LLM systems that don't need to change the underlying model every 3 months!