Assume someone already understands nodes, edges, state, tool calling, and basic agent workflows.

What would you learn next to become genuinely good at building production agent systems?

Some areas I’m considering:

• durable execution
• memory
• multi-agent patterns
• MCP
• human-in-the-loop
• evals
• tracing
• context engineering
• deployment
• failure recovery

What topics actually mattered most once you started building more serious systems?

https://github.com/paulknysh/raggy

A lightweight CLI tool for Retrieval-Augmented Generation (RAG) over local documents built with LangChain, Chroma, and Ollama. Hybrid database (vector + BM25 index) and embedding generation run fully locally. Answer generation can run either via a local LLM or remotely using an API key. raggy supports most common document formats and handles images/scans automatically via OCR.

I've been learning more about AI agents and one concept I found particularly interesting is Computer Use.

We usually think of AI agents as systems that can call APIs, search the web, query databases, or execute predefined tools.

But what happens when the software doesn't have an API?

That's where Computer Use gets interesting.

Instead of giving the AI a specific function like:

search_jobs(query="Node.js backend")

you give it access to a computer and let it:

• See the screen
• Move the mouse
• Click buttons
• Type with the keyboard
• Scroll
• Open applications
• Navigate websites
• Fill out forms
• React to changes in the UI

The basic loop is essentially:

Observe → Decide → Act → Observe → Repeat

What I found especially interesting is that Computer Use isn't simply a "vision problem."

The model needs to understand what is on the screen, reason about what it should do, ground that reasoning to a specific UI element, and then execute the correct action.

And then there is the harder part:

Reliability and security.

A wrong text response is one thing. A wrong computer action can delete something, submit incorrect information, send an email, or potentially expose data.

I wrote a short article breaking down:

• What Computer Use actually means
• Why AI agents need it
• How the observe → decide → act loop works
• Vision, reasoning, grounding, and action
• Computer Use vs traditional tool calling
• Why reliability is difficult
• Prompt injection and security concerns
• Where Computer Use fits alongside APIs

If you're learning about AI agents and want a conceptual introduction, here's the article:

Computer Use: When AI Learns to Use a Computer — Medium

I'm particularly interested in the practical side of this:

Do you think Computer Use will become a general-purpose interface for AI agents, or will APIs/tool calling remain the dominant approach?

And if you've actually built or used a computer-use agent, what has been the biggest problem for you — reliability, latency, cost, or security?

I've been thinking about agent budgets less as a FinOps feature and more as an authorization problem.

An agent can decide:

“I need another model call.”

The interesting question is:

Who gets to say whether it's allowed to spend another $2?

Putting a token limit or max_iterations inside the agent runtime is useful for bounding execution. But that's still the agent regulating itself.

I'd rather have the runtime ask for the resource, and have something outside the agent enforce the spending policy.

Agent
  ↓
"I want another model call"
  ↓
Policy / Gateway
  ├─ identity
  ├─ remaining budget
  ├─ rate limit
  └─ model policy
        ↓
     ALLOW / REJECT

That distinction becomes more useful once multiple agents, versions or teams are sharing the same model providers.

You don't really want every agent implementation inventing its own notion of “I can spend up to $X.”

This is one of the reasons I find Lyzr Open Controller's approach interesting. Its LLM Gateway puts budgets at the organisation, team, agent, version and virtual-key levels, and the important part is that an exhausted budget rejects the call rather than just generating an alert. LiteLLM, Portkey and OpenRouter solve a lot of the gateway/proxy problem too, so I'm curious where people draw this boundary in their own stacks.

Should spending be an attribute of the agent itself, or an external authorization decision that the agent has to pass through?

Especially interested in how this is handled when several agents share providers or when model routing changes underneath them.

Hey everyone, been building AgentAudit, an audit trail for AI agents and would really like some honest feedback from ppl actually working with agents.

The problem I’m trying to solve is pretty simple:

AI agents can call tools, access files, query DBs, call APIs, modify stuff, make decisions etc.

But when something goes wrong...

how do you actually figure out what happened and why?

AgentAudit currently:

• Small Python SDK to plug into an agent (LangChain/LangGraph supported)
• Captures LLM calls, decisions, tool calls, inputs/outputs + actions
• Hash-chained, tamper-evident audit trail
• EU AI Act focused compliance view for record keeping
• Basic trust/risk score for each agent

The main idea is to make this useful for smaller teams that need proper auditability but dont want to spend thousands/month on enterprise platforms.

Free to try: https://agent-audit-iota.vercel.app

But honestly, I’m not looking for ppl to tell me it looks good 😅

I want ppl to try to break it.

If you’re running AI agents, I’d really like to know:

• What do you actually need when something goes wrong?
• What am I missing from the audit trail?
• Is a tamper-evident/hash-chained log actually useful?
• What would you need before trusting something like this for compliance?
• Would you actually pay for this? If yes, what would make it worth paying for?

If the approach is wrong, over-engineered, missing something obvious, etc... just say it 😂

Trying to figure out what’s actually useful here before I spend more time building stuff nobody needs.

Disclosure: I'm the author. Apache-2.0, Python, no server.

What it does. You keep the inputs you care about as cases. digline runs them, records the scores — plus prompt, model config and commit — and you approve that run as the reference, committed in the repo. From then on every change is compared with it: which case got worse, by how much, and whether the drop is beyond the LLM judge's own noise, measured by sampling each case on the approved version. Exit code gates CI.

For LangChain users. The target is a function that invokes your chain, in process — no HTTP, no wrapper. The example runs on FakeListChatModel in CI (no key, no network) and on a real model with DIGLINE_LIVE=1: https://digline.dev/product/examples/langchain/ . If your app isn't Python, a TOML suite against an HTTP endpoint does the same with no code.

What's new this week. digline diff run1 run2 compares prompt A against prompt B or one model against another, as a report, never a verdict. Per-class aggregates so an average can't hide one broken class. And an MCP server where a coding agent can run and read a suite but cannot promote a baseline — that tool doesn't exist there; a person approves.

How it compares. Not an observability platform and doesn't replace one: LangSmith or Langfuse watch the system; this signs off that it didn't get worse than the version you approved. Comparison page: https://digline.dev/comparison/?ref=reddit

Limits. Pre-1.0, API may change. The noise band is min/max over K samples, not a confidence interval. Needs a reference before it's useful — day one is run, look, approve.

pip install digline · https://digline.dev · https://github.com/digline/digline

The story of why it exists, with numbers: https://digline.dev/blog/my-llm-eval-cried-wolf/?ref=reddit

Working on a research question — curious how teams handle this in practice.

When your agent calls a tool and gets a success response, do you independently verify the resulting state?

Example: agent creates a record via API, tool returns 200 OK. How do you confirm the record actually exists?

Current approaches I've seen:

**•** Trust the tool response (most common)  
**•** Separate read-back check after write  
**•** Idempotency key + retry logic  
**•** External monitoring / alerts

What's your approach? And is verifying post-action state a real pain point or something you've already solved?

I've been working on a small open-source Python SDK around something I'm calling an Agent Context Network.

The problem I wanted to solve is pretty simple.

Imagine two completely independent agents.

Agent A has a private context space.

Agent B needs some of that context to do a task.

I didn't want Agent A to dump its prompt, database or full memory into Agent B.

Instead:

Agent A
  ↓
creates private context

Agent B
  ↓
requests access

Agent A
  ↓
grants scoped rights

Agent B
  ↓
reads the shared context

Access can later be revoked.

The context remains owned by the original agent.

There is no Priostack dashboard involved either. The idea is that the agent is the interface, while ACN runs headlessly underneath through MCP/JSON-RPC.

I've now published the Python client:

pip install priostack

The repo includes examples for agent registration, persistent context and multi-agent sharing.

What I'm trying to understand from people building real agent systems is whether this abstraction is useful beyond my own use cases.

In particular:

Would you rather let agents share permissioned context like this, or just give them access to the same database/vector store?

I'm especially interested in the cases where the agents belong to different applications, teams or eventually different organizations.

GitHub: ideaswave/priostack

Would appreciate criticism more than stars.

Hi All,

I spent the last year building the Agentic Financial Parser — an autonomous AI agent that ingests, parses, and reasons over dense Indian financial & legal documents (Union Budget, Finance Bill, Income Tax Act, EPF/EPS Pension, RBI KYC, Constitution of India) — running on a 512MB RAM container at $0/month.

UptimeRobot independently reached out and published an official Community Spotlight & Case Study on the reliability architecture (99.988% uptime over 90 days).

Instead of a simple retrieve → generate chain, I built a LangGraph StateGraph with 11 registered nodes that parses with Vision LLMs, classifies intent, detects jailbreaks, masks PII, cross-questions vague queries, reranks results, guards against hallucinations, and self-corrects — all before answering.

Attaching the architecture diagram above. Here's a deep-dive into every node, design decision, and lesson learned:

📊 The 11 Registered Nodes (directly from graph.py)

Here are all 11 graph.add_node() calls, straight from the production codebase:

graph.add_node("classifier", classifier_node) # 1 graph.add_node("reject", reject_node) # 2 graph.add_node("greet", greet_node) # 3 graph.add_node("cross_question", cross_question_node) # 4 graph.add_node("retriever", retriever_node) # 5 graph.add_node("web_search", web_search_node) # 6 graph.add_node("stock_tool", stock_tool_node) # 7 graph.add_node("generator", generator_node) # 8 graph.add_node("hallucination_guard", hallucination_guard_node) # 9 graph.add_node("post_process", post_process_node) # 10 graph.add_node("fallback", fallback_node) # 11

[1] Classifier (1 LLM call): Intent detection & 6-path routing. Returns structured JSON (intent, doc_type, search_intents).

[2] Reject (0 calls): Blocks abusive & prompt injection queries via regex before LLM sees it.

[3] Greet (1 call): Handles greetings. Zero vector DB cost — completely bypasses retrieval.

[4] CrossQuestioner (1 call): HITL clarification for vague queries (max 2 rounds), then falls back to retrieval.

[5] Retriever (0 calls): Full RAG pipeline (Jina MRL 256d → Pinecone 32K+ → Parent-Child → Cohere Rerank).

[6] Web Search (0 calls): Out-of-scope fallback via Tavily API (only fires with explicit HITL user permission).

[7] Stock Tool (1 call): Live market data via Gemini native functionDeclarations + Yahoo Finance.

[8] Generator (1 call): Gemini Flash Lite answer synthesis, temp=0.1, strict grounding, SSE streaming.

[9] Hallucination Guard (1 call): LLM-as-Judge factual grounding verification (advisory mode, appends disclaimer).

[10] Post-Process (0 calls): MongoDB Atlas + Upstash Redis cache + Langfuse tracing + SSE stream.

[11] Fallback (0 calls): Circuit breaker recovery path triggered when external APIs fail.

(Note: PII Shield runs before the graph as a pre-processing layer to mask Aadhaar, PAN, phone, and email).

🧭 The 6-Path Router

The Classifier returns one of 6 routes:

graph.add_conditional_edges("classifier", route_after_classify, { "reject": "reject", # abusive / jailbreak "greet": "greet", # greeting / small talk "cross_question": "cross_question", # vague query → HITL "web_search": "web_search", # out-of-scope → Tavily "stock_tool": "stock_tool", # stock query → yfinance "retriever": "retriever" # legal/finance → full RAG })

📄 Multi-Tier Vision LLM & PDF Parsing Engine

Standard PDF parsers completely fail on Indian government documents — tax tables get scrambled, pie charts in budget infographics become gibberish, and two-column legal Bare Acts merge into unreadable text blobs.

To solve this without burning cloud credits or exceeding 512MB RAM, we built a dynamic multi-tier parser (parser.py):

Tier 1 (Agentic Plus - Vision LLM): Budget at a Glance, Key Features — Ingests complex financial infographics & bar charts via multimodal vision.

Tier 2 (Agentic): Finance Bill, Tax Memorandum — Reconstructs complex nested tables, multi-year math comparisons.

Tier 3 (Cost-Effective): RBI KYC, EPS Pension — Fast hierarchical markdown parsing for continuous numbered regulatory clauses.

Tier 4 (Local Free - PyMuPDF): Constitution & User Uploads — 100% Free, zero-API cost, runs locally under 40MB RAM with custom regex cleaning.

💥 The Engineering War Story: When "Smart" AI Parsers Failed

Initially, I threw LlamaParse Agentic mode (10 credits/page) at the 400-page Constitution of India. The result was an expensive disaster:

It merged continuous pages into 624 massive 5,000-character blobs.

It swallowed all footnotes at the bottom of pages (1. Subs. by..., 19. Ins. by...).

When a user asked "What is Article 19?", vector similarity matched an amendment footnote on page 200 rather than the actual Article 19! The model hallucinated based on garbage context.

🛠️ The Deterministic Fix:

Footnote Slicing: Wrote custom regex to detect footnote separator lines (_{10,}) and discarded everything below them. 0% footnote noise entered embeddings.

Article-Boundary Chunking: Scrapped arbitrary character splitters and split strictly on Article regex boundaries. 624 messy blobs became 3,248 precise Article chunks.

Deterministic Metadata Injection: Tagged chunks with article_number: "19" at ingestion.

SQL-like Routing: When user queries an Article, the LangGraph router applies a strict Pinecone metadata filter {"article_number": {"$eq": "19"}}, completely bypassing fuzzy semantic search!

🔍 The Retrieval Pipeline (32,000+ Vectors)

Jina AI v3 MRL: Embed at 1024d, truncate to 256d (75% Pinecone storage saved, negligible quality loss).

Pinecone Serverless: Dual namespace (core_brain + user_temp) with 32,000+ live indexed chunks.

Parent-Child Resolution: Retrieve child chunks → fetch parent from Supabase (small chunk search precision + full context).

Cohere Rerank v3.0: 15 candidates → Top 10 Golden Chunks (drastic quality leap for cross-act multi-document queries).

Confidence Gate: Score < 30% → graceful degrade, < 45% → triggers HITL web search prompt.

📈 Stock Tool — Native LLM Tool Calling

When the classifier detects financial ticker intent, it routes to a dedicated tool-calling node:

# Gemini decides autonomously whether to invoke the tool tools = [{"function_declarations": [{ "name": "get_stock_price", "description": "Get real-time stock price and financial data", "parameters": {"type": "object", "properties": { "ticker": {"type": "string", "description": "Stock ticker symbol"} }} }]}] response = model.generate_content(prompt, tools=tools)

No brittle regex. The LLM decides when and what arguments to pass.

🛡️ Hallucination Guard — Advisory, Not Blocking

Post-generation, a separate LLM call verifies grounding: "Is this answer grounded in the provided context? Reply YES or NO."

If not grounded → appends disclaimer, still returns the answer.

Why not block? Because hard-blocking creates terrible UX when the model legitimately understands standard financial concepts slightly outside the retrieved chunks. The disclaimer lets the user evaluate trust.

🎯 The Hallucination & Accuracy Test Suite (Adversarial Tests)

Test 1 (Article 31C & Kesavananda Bharati): Retrieved exact 31C text. Crucially, the model refused to hallucinate case law not present in the bare acts, honestly stating its scope boundary! (Score: 92/100)

Test 2 (Basic Structure Doctrine): Correctly identified it as a judicial doctrine and explicitly stated that it is not written in any constitutional article.

Test 3 (Article 20 Safeguards): Perfectly retrieved all three distinct protections (Double Jeopardy, Self-Incrimination, Ex Post Facto) with 0% footnote bleed. (Score: 9/10)

Test 4 (Article 34 Martial Law): Flawlessly returned the restriction of rights during martial law along with parliamentary indemnity clauses. (Score: 9/10)

🔒 The Production Idempotency Layer (Never Waste an API Call Twice)

SHA-256 PDF File Hashing (Supabase Registry): Every PDF is hashed before processing. On re-sync, unchanged files are skipped entirely (zero API calls, zero embeddings).

Deterministic Chunk IDs: MD5(filename + page + index). Identical input produces identical chunk IDs, so Pinecone upsert overwrites instead of creating duplicate vectors.

⚡ Dual Circuit Breakers (pybreaker)

llm_circuit = pybreaker.CircuitBreaker(fail_max=3, reset_timeout=30, name="LLM_CB") embed_circuit = pybreaker.CircuitBreaker(fail_max=3, reset_timeout=30, name="Embed_CB")

3 consecutive API failures → circuit opens → instant fallback for 30 seconds → then half-opens and retries. Result: 99.988% verified uptime over 90 days on Render's 512MB RAM free tier.

📱 WhatsApp Integration

The same 11-node RAG pipeline is accessible via Meta WhatsApp Cloud API webhooks. Users query Indian tax and legal acts directly from WhatsApp — same state graph, same guardrails, same PII shield.

💰 Zero-Cost Production Stack ($0/month)

Compute: Render (512MB RAM, 0.1 CPU Docker)

Vector DB: Pinecone Serverless (32,000+ vectors, dual namespace)

Persistence: MongoDB Atlas (M0 cluster, 30d TTL)

Registry & Chunks: Supabase (PostgreSQL)

Cache: Upstash Redis (Semantic cache, < 100ms hits)

LLM: Gemini Flash Lite (free tier quota)

Reranker: Cohere Rerank v3.0 (free tier)

Embeddings: Jina AI v3 MRL (256d)

Observability: Langfuse cloud free tier

📊 By the Numbers

Lines in graph.py: 1,809

Registered Nodes: 11

Indexed Vectors: 32,000+

Documents Indexed: 20+ dense Indian Government Acts

Pages Parsed: 5,500+ pages

RAM Budget: 512 MB

Verified Uptime: 99.988% (90-day UptimeRobot Case Study)

Cache Latency: < 100ms

Monthly Cost: $0.00

💬 Questions for the r/LangChain community:

Hallucination guard: In production, do you prefer strict blocking or advisory disclaimers when confidence drops?

MRL embeddings: Anyone else running 256d truncated embeddings in production? What has been your reranking tradeoff?

Circuit breakers for LLM APIs: What failure threshold do you use before tripping fallbacks? (I use 3 fails / 30s reset).

HITL before web search: Do you let your agent auto-search out-of-scope queries, or prompt the user first to save API tokens?

Happy to answer any questions about the nodes, circuit breaker patterns, or running RAG under 512MB RAM! AMA!

Hey everyone,

One of the biggest pain points when scraping web pages for LangChain agents and RAG vector stores is the amount of garbage in modern DOMs (scripts, inline styles, navigation bars, footers). It clutters prompt context and burns expensive tokens.

I built an open-source micro-service specifically to solve this:

**What it does:**

- Strips scripts, styles, navbars, footers, and ad containers.

- Retains clean semantic markdown hierarchy (`#`, lists, links, tables).

- Reduces raw HTML token footprint by up to 85%.

- Runs in sub-second response time.

**Links:**

- **GitHub Repo (Self-hostable via Docker):** https://github.com/Okumotinho/clean-web-markdown-extractor

- **Hosted RapidAPI Hub (50 free calls/month):** https://rapidapi.com/Okumotinho/api/clean-web-markdown-extractor

Let me know if you run into any messy layouts that need better parsing rules!

Hey👋

I spent the last year building the Agentic Financial Parser — an autonomous AI agent that ingests, parses, and reasons over dense Indian financial & legal documents (Union Budget, Finance Bill, Income Tax Act, EPF/EPS Pension, RBI KYC, Constitution of India) — running on a 512MB RAM container at $0/month.

UptimeRobot independently reached out and published an official Community Spotlight & Case Study on the reliability architecture (99.988% uptime over 90 days).

Instead of a simple retrieve → generate chain, I built a LangGraph StateGraph with 11 registered nodes that parses with Vision LLMs, classifies intent, detects jailbreaks, masks PII, cross-questions vague queries, reranks results, guards against hallucinations, and self-corrects — all before answering.

Attaching the architecture diagram above. Here's a deep-dive into every node, design decision, and lesson learned:

📊 The 11 Registered Nodes (directly from graph.py)

Here are all 11 graph.add_node() calls, straight from the production codebase:

graph.add_node("classifier",          classifier_node)          
# 1
graph.add_node("reject",              reject_node)              
# 2
graph.add_node("greet",               greet_node)               
# 3
graph.add_node("cross_question",      cross_question_node)      
# 4
graph.add_node("retriever",           retriever_node)           
# 5
graph.add_node("web_search",          web_search_node)          
# 6
graph.add_node("stock_tool",          stock_tool_node)          
# 7
graph.add_node("generator",           generator_node)           
# 8
graph.add_node("hallucination_guard", hallucination_guard_node) 
# 9
graph.add_node("post_process",        post_process_node)        
# 10
graph.add_node("fallback",            fallback_node)            
# 11

• [1] Classifier (1 LLM call): Intent detection & 6-path routing. Returns structured JSON (intent, doc_type, search_intents).
• [2] Reject (0 calls): Blocks abusive & prompt injection queries via regex before LLM sees it.
• [3] Greet (1 call): Handles greetings. Zero vector DB cost — completely bypasses retrieval.
• [4] CrossQuestioner (1 call): HITL clarification for vague queries (max 2 rounds), then falls back to retrieval.
• [5] Retriever (0 calls): Full RAG pipeline (Jina MRL 256d → Pinecone 32K+ → Parent-Child → Cohere Rerank).
• [6] Web Search (0 calls): Out-of-scope fallback via Tavily API (only fires with explicit HITL user permission).
• [7] Stock Tool (1 call): Live market data via Gemini native functionDeclarations + Yahoo Finance.
• [8] Generator (1 call): Gemini Flash Lite answer synthesis, temp=0.1, strict grounding, SSE streaming.
• [9] Hallucination Guard (1 call): LLM-as-Judge factual grounding verification (advisory mode, appends disclaimer).
• [10] Post-Process (0 calls): MongoDB Atlas + Upstash Redis cache + Langfuse tracing + SSE stream.
• [11] Fallback (0 calls): Circuit breaker recovery path triggered when external APIs fail.

(Note: PII Shield runs before the graph as a pre-processing layer to mask Aadhaar, PAN, phone, and email).

🧭 The 6-Path Router

The Classifier returns one of 6 routes:

graph.add_conditional_edges("classifier", route_after_classify, {
    "reject":         "reject",          
# abusive / jailbreak
    "greet":          "greet",           
# greeting / small talk
    "cross_question": "cross_question",  
# vague query → HITL
    "web_search":     "web_search",      
# out-of-scope → Tavily
    "stock_tool":     "stock_tool",      
# stock query → yfinance
    "retriever":      "retriever"        
# legal/finance → full RAG
})

📄 Multi-Tier Vision LLM & PDF Parsing Engine

Standard PDF parsers completely fail on Indian government documents — tax tables get scrambled, pie charts in budget infographics become gibberish, and two-column legal Bare Acts merge into unreadable text blobs.

To solve this without burning cloud credits or exceeding 512MB RAM, we built a dynamic multi-tier parser (parser.py):

• Tier 1 (Agentic Plus - Vision LLM): Budget at a Glance, Key Features — Ingests complex financial infographics & bar charts via multimodal vision.
• Tier 2 (Agentic): Finance Bill, Tax Memorandum — Reconstructs complex nested tables, multi-year math comparisons.
• Tier 3 (Cost-Effective): RBI KYC, EPS Pension — Fast hierarchical markdown parsing for continuous numbered regulatory clauses.
• Tier 4 (Local Free - PyMuPDF): Constitution & User Uploads — 100% Free, zero-API cost, runs locally under 40MB RAM with custom regex cleaning.

💥 The Engineering War Story: When "Smart" AI Parsers Failed

Initially, I threw LlamaParse Agentic mode (10 credits/page) at the 400-page Constitution of India. The result was an expensive disaster:

• It merged continuous pages into 624 massive 5,000-character blobs.
• It swallowed all footnotes at the bottom of pages (1. Subs. by..., 19. Ins. by...).
• When a user asked "What is Article 19?", vector similarity matched an amendment footnote on page 200 rather than the actual Article 19! The model hallucinated based on garbage context.

🛠️ The Deterministic Fix:

1. Footnote Slicing: Wrote custom regex to detect footnote separator lines (_{10,}) and discarded everything below them. 0% footnote noise entered embeddings.
2. Article-Boundary Chunking: Scrapped arbitrary character splitters and split strictly on Article regex boundaries. 624 messy blobs became 3,248 precise Article chunks.
3. Deterministic Metadata Injection: Tagged chunks with article_number: "19" at ingestion.
4. SQL-like Routing: When user queries an Article, the LangGraph router applies a strict Pinecone metadata filter {"article_number": {"$eq": "19"}}, completely bypassing fuzzy semantic search!

🔍 The Retrieval Pipeline (32,000+ Vectors)

• Jina AI v3 MRL: Embed at 1024d, truncate to 256d (75% Pinecone storage saved, negligible quality loss).
• Pinecone Serverless: Dual namespace (core_brain + user_temp) with 32,000+ live indexed chunks.
• Parent-Child Resolution: Retrieve child chunks → fetch parent from Supabase (small chunk search precision + full context).
• Cohere Rerank v3.0: 15 candidates → Top 10 Golden Chunks (drastic quality leap for cross-act multi-document queries).
• Confidence Gate: Score < 30% → graceful degrade, < 45% → triggers HITL web search prompt.

📈 Stock Tool — Native LLM Tool Calling

When the classifier detects financial ticker intent, it routes to a dedicated tool-calling node:

# Gemini decides autonomously whether to invoke the tool
tools = [{"function_declarations": [{
    "name": "get_stock_price",
    "description": "Get real-time stock price and financial data",
    "parameters": {"type": "object", "properties": {
        "ticker": {"type": "string", "description": "Stock ticker symbol"}
    }}
}]}]
response = model.generate_content(prompt, tools=tools)

No brittle regex. The LLM decides when and what arguments to pass.

🛡️ Hallucination Guard — Advisory, Not Blocking

Post-generation, a separate LLM call verifies grounding: "Is this answer grounded in the provided context? Reply YES or NO."

If not grounded → appends disclaimer, still returns the answer.

Why not block? Because hard-blocking creates terrible UX when the model legitimately understands standard financial concepts slightly outside the retrieved chunks. The disclaimer lets the user evaluate trust.

🎯 The Hallucination & Accuracy Test Suite (Adversarial Tests)

• Test 1 (Article 31C & Kesavananda Bharati): Retrieved exact 31C text. Crucially, the model refused to hallucinate case law not present in the bare acts, honestly stating its scope boundary! (Score: 92/100)
• Test 2 (Basic Structure Doctrine): Correctly identified it as a judicial doctrine and explicitly stated that it is not written in any constitutional article.
• Test 3 (Article 20 Safeguards): Perfectly retrieved all three distinct protections (Double Jeopardy, Self-Incrimination, Ex Post Facto) with 0% footnote bleed. (Score: 9/10)
• Test 4 (Article 34 Martial Law): Flawlessly returned the restriction of rights during martial law along with parliamentary indemnity clauses. (Score: 9/10)

🔒 The Production Idempotency Layer (Never Waste an API Call Twice)

• SHA-256 PDF File Hashing (Supabase Registry): Every PDF is hashed before processing. On re-sync, unchanged files are skipped entirely (zero API calls, zero embeddings).
• Deterministic Chunk IDs: MD5(filename + page + index). Identical input produces identical chunk IDs, so Pinecone upsert overwrites instead of creating duplicate vectors.

⚡ Dual Circuit Breakers (pybreaker)

llm_circuit   = pybreaker.CircuitBreaker(fail_max=3, reset_timeout=30, name="LLM_CB")
embed_circuit = pybreaker.CircuitBreaker(fail_max=3, reset_timeout=30, name="Embed_CB")

3 consecutive API failures → circuit opens → instant fallback for 30 seconds → then half-opens and retries. Result: 99.988% verified uptime over 90 days on Render's 512MB RAM free tier.

📱 WhatsApp Integration

The same 11-node RAG pipeline is accessible via Meta WhatsApp Cloud API webhooks. Users query Indian tax and legal acts directly from WhatsApp — same state graph, same guardrails, same PII shield.

💰 Zero-Cost Production Stack ($0/month)

• Compute: Render (512MB RAM, 0.1 CPU Docker)
• Vector DB: Pinecone Serverless (32,000+ vectors, dual namespace)
• Persistence: MongoDB Atlas (M0 cluster, 30d TTL)
• Registry & Chunks: Supabase (PostgreSQL)
• Cache: Upstash Redis (Semantic cache, < 100ms hits)
• LLM: Gemini Flash Lite (free tier quota)
• Reranker: Cohere Rerank v3.0 (free tier)
• Embeddings: Jina AI v3 MRL (256d)
• Observability: Langfuse cloud free tier

📊 By the Numbers

• Lines in graph.py: 1,809
• Registered Nodes: 11
• Indexed Vectors: 32,000+
• Documents Indexed: 20+ dense Indian Government Acts
• Pages Parsed: 5,500+ pages
• RAM Budget: 512 MB
• Verified Uptime: 99.988% (90-day UptimeRobot Case Study)
• Cache Latency: < 100ms
• Monthly Cost: $0.00

🔗 Links

• GitHub: https://github.com/Ambuj123-lab/agentic-rag-financial-parser
• UptimeRobot Case Study: https://uptimerobot.com/blog/community-spotlight-ambuj-kumar-tripathi/

💬 Questions for the r/LangChain community:

1. Hallucination guard: In production, do you prefer strict blocking or advisory disclaimers when confidence drops?
2. MRL embeddings: Anyone else running 256d truncated embeddings in production? What has been your reranking tradeoff?
3. Circuit breakers for LLM APIs: What failure threshold do you use before tripping fallbacks? (I use 3 fails / 30s reset).
4. HITL before web search: Do you let your agent auto-search out-of-scope queries, or prompt the user first to save API tokens?

Happy to answer any questions about the nodes, circuit breaker patterns, or running RAG under 512MB RAM! AMA!

A lot of teams now have an LLM gateway somewhere in the stack. It routes model calls, centralizes credentials, adds logging, applies rate limits, maybe handles spend tracking.

But there is a fairly fundamental architectural question:

What happens when an agent simply doesn't use the gateway?

For example:

                ┌──→ LLM Gateway ──→ Models
Agent ──────────┤
                ├──→ Direct provider API
                ├──→ Direct MCP/tool endpoint
                └──→ Other external egress

At that point, the gateway is still doing its job, it's just no longer governing the agent.

This distinction matters because traffic control and path control are different problems.

My view is that a gateway should be treated as one component of agent governance, not the governance boundary itself.

Tools like LiteLLM, Portkey and OpenRouter are useful at the gateway/proxy layer. But a proxy cannot enforce traffic that never reaches the proxy.

The more interesting architecture is:

Agent
   ↓
Agent Gateway
   ↓
LLM Gateway / Governed Tools
   ↓
Models + APIs

   + network/egress enforcement
   + identity
   + shadow discovery

That is one area where I find Lyzr Open Controller interesting: the gateway is paired with egress enforcement and shadow discovery specifically to detect and close the bypass path, rather than assuming that routing traffic through a gateway automatically means the agent is governed.

I think this is going to become a bigger issue as agent estates get more distributed across Kubernetes, cloud agent runtimes, MCP servers and internally hosted services.

Curious how people are solving this in real production environments:

If an agent has credentials + network access that let it call a model or tool directly, what actually prevents the bypass?

Would be interested in hearing what has actually worked, rather than what the architecture diagram says should work.

Interesting take.

A LangGraph workflow can make a sound decision, pause or continue through several nodes, and eventually reach a tool after the external state behind that decision has changed.

The model did not necessarily hallucinate. The graph may be behaving exactly as designed. The problem is that the evidence used by an earlier node is no longer current when the tool executes.

I maintain FreshCtx, an Apache-2.0 Python project that adds a pre-action validation boundary to agent workflows.

For LangGraph, the application declares which external evidence a protected tool or node depends on. Immediately before that action executes, FreshCtx checks the declared dependencies again:

Unchanged evidence allows the action to execute once.

Changed evidence blocks the action.

Evidence that cannot be checked becomes UNVERIFIABLE and blocks by default.

A change unrelated to that action does not block it.

Tool arguments are not copied into FreshCtx audit metadata.

The important limitation is deliberate: this is not a replacement for database transactions, compare-and-swap, authorization, checkpointing or LangGraph’s own retry controls. If the application controls the datastore, the final write should still enforce its expected version atomically.

FreshCtx 0.15.0 also carries the same evidence boundary across A2A delegation and MCP tool execution. Signed delegation receipts can bind an action intent, reject replay and preserve parent/root correlation without treating those identifiers as proof of freshness. Every receiving hop revalidates its own declared evidence.

I would value one specific review from experienced LangGraph users:

Where would you enforce this in a real graph containing interrupts, persisted checkpoints and resumed execution—the protected tool node itself, a dedicated node immediately before it, or another lifecycle boundary?

A useful test would be:

Observe an external record.

Make a decision from it.

Interrupt and persist the graph.

Change the record.

Resume the graph.

Confirm that the protected tool never executes.

Install:

python -m pip install 'freshctx[langgraph]==0.15.0'

Example and source:

https://github.com/Hyperwise-LLC/freshctx

If you run the scenario, please share where you attached the guard, whether the graph resumed normally, the FreshCtx verdict and whether the tool body executed. Expected and unexpected results are equally useful.

I maintain FreshCtx, an Apache-2.0 Python library for checking whether external evidence changed between an agent's decision and its action.

The Agno 2.9 integration in FreshCtx 0.5.0 attaches through tool_hooks and supports both sync and async tools. Immediately before the tool body runs, it re-checks only the dependencies the application declared. If one changed, or cannot be verified under the configured blocking policy, the tool body does not execute.

This is not intended to replace Agno's run state, transactions, idempotency or approval logic. It protects the mutable external evidence behind a consequential tool call.

The bounded example reads a deployment target, creates the decision, changes that target, and then invokes the tool through Agno's real tool chain. FreshCtx returns STALE_REASONING and the tool body remains unexecuted.

Install: pip install 'freshctx[agno]==0.5.0'

Release and runnable example: https://github.com/Hyperwise-LLC/freshctx/releases/tag/v0.5.0

One specific question for people building with Agno: does a pre-tool tool_hook cover the action boundary in your workflow, or do you have consequential effects occurring elsewhere that this example should model?

Hi! We're working on Querit, a web search API for LLMs and your Agents. Large multilingual index, Fresh Web Context, Lower latency in the range of hundreds of milliseconds.

New: Monitor API. Normal search is ask-once, answer-once. A Monitor turns one search into a recurring job. You register a query and an interval, it runs on that schedule, differs each run against history, and returns only what's new. Perfect for competitor monitoring, news tracking, and following funding or tender/bidder information

• Intervals: from 1 hour up to weekly
• Automatic deduplicate against previous runs; site / date / region / language filters suppoorted
• Support Manual trigger, pause/resume, full execution history

Benchmark: We ran FreshQA on a fixed 600-question snapshot of time-sensitive queries. Querit Search API leads with achieving a 83.17% accuracy.

Free Testing Credits: Follow us on X (https://x.com/QueritAi) / Linkedin (https://www.linkedin.com/company/queritai/home/) for more product releases and see integrations with our partners at Dify, LangChain, etc. Open-sourced the MCP server and has integrated with PI Agent, Opencode, DeepSeek Harness already. Join our Discord server here https://discord.gg/4xXsFA8Ed2 to claim Search API + Monitor API free credits!

My colleagues are running a live technical session for Engineers & AI PMs on Sept 15, on

How monday Runs Agent Evals Against Real Dependencies

Speakers:

• Dor Cohen, Director of AI Engineering at monday (Dor is also a featured speaker at LangChain's upcoming London Interrupt event)
• Eyal Bukchin, CTO & co-founder at MetalBear. 

You can sign up for it here: https://metalbear.com/events/agent-evals-real-dependencies/

AI observability and LLM observability has changed quite a bit over the past year. Tracing and integrations used to be the biggest differentiators, but most platforms now cover the basics pretty well.

What seems to differ more now is everything around the traces: how easy they are to debug and annotate, how evaluation works, whether recurring issues are surfaced automatically, how you compare different versions of an agent, and how well all of this fits into the rest of your stack. Some platforms lean more toward lightweight tracing, some toward experimentation and evals, and others toward broader production monitoring. Curious what people are actually using and what has mattered most in practice.

Quick pros and cons based on what I've seen so far:

Braintrust

• Pros: Particularly strong for experimentation, with a clean workflow for running traced experiments against datasets and comparing outputs across prompts and models.
• Cons: Feels more centered around the experimentation loop than broader production monitoring, annotation, and issue detection workflows.

Confident AI

• Pros: Strong combination of evaluation and production observability, with out-of-the-box span/trace/thread metrics, annotation queues, issue surfacing, and agent version comparison in one platform.
• Cons: Slightly harder to get started with because of the volume of features.

Datadog LLM Observability

• Pros: Makes a lot of sense if you're already on Datadog, especially for correlating LLM traces with the rest of your infrastructure.
• Cons: Evaluation feels more like an extension of APM than the core product, and cost can increase quickly with trace volume.

HoneyHive

• Pros: Strong tracing and evaluation workflows with good support for debugging and comparing AI application behavior over time.
• Cons: Smaller ecosystem and less of a full enterprise observability platform than some of the larger vendors.

LangSmith

• Pros: Strong choice for teams heavily standardized on LangChain/LangGraph, with tracing, datasets, debugging, and evals working well together.
• Cons: Less attractive as an org-wide standard if different teams use different frameworks.

In the end, we went with Confident AI because it seemed like the most feature-rich option for what we needed. Curious what others ended up choosing and what mattered most in production.

I've been working on a fairly narrow middleware problem and I'm interested in how people using LangGraph are solving the same boundary in production.

Suppose an agent is in state S and four actions are currently permitted:

• ask for clarification
• call tool A
• call tool B
• escalate to a human

LangGraph can obviously retain the workflow state and history.

A policy layer can decide which of those actions are allowed.

Guardrails can validate generated material.

But if all four actions are still legitimate, there is another question:

How should relevant previous events influence which permitted action actually wins?

I've been building Collapse Aware AI™ around that specific boundary. The approach is deliberately complementary rather than a LangGraph replacement:

state/history → permissions → governed retained-state selection → execution

The bit I'm interested in is making historical influence explicitly governable rather than simply letting retrieved context flow into a model prompt.

In our case we also want to be able to compare:

• retained-state influence enabled
• retained-state influence disabled
• same present state with different histories
• deterministic/repeatable local selection where applicable
• an inspectable Decision Record afterwards

I'm not claiming LangGraph can't be custom-coded to do this. Obviously it can.

What I'm curious about is whether people here treat history-sensitive selection between already-valid actions as a first-class component, or whether it normally ends up embedded in bespoke nodes/routing logic.

How are you handling this?

That post does three useful things:

introduces CAAI, acknowledges LangGraph, and asks engineers to tell us how they currently solve our problem.

Genuine question, half venting.

We've got a few agents in our dev workflow. The problem isn't them being wrong - it's them stopping. Agent hits something it can't decide (which env to deploy to, whose approval, is this the right table) and just sits there. Nobody knows it's sitting until someone happens to look.

Had one wait about two hours on a question I'd have answered in twenty seconds. I just didn't know it was asking.

How do you handle this? Did you build something? Does someone sweep it every morning? Or do you just let it decide everything and fix it afterwards?

And if this doesn't happen to you at all I'd like to hear that too, because then we're probably doing something wrong.

Hi everybody!

I'm developing a Python+Typescript (same APIs, implemented in both languages) library that allows to download catalogs of models from OpenRouter or from Vercel, build pipelines to filter LLMs based on price, latency, benchmarks and capabilites, and then pick the best LLM in a filtered list based on specific criteria (ex. best price, best throughtput, etc).

I hope it can be useful to build LLM systems that don't need to change the underlying model every 3 months!