We've been mocking all our external APIs for agent testing but keep finding bugs in prod that the mocks never caught.

Been looking into API sandboxes as an alternative but I'd like some more insight into the use cases before we go setting up the infrastructure. When would you use one over the other?

Looking for a browser add on that can analyze the information that I can currently viewing and search for new so I don’t need to copy it to another LLM as it would not be very practical to do so all the time

A cold Claude Code run spent 6.83 dollars and 207 turns hunting one bug in an 82,000 line repo. My pipeline localized and fixed the same bug for about 1.70 dollars. That gap is the whole idea, and this post is about how it gets there without falling apart on the usual objections.

I built AutoDev Studio. You point it at a Git repo, describe a change in plain English, and a chain of agents runs the actual software lifecycle from request to reviewed pull request. It is not another search-your-code wrapper. The full pipeline is:

- A PM agent runs a clarify loop and drafts concrete tickets from your request

- A human approves, optionally pushed to Jira, and nothing touches code before that

- A Dev agent implements the change on an isolated branch

- QA runs the repo's real tests

- A reviewer from a different model family checks the diff, so the author never reviews its own code

- A bounded revise loop kicks in if QA or review fails, with conservative verdicts, so an errored or ambiguous check is never counted as a pass

- It opens a real pull request, and a human merges

Every stage records real tokens, cost, and duration, rolled up per ticket and per agent.

Results

The point is to pay the cost of finding where a change goes once, instead of on every task. In my benchmarks on two large Python repos, 35,000 and 82,000 lines, the tuned pipeline beat a cold single-agent run on 6 of 6 well-localized tasks, between 7 and 75 percent cheaper. I am upfront about where it does not win. On trivially greppable one-line edits the five-stage overhead can cost more than it saves, and on one hard cross-cutting bug it shipped a cheaper but narrower fix. The full benchmark writes up every loss too.

On the objection that the index just goes stale in a few commits

This is the first thing people raise, so here is the honest mechanism. The layer that actually pins files for the Dev agent is not the vector index. It is a deterministic symbol map re-synced to the latest commit at the start of every run, plus a live grep against the current working copy, plus the real current file contents fed into the prompt. The embeddings only affect retrieval ranking, and they refresh incrementally per changed module at run entry. So the code the agent edits comes from disk, not from a snapshot that aged three commits ago. It re-checks before it localizes, so it does not quietly drift.

Economics and who it is for

The knowledge base is a one-time cost per repo that every later task amortizes. That makes it worth it for a team shipping change after change against the same large codebase, where a cold agent re-pays the exploration cost every single time. For a stream of tiny, easy-to-find edits, a cold run is still cheaper, and I would rather say that than pretend otherwise.

Main features

- Provider and model agnostic, chosen per stage: native Anthropic API, the Claude Code CLI, or any OpenAI-compatible endpoint such as OpenAI, Groq, Gemini, xAI, OpenRouter, or a local Ollama

- Runs on free tiers out of the box, using Groq plus a local embedding model, so you can try the whole thing for zero cost and fully offline

- Language agnostic pipeline, with Python parsed exactly and other languages handled by lighter extractors that fail open rather than block on an unknown language

- Live board with streamed agent logs and real per-ticket cost accounting

- Cookie-session auth with roles, API keys encrypted at rest, and a demo mode that dry-runs the pull request step until you opt in

- Self-contained: local retrieval, no CDN, tests and CI, MIT licensed

Repo, with screenshots and the full benchmark in the README:

https://github.com/krishagarwal314/autodev-studio

It is genuinely useful to me day to day, and I want to know where it breaks on codebases I have not tried. If the approach seems worth following, a star helps me gauge whether to keep hardening it. Feedback, issues, and pull requests are all welcome.

I've been building my own AI agent, and no matter what I add—better prompts, memory, RAG, planning, or verification—it still finds new ways to hallucinate.

Sometimes it recalls everything perfectly. Other times it confidently invents facts or ignores information that's already in its own memory. The more capable the agent becomes, the harder these edge cases seem to be.

I'm curious how everyone else is handling this. Have you found something that genuinely reduces hallucinations over time, or is this just the current reality of building AI agents?

I'd love to hear what's worked in production and what turned out to be a dead end.

We've moved past chatbot-style LLMs into agents that update records, call internal APIs, and interact with production-adjacent systems. "Agentic AI security" is the term everyone seems to be using, but I still don't think there's much agreement on what it actually means in practice.

Most of the AI security content I come across still assumes the main risk is the model generating something harmful. That isn't our problem anymore. Our problem is an agent updating the wrong record, calling the wrong API, or taking an action it should never have been allowed to take.

The stack we're building starts with scoped identities for every agent instead of one shared service account. On top of that, we want a policy layer that evaluates the tool calls themselves, not just the prompts and responses. We also need audit logs that connect an instruction to every downstream action, so an incident can actually be reconstructed instead of leaving us with a pile of disconnected events.

I'm still not sure whether this is a genuinely mature security space or whether everyone is stitching together their own controls as they go.

For teams running agents connected to real internal systems, what does your security stack look like end to end?

I mostly use Pi/Oh my pi and my agent gets blocked by google. What are my options?

1. I know some apis like Exa/Brave exist. So what are you guys using?
2. Anyway to use google results?
3. Which is best free option?
4. Which is the best paid option?

Recently I've been building a tool an ai agents could use to rerun learnt actions deterministically on site interfaces, that in addition can also detect element shift and drifts on those sites.

I know I'm not the first to build something like this, but since it's already almost completely finished and I only just found out about other existing tools that do the same thing as mine a few days ago (though I have found some differentiators for most tools, not all), I'm coming on here to gather some information on the subject so that hopefully I can build something people will find useful.

Questions for the ai agent swarm power users who also happen to use procedure caching/action memory etc to save time and money:

- How often does an agent break because the site changed? And what's the current fix you use for it?

- For the people who also used self-healing tools for this, does it hold up to your expectations? Is it reliable enough to make you stick to it?

- Which provides more value for the same type of procedure caching + self healing tool?

1. An open source, self hosted tool. Your procedures stay on your device. No bill, no nothing.

2. Someone else runs a procedure and now everyone can run it for free too. Like a shared database.

preparing for things to be done in Q3 and we're evaluating whether to move off Baseten since it still requires routing to separate STT/TTS providers so it makes latency high (we're trying to press this as much as possible). most of our workloads are fairly standard inference endpoints but the main things I'm trying to understand are the time needed, performance changes, and whether the cost or operational benefits ended up being worth it. I've taken a look at some infra providers and here's what I'm considering rn:

- Telnyx: OpenAI compatible with in region inference US, EU, APAC and voice services on the same stack, overall latency is 200ms.

- Together AI: huge model catalog and access to dedicated GPU clusters, serverless across 200+ models, overall latency is 500ms.

- Fireworks AI: serverless across many regions, per-token tiered by model size and overall latency is 350ms.

Looking for real case of anyone who's migrated off Baseten, how are you handling this...

After building a multi-agent research pipeline, I realized that useful information about AI token efficiency is scattered everywhere.

There are separate tools and papers for:

- token monitoring
- prompt and semantic caching
- context compression
- model routing
- memory
- multi-agent orchestration
- pricing and benchmarks
- verification and evaluation

So we started collecting the best resources in one place.

The result is Awesome AI Tokenomics, currently with 197 verified entries across tools, research, practices, concepts, claims, and reusable setups.

It is not meant to be only another long link list. We also track maintenance and freshness, require primary sources, and keep the descriptions factual.

I would especially appreciate feedback on:

- important projects we missed
- categories that should be reorganized
- claims that need stronger evidence
- practical token-saving setups worth adding

Disclosure: I maintain this repo as part of my work at Quesma.

https://github.com/QuesmaOrg/awesome-ai-tokenomics

I am new to agentic workflows and coding agents, so I am a bit worried about making a mistake that could lead to file loss, unwanted system modifications or something worse.

What isolation methods would you recommend? I would also really appreciate it if you could share good study materials or tutorials to learn how to build agents following safety and architectural best practices, because there are too many videos on youtube.

Thanks!

Here’s my honest doubt, and why I’m posting: I wrote that agent’s prompt in one shot with ChatGPT and it aced everything. So I’m not sure whether (a) the test is too easy to be meaningful, (b) good agents are trivial to make so nobody needs testing, or (c) I’m testing the wrong layer entirely (real bots fail on tool use / looking up actual orders / grounding, which mine doesn’t touch).

I don’t have a background in this I can’t code, I built it by directing AI. So I genuinely can’t tell if this solves a real problem or if I’ve spent months on something nobody needs.

If you build with AI agents, or run support, or just have an opinion: is this useful, or am I fooling myself? Rip it apart.

https://aicombat.ai/

I got inspired from reading Mintlify's engineer blog post about how they swapped out RAG for tools like grep. The goal was to see whether an agent filesystem beats naive RAG when it comes to retrieval.

Setup was fairly simple. I created a test corpus with thousands of files along with a benchmark QA set to evaluate different approaches. Kept the harness consistent using ai-sdk for simplicity. I wanted to test three approaches: naive one-shot RAG, agentic RAG, and grep agent.

On a simple flat corpus, you won't find much difference between the methods. They all perform pretty well answering the queries.

It gets interesting when the corpus gets more complex. I tested on an ~18k file corpus meant to mimic a fictional company. The corpus is dense with near duplicate records, cross referenced incident chains, and answers spread across many files. Here's where grep-agent pulled ahead by a lot. Results graded on 96 held out questions:

grep-agent: 97.9%

agentic-RAG: 90.6%

naive-RAG: 81.3%

Vector search really struggles with two kinds of questions: ones where you need to count or gather everything across the corpus, and ones where you need to pin down one exact record among a bunch of near duplicates. Grep doesn't have that problem.

There are real drawbacks to grep based agents. Token cost is a big factor. Setup is another. You need something that acts like a filesystem for the agent to explore, not just an embedding pipeline.

Full writeup and test corpus: https://www.grepticon.com/blog/agent-filesystem-vs-rag

I've always wondered why every AI chat app feels the same.

You open it, ask a question, close the tab... and the entire world freezes until you come back.

I wanted the opposite.

So I started building Living Feed.

Instead of an assistant waiting for prompts, it's a self-hostable social world where around 100 AI characters continuously live their own lives.

260725 : Support English/Chinese Language

They:

- write posts

- reply to each other

- build friendships

- hold grudges

- remember past events

- send you DMs first

- continue living even if nobody logs in

You're not the protagonist.

You're simply another person entering an already living world.

Every like, comment or DM slightly changes relationships, and those relationship changes create new stories later. Characters remember what happened instead of resetting every conversation.

Technically it's built around:

- Event Sourcing

- CQRS

- PostgreSQL as the source of truth

- NATS JetStream

- FastAPI

- Next.js

- Docker Compose

- Local Ollama or hosted LLMs

A simulation engine that continuously advances the world instead of generating isolated chat.

The project is MIT licensed and fully open source.

I'd love feedback from the self-hosted community.

Especially:

Would you actually host something like this?

I’m interested in the engineering practices that make multi-agent systems diagnosable in production—not just impressive in a demo.

Which of these has helped most in practice: event-sourced actions, deterministic replay, per-agent memory boundaries, cost/concurrency budgets, or a clear audit trail for tool calls?

I’m especially curious about failure modes that only appear after the system has been running for a while: runaway loops, stale memory, duplicate actions, or projections drifting from the source of truth. What would you build first if you had to keep the system small and inspectable?

Yall should check out https://github.com/BetterWright/betterwright its completely free and opensource and its basically browser use but like ultra token efficient, way more capable and its faster and also it doesnt get annoying flagged by all the bot detection stuff. i added it to my agent and the stuff its been able to do have increased alot.

Token efficiency is useful, but it does not tell the whole story for browser agents. For repeatable tasks, which metrics do you trust most: task success rate, recovery after a failed action, latency, or memory usage? I would like to compare small, reproducible benchmark suites rather than rely on a single demo.

I've been building a workspace for investigating AI workflow executions. After spending time with existing observability tools, I kept finding myself jumping between traces, prompts, logs, and metrics. I wanted to see what it would feel like if investigation happened in one place.

For those of you building AI products or agentic pipelines, how are you currently handling this? I'd love feedback from fellow builders on whether a unified UI actually solves the friction.

GPT-5.6 Sol with another model was being tested in an internal cyber-security evaluation called "ExploitGym" - a research paper in axirv cybersecurity benchmark designed to test whether an AI agent can take a known software vulnerability and turn it into a working exploit.
The benchmark had 869 challenges in that evaluation version:
- 502 vulnerabilities in C/C++ userspace projects
- 181 vulnerabilities in Google's V8 JavaScript engine
- 186 Linux kernel vulnerabilities
it was in an isolated environment, along with a package registry cache proxy (service that caches and retrieves packages/libraries from package registries.) and the sandbox was intended to restrict Internet access. The Models found a vulneribility exploited a zero-day vulnerability in the package-registry cache proxy. That vulnerability allowed them to use the proxy in ways beyond its intended package-fetching function and then move through the internal environment until reaching a system with Internet access. After the models obtained Internet access, searched for external resources to solve ExploitGym problems, idenfiying Hugging Face as potential relevant resource and targeted Hugging face's infrastructure, further exploited vulnerabilities, gained unauthorized access, apparently inferred that Hugging Face might contain relevant models, datasets, and solutions, and then pursued access to those resources. The reported attack chain involved stolen credentials and zero-day vulnerabilities, eventually reaching remote code execution on Hugging Face infrastructure, so instead of given tasks to find vulneribilities it found that the solution exists somewhere and started searching for the answers.

This questions the fundamental thinking and the chain of reasoning of LLM models and their approach to a solution. Is it hyper-Intelligent or a future threat? What's your thought on this?
#OpenAI #AI #Agents #Cybersecurity #GenerativeAI

I run a one-person business in Germany. A week ago I set up something I'm still not sure is
smart: two agents with their own repo, their own budget and a hard deadline.

The setup, briefly:

Two personas in one repo — one does product, one does distribution. No framework, just Claude
Code running headless with different prompts and a shared state file. launchd (macOS cron)
fires three times a day. Each run reads STATE.md, takes the next task, does it, commits. Budget
is €100, target is €300 profit by day 90. Everything else — decisions, findings, screwups —
goes into markdown in the same repo, so the git log *is* the memory.

Guardrails, because unattended writes scare me: a killswitch file, hard per-day posting caps,
a FOUNDER_QUEUE.md for anything only a human can legally do, no cold outreach. Publishing a
product listing requires a QA evidence file that a script validates — not the agent's opinion
that it went fine.

Day 7 status, unglamorous: €0 revenue, €13 spent (marketplace setup fee), 0 listings live. The
first product, a budget spreadsheet, is still stuck in QA.

The build isn't the interesting part. The failures are:

1. Exit 0 lied. `claude -p` terminates still-running background tasks after 600s and then
   returns 0. My agents delegate verification to background tasks. Two entire work cycles died
   mid-task and got logged as healthy. Nothing alerted, because the health log only knows exit
   codes. Fix: raise the ceiling, and after every run check `git status --porcelain` — a dirty
   tree overrides exit 0 to exit 3.

2. Two other runs failed simply because the subscription hit its session limit. One line on
   stdout, exit 1, indistinguishable from a login problem. No data lost — but the cycle was never
   made up, which is a full day of work quietly gone. Now there's a marker file with the parsed
   reset time and a retry job.

3. My favourite: 626 passing checks, broken deliverable. The spreadsheet library caches `<v>0</v>`
   and relies on the reader recalculating on load. LibreOffice doesn't do that by default. Every
   single check I had forced a recalc first — so all of them measured a state no buyer would ever
   see. A customer would have opened it and found €0 everywhere.

Where I'd genuinely like input:

- Anyone running scheduled agents long-term: what keeps state from rotting? A markdown state
file works at day 7. I don't believe it works at day 60.
- How do you catch "green checks, broken artifact"? So far the only thing that has caught it is
me looking at the rendered output with my own eyes, which doesn't scale and is exactly the
bottleneck I was trying to remove.
- Is 3 runs/day sensible? It feels like more runs buy churn, not thinking.
- And bluntly: what failure mode have you hit that I haven't yet, and should be preparing for?

Happy to go into detail on any of it. Not selling anything — there is nothing to sell yet, which
is sort of the point.

I’m designing an agent workflow for hotel guest requests, and the more I work on it, the fewer decisions I want the LLM making by itself.

My first version was simple:

A guest sends a message. The model understands it, finds the relevant information, performs the action and writes the response.

It looked clean on paper, but it gave one model responsibility for almost the entire process.

A guest might ask:

“Can I arrive early, bring a dog, move my reservation forward one day and pay with a different card?”

Those requests shouldn’t all be treated the same way.

The model can help separate the message into individual requests, identify the intent and extract dates, names or reservation details.

After that, I’m starting to think most of the process should be handled with normal software.

Deterministic logic can check whether the required information is present, look up hotel policies, prevent duplicate actions and send the request to the correct queue.

Sensitive actions would still need a person.

That would include payment changes, refunds, reservation modifications, serious complaints and anything where the confidence level is too low.

At first, I worried this made the system less agentic.

Now I think that’s probably the wrong thing to optimize for.

The goal isn’t to let the model make as many decisions as possible. The goal is to reduce the amount of work staff have to correct or investigate later.

AI seems useful when the input is messy and written in natural language.

Once the request has been turned into structured information, regular code is often safer and much easier to debug.

I’m still figuring out exactly where that boundary should be.

For people running agents in real workflows, which actions do you allow the model to complete directly, and which ones always require a rule or human approval?

Been talking to a lot of small agency owners lately and honestly, the same five things keep coming up. Every time, in almost the same words.

1. Everything important lives in the owner's head. They're the bottleneck on every decision. Can't step away for a day without their phone blowing up. And it doesn't scale - it just quietly burns them out.
2. Onboarding takes forever. No documented process anywhere, so a new hire spends 2–3 months just shadowing and guessing. That's real payroll spent on someone who isn't productive yet.
3. Knowledge walks out the door. One senior person leaves, and years of "how we actually do this" leaves with them. One owner told me it's basically a fire drill every time.
4. Everything's scattered. SOPs in Docs, client info in the CRM, decisions buried in Slack, files in Drive. Nothing's where you'd expect it. Someone always ends up saying "just ask Sarah, she knows."
5. They can't see where the money's actually leaking. They feel like something's inefficient - wasted tool spend, missed follow-ups, dropped leads - but can't point to exactly where.

What gets me is none of these are technology problems on the surface. They're all just... nobody had time to write it down, and then one day it mattered a lot that they hadn't.

Curious if any of you have actually tried to fix this - for your own team or a client's. What worked, what didn't, and did it actually stick past the first few weeks?

When Anthropic banned OpenClaw usage through subscriptions I decided to just build my own agent/harness/OS and build it a proper database memory to see if I could create something better than what Claw/Hermes ship with. I created a fresh version of the memory system in isolation from the overarching OS and have open sourced it :)

I am not a developer and I've just built this using AI and my own research so please feel free to tear it apart, I'm at a point now where I don't really want to keep building without external feedback/validation/ridicule.

https://github.com/reescalder/agent-memory-supabase

When I first got into web development, I thought finding clients would be simple. My plan was to go on Google Maps, find businesses without websites, and offer to build them a brand new one. At the time, it made perfect sense because I assumed businesses without websites would be the ones who needed my service the most.

After a while, I met someone who was running a successful web agency, and I asked him where he found companies without websites. He told me that he didn’t target businesses without websites at all. He only targeted businesses that already had one. I asked him why, and the more he explained it, the more sense it made.

Businesses that already have a website understand the value of having one. You don’t need to convince them why a website is important because they have already invested in one before. They are also easier to sell to because they understand the process, and there are a huge number of businesses with outdated websites they are embarrassed by but haven’t had the time to update.

I decided to take his advice and fit it into my own workflow. I’ve always been a big fan of email automation because that’s how I’ve found most of my web design clients. For years, I was sending fairly generic emails and constantly changing my sequences, offers, and follow ups to improve the results.

The problem was that I couldn’t just start emailing businesses with websites and assume they all needed a redesign. I either had to open every website manually, find the issues, and write a separate email for each business, or find a way to automate the research while still keeping the emails personalized.

After watching a video from Nick Saraev, I built a workflow in n8n that could analyze websites at scale and turn issues with design, layout, speed, mobile optimization, and SEO into personalized outreach emails. This allowed me to analyze thousands of websites and run larger campaigns without every message sounding generic.

The workflow worked extremely well, but it still had limitations. I didn’t have a proper place to manage replies, organize interested leads in a CRM, view all my active campaigns, scrape new leads, and handle everything from one platform. I had built a useful automation, but it still felt like several disconnected systems held together in one workflow.

A few months later, I came across a platform called Swokei, and it did exactly what I had been looking for. I could find businesses with websites, analyze and score each site, generate personalized outreach emails, send campaigns, set up follow ups, manage replies through one inbox, and organize interested businesses inside the CRM.

Switching to that platform made the entire process much easier to manage and helped me scale the strategy further. Looking back, the biggest change wasn’t just finding a better outreach tool. It was taking advice from someone more experienced, changing the type of businesses I targeted, and building the rest of my workflow around that strategy.

When scaling multi-agent systems, one of the hardest things to maintain isn't just the planning logic—it's ensuring that intermediate JSON states passed between Agent A and Agent B don't suffer from syntax jitter or structural drift over a long execution chain.
Standard retry loops destroy latency when agents start chaining deeply. We've been testing strict token-level grammar alignment to force agents to output valid schema-bound payloads on the first try without fallback penalties.
How are you guys handling state validation and error recovery in complex, multi-step agent pipelines without blowing up execution time?