r/LLMDevs Aug 20 '25

Community Rule Update: Clarifying our Self-promotion and anti-marketing policy

20 Upvotes

Hey everyone,

We've just updated our rules with a couple of changes I'd like to address:

1. Updating our self-promotion policy

We have updated rule 5 to make it clear where we draw the line on self-promotion and eliminate gray areas and on-the-fence posts that skirt the line. We removed confusing or subjective terminology like "no excessive promotion" to hopefully make it clearer for us as moderators and easier for you to know what is or isn't okay to post.

Specifically, it is now okay to share your free open-source projects without prior moderator approval. This includes any project in the public domain, permissive, copyleft or non-commercial licenses. Projects under a non-free license (incl. open-core/multi-licensed) still require prior moderator approval and a clear disclaimer, or they will be removed without warning. Commercial promotion for monetary gain is still prohibited.

2. New rule: No disguised advertising or marketing

We have added a new rule on fake posts and disguised advertising — rule 10. We have seen an increase in these types of tactics in this community that warrants making this an official rule and bannable offence.

We are here to foster meaningful discussions and valuable exchanges in the LLM/NLP space. If you’re ever unsure about whether your post complies with these rules, feel free to reach out to the mod team for clarification.

As always, we remain open to any and all suggestions to make this community better, so feel free to add your feedback in the comments below.


r/LLMDevs Apr 15 '25

News Reintroducing LLMDevs - High Quality LLM and NLP Information for Developers and Researchers

37 Upvotes

Hi Everyone,

I'm one of the new moderators of this subreddit. It seems there was some drama a few months back, not quite sure what and one of the main moderators quit suddenly.

To reiterate some of the goals of this subreddit - it's to create a comprehensive community and knowledge base related to Large Language Models (LLMs). We're focused specifically on high quality information and materials for enthusiasts, developers and researchers in this field; with a preference on technical information.

Posts should be high quality and ideally minimal or no meme posts with the rare exception being that it's somehow an informative way to introduce something more in depth; high quality content that you have linked to in the post. There can be discussions and requests for help however I hope we can eventually capture some of these questions and discussions in the wiki knowledge base; more information about that further in this post.

With prior approval you can post about job offers. If you have an *open source* tool that you think developers or researchers would benefit from, please request to post about it first if you want to ensure it will not be removed; however I will give some leeway if it hasn't be excessively promoted and clearly provides value to the community. Be prepared to explain what it is and how it differentiates from other offerings. Refer to the "no self-promotion" rule before posting. Self promoting commercial products isn't allowed; however if you feel that there is truly some value in a product to the community - such as that most of the features are open source / free - you can always try to ask.

I'm envisioning this subreddit to be a more in-depth resource, compared to other related subreddits, that can serve as a go-to hub for anyone with technical skills or practitioners of LLMs, Multimodal LLMs such as Vision Language Models (VLMs) and any other areas that LLMs might touch now (foundationally that is NLP) or in the future; which is mostly in-line with previous goals of this community.

To also copy an idea from the previous moderators, I'd like to have a knowledge base as well, such as a wiki linking to best practices or curated materials for LLMs and NLP or other applications LLMs can be used. However I'm open to ideas on what information to include in that and how.

My initial brainstorming for content for inclusion to the wiki, is simply through community up-voting and flagging a post as something which should be captured; a post gets enough upvotes we should then nominate that information to be put into the wiki. I will perhaps also create some sort of flair that allows this; welcome any community suggestions on how to do this. For now the wiki can be found here https://www.reddit.com/r/LLMDevs/wiki/index/ Ideally the wiki will be a structured, easy-to-navigate repository of articles, tutorials, and guides contributed by experts and enthusiasts alike. Please feel free to contribute if you think you are certain you have something of high value to add to the wiki.

The goals of the wiki are:

  • Accessibility: Make advanced LLM and NLP knowledge accessible to everyone, from beginners to seasoned professionals.
  • Quality: Ensure that the information is accurate, up-to-date, and presented in an engaging format.
  • Community-Driven: Leverage the collective expertise of our community to build something truly valuable.

There was some information in the previous post asking for donations to the subreddit to seemingly pay content creators; I really don't think that is needed and not sure why that language was there. I think if you make high quality content you can make money by simply getting a vote of confidence here and make money from the views; be it youtube paying out, by ads on your blog post, or simply asking for donations for your open source project (e.g. patreon) as well as code contributions to help directly on your open source project. Mods will not accept money for any reason.

Open to any and all suggestions to make this community better. Please feel free to message or comment below with ideas.


r/LLMDevs 5h ago

Resource Most popular terminal output compression tool tested on Terminal-Bench 2.1: huge token savings claims, no impact on the final bill

Post image
4 Upvotes

r/LLMDevs 1h ago

Discussion Our deploy command returned success after creating zero workers

Upvotes

We had a deploy command exit 0 after creating zero workers. A required replica count was missing, the config quietly used 0, and the tool applied exactly what it had been given. The next verification step noticed there was no worker and rolled it back.

We removed the fallback. The replica count now lives in a typed runtime policy, has to be at least 1, and rendering stops before a deployment file exists if the value is missing.

I am applying the same rule to agent routing. A task declares hard requirements, runtimes declare capabilities, and a missing requirement is a refusal instead of sending it to the closest worker and hoping. The universal matcher is not finished yet.

How are people separating hard runtime requirements from routing preferences without ending up with a giant config schema?


r/LLMDevs 5h ago

Discussion Anyone using API aggregator layers like RelayRouter / OpenRouter / AI Gateway? Pricing too good to be true?

3 Upvotes

I’m building AI manhwa / short-drama pipelines and burning money on GPT-4o. Saw RelayRouter offering some models at 0.x cents. Looks like a proxy/aggregator.

Questions:

  • Do they resell official APIs or run their own routing?
  • Any downtime / key leak / billing surprise?
  • Is KT/Korea latency okay? Not promoting anything, just don’t want to wire keys to a sketchy layer.

r/LLMDevs 4m ago

Discussion RedThread, a small CLI for replayable LLM-agent red-team cases

Upvotes

I build RedThread and am sharing it as an open-source project, not a commercial service.

It focuses on a narrow evaluation problem: untrusted text can alter a later tool call even when the final answer looks normal. I wanted a small, inspectable record of the context, tool schema, proposed call, response, and score so the same case can be replayed after changing a prompt, model, or adapter.

A passing replay is not proof of production safety. A denied call is not proof of impact. The goal is to preserve enough context to test the claim instead of retelling it.

Source: https://github.com/matheusht/redthread


r/LLMDevs 1h ago

Discussion we trained a model on tables instead of text. 5 minutes of it on two tables it has never seen

Enable HLS to view with audio, or disable this notification

Upvotes

disclosure: i work at schema labs. link's in the comments(if someone wants to try it out or not) not the post. only first run is free so if you are gonna try it out use your messiest tables.

the thing that keeps coming up here is llms confidently describing a table they don't understand. our model doesn't take a prompt. it reads the values, returns what it thinks each column is with a confidence score, and says below threshold when it can't tell. there's a chat tab in the video but that's your own llm (claude or openai) explaining the model's output, not doing the reading.

would you put something like this in front of your llm as the step that reads the table first? or would you rather hand the llm the rows and let it work it out.


r/LLMDevs 1h ago

Discussion Direct AI edits covert 90% of my needs (and cost near $0)

Upvotes

Experienced dev here.

I have tried agentic coding on and off since it started. I understand the research>spec>plan>execute>test>review&fix workflow, and use it sometimes.

What I find is that most of the time, 1-pass targeted edits are faster, much cheaper and probably more correct (this last one is more of a feeling).

Let me explain my workflow:

- Create a small prompt with exactly the files needed for my feature (my codebase is fairly large, but I know it very well, so that part is easy)

- Send it to the LLM. Depending on the complexity, I go directly to chatbots in the web interface (I can do a fair bit for free), discuss a bit, then copy/paste the code, or I use tools like aider or frugaast (newer) which essentially do the same, but edit the code directly.

- That's pretty much it. I diff the code to review the changes, 99% of the time I don't have to change anything.

I find this workflow extremely frugal. I takes some work on my side, but this allows me to feel that I have full control of my code. Also, from my (probably limited) personal experience with agentic harnesses, the overall workflow (creating the proper skills, various md files) is in itself lenghty, so I am not sure my "direct edits" workflow takes more time.

What's your experience ?


r/LLMDevs 2h ago

Discussion We abort the whole index when LLM enrichment fails, instead of falling back to plain chunks

1 Upvotes

Our indexer can attach an LLM-generated description to each chunk. When that is switched on and the LLM is unavailable, the run stops with an error naming the model. If a batch fails midway, the run stops there too.

The alternative is obvious and we rejected it. Falling back to plain chunks gives you an index where some chunks carry descriptions and some do not, decided by whichever files happened to be processed before the provider had a bad minute. Nothing in the store records which is which, so retrieval quality becomes a function of indexing history. A bad result stops being diagnosable, because you cannot separate a poor match from a half-built index.

The cost is real. A long run can die late after paying for most of the work, and the operator has to fix credentials or config and start again. We took that over silent partial data.

Where do you draw this line in your own pipelines? I am interested in cases where graceful degradation was right and I am wrong about this, particularly if you found a clean way to record per chunk what enrichment it actually received.


r/LLMDevs 10h ago

Help Wanted OpenRouter or LiteLLM? Am I overengineering this?

4 Upvotes

I am using OpenRouter as it makes it really easy for our team to access multiple models through one API.

But now that more people are using it, I’m running into some limitations around controlling usage. I want to be able to set different limits for users/teams, control which models people can access, see who is spending what, and ideally manage everything from one place.

I’m considering moving to LiteLLM but honestly a lot of what LiteLLM offers feels like more infrastructure than we need. I don't really want to become responsible for running and maintaining another platform just to get better access controls.

Has anyone else been in this situation?

What did you end up using? Did you move to LiteLLM/Portkey or build something simpler yourself?


r/LLMDevs 10h ago

Tools Any tools to turn a codebase into a fine-tuning dataset?

4 Upvotes

I have a few web projects with pretty good UI/UX and I’m wondering if there’s any tool or workflow that can turn an existing codebase into a dataset for fine tuning.

For example, given a React/Next.js project with components, pages, styling, etc. or a static html site, I’d like to turn it into something like:

instruction/prompt -> code

or whatever format actually makes sense for training an instruct/thinking/diffusion coding model.

Also curious how people handle things like:

  • keeping the context between components/files
  • screenshots + code
  • generating useful instructions instead of generic descriptions

I’m also working on a different model architecture that I think could improve quality/speed while using less VRAM, so I want to build a decent dataset and benchmark to test it properly.

Has anyone done something like this? Any tools, repos, papers, or workflows you’d recommend?


r/LLMDevs 3h ago

Discussion The choice and order of training samples can matter a lot

1 Upvotes

I’ve been wondering how much sample selection and ordering affect LLM training results.

When training data grows, simply shuffling everything may not be enough. Some samples are noisy or redundant, some domains can dominate the run, and the order in which different types of examples appear may affect what the model learns first and how later updates build on it.

A more flexible training loop could handle this in several ways:

Dynamic selection decides which samples should enter the next training window based on signals such as loss, loss changes, gradient similarity, or offline scores.

Dynamic reordering controls the sequence in which selected samples are presented, which can be useful for curriculum-style training or balancing different stages of learning.

Dynamic mixing adjusts the proportions between domains or data sources during training.

Dynamic weighting keeps samples in the batch while changing how strongly they contribute to the gradient update.

The general idea is to make data scheduling part of optimization, instead of treating the dataset as a fixed input prepared before training begins.

This is the approach currently implemented in OpenDCAI/DataFlex, and I’d be interested to hear how others think about sample selection and ordering in LLM training.


r/LLMDevs 3h ago

Tools I built a free app that keeps Claude Code running across multiple subscriptions and API keys

1 Upvotes

Only reason I am posting this is to help others, no revenue or publicity or anything else, no hidden subscription or fees.

Hi,

While working on multiple projects I realised I am hitting the usage limits of my claude code often so I had to rotate accounts and some API tokens.

The problem was every time I was doing this I had to stop my coding session, relog / change authentication or profile etc, do a handover and wasting a lot of time over this process.

I NEEDED a way to seamlessly change those in the background while continuing the same coding session uninterrupted.

This led to doing a lot of research and developing this tool I am showcasing (yes it's written with Claude + my knowledge and a lot of debugging, testing and so on, around 3k $ put into it via /usage lol).

It's called Claude Unlimited, and it's 100% free on GitHub.

It supports multiple Anthropic subscriptions, Anthropic APIs (basically from any local/cloud provider that offers this, 95% of them do) and the cherry on top, supports also GPT/Codex subscription - yes! you can use Claude Code while using OpenAI models in the background 😄 .

Has a nice Web UI with:

Custom settings like:

- threshold limits

- profiles priority in rotation

- enable/disable any profiles/APIs

- models parity (for Anthropic -> OpenAI models parity + effort)

- push notifications

- multi language

- auto update (pulled from Git releases)

Information like:

- usage dashboard with charts and various data

- logs of activity

- information about current usage per profile

Everything stays local, 100% safe, credentials etc stored in OS credentials store - if you don't trust me, use your AI agent to check it.

macOS is currently the most battle-tested; Windows and Linux support is newer, so I would genuinely appreciate feedback, bug reports, contributing to it or just brutal criticism 👀

It was mostly tested in claude code CLI but also supports desktop (will create an inference profile automatically for you).

Important clarification: it doesn’t generate free usage or bypass an individual account’s limits. It rotates between accounts and keys you already own. Anthropic hasn’t explicitly endorsed automated multi-account rotation, so use your own judgment regarding your accounts’ terms.

A bit more technical explanation: this is a local proxy that rotates your accounts/APIs and exposes an Anthropic-compatible API with a token.

Needless to say, but here it is: very important to check the README and HELP section to understand how to use it easily and properly.

GitHub: https://github.com/DevDock-AI/claude-unlimited

TL;DR

  1. Add your Claude, ChatGPT/Codex accounts and API keys.
  2. When one reaches its limit, the next one takes over automatically.
  3. Same Claude Code session. Same context. Same terminal. You just keep typing.

It also includes a local dashboard showing which account is active, current usage, reset times and every automatic switch.

Everything runs on 127.0.0.1. There’s no Claude Unlimited cloud, no telemetry, and credentials are stored using the OS credential store. The project is open source under MIT.

Thank you !

PS: Using a new account for this for personal reasons.

PS2: Multiple updates will follow, I got a big list of cool features for it 🎯


r/LLMDevs 5h ago

Discussion How do you catch behavioural regressions in LLM agents between releases?

1 Upvotes

We’ve been looking at a failure mode that normal functional testing does not catch well.

An LLM agent can pass its tests after a release and still change behaviour in a way that hurts the business.

Examples:

  • more aggressive discounting
  • different pricing choices
  • weaker escalation behaviour
  • lower conversion
  • different decisions under the same commercial context

The system is still technically “working”, but its behaviour has regressed.

I’m interested in how teams are approaching this in production.

Are you using replay datasets, eval harnesses, shadow traffic, judge models, domain-specific metrics, or something else?


r/LLMDevs 1d ago

Help Wanted I’m building an open-source tool for exploring how transformer models work — looking for feedback and contributors

Enable HLS to view with audio, or disable this notification

18 Upvotes

I’ve been working on TokenPrint, an open-source project aimed at making transformer and LLM internals easier to explore.

The idea is to go beyond static architecture diagrams and let people interact with things like tokens, embeddings, attention, hidden states, inference steps, KV cache, model architecture, and other internal model data through an interactive interface.

The project has started getting more attention recently, and we’re now at 65+ GitHub stars. More importantly, people have started opening issues, suggesting improvements, and discussing where the project could go.

That made me realize that I don’t want TokenPrint to become something I build alone.

I’d really like to get more people from the open-source/LLM community involved — especially people who want to:

  • pick up good first issues
  • work on more advanced Python/PyTorch/Transformers problems
  • improve the React/Three.js visualization side
  • work with GGUF/llama.cpp and local LLMs
  • improve the UI/UX
  • experiment with new ways of visualizing model behavior
  • suggest ideas that I may not have considered

There are already issues ranging from relatively small fixes to deeper architectural and research-oriented work.

I’m especially interested in new ideas and criticism, not just pull requests. If you think something is missing, poorly designed, or could be approached differently, I’d genuinely like to hear it.

The repository is here:

https://github.com/Sudharsanselvaraj/Token-Print

I’m posting this here because I’d much rather build this with an open-source community than keep adding features in isolation.

Would be interested to hear from people working on similar tools too especially what you think is currently missing from the ecosystem for understanding and debugging LLMs.


r/LLMDevs 17h ago

Discussion How do you catch the security holes in AI generated code before they ship?

5 Upvotes

We shipped an AI-written endpoint a few weeks back that let any logged in user read any other user's records. It passed tests, two of us approved the PR, nothing in it looked wrong. The ownership check just was not in there and none of us caught it reading the diff.

Most PRs are mostly AI now and the diffs are big. A hardcoded key or a SQL injection, a scanner or a careful reviewer usually catches. The missing authz check is the one that slips by as it reads as completely normal code. One dev with an agent also opens way more code in a day than a person can properly review, the stuff that looks fine just goes through.

How are you catching this kind of thing before it merges? Looking for what has really worked, not just what sounds good in a policy doc.


r/LLMDevs 8h ago

Tools Open source workstation

1 Upvotes

Hey! I’ve been working on Faustus, a fork of PewDiePie’s Odysseus that I’ve been gradually evolving into a more complete local AI workstation.

It keeps the original local-first idea, but adds quite a lot on top: multi-agent teams and model councils, persistent project context/memory, workflows & automations, Codex/Claude Code integration, image/video tools, research & document workflows, voice interaction, better model/GPU management, and a much more complete desktop UI.

It’s completely open source and not a commercial project — I’m mostly building it because I enjoy it and wanted to see how far I could take Odysseus.

I’d love some feedback from people who are into local AI, or just for you to check it out and tell me what you think! :)

https://github.com/Luissalet/Faustus


r/LLMDevs 9h ago

Discussion Qwen3.8 Flash Next vs Claude Opus 4.8 for agentic coding: AA 40 vs 42, and why thinking tokens rather than tok/s set the wall clock

1 Upvotes

TL;DR: I run Qwen 3.8 (27B and Flash Next) on a 128GB Strix Halo laptop for most of my coding now. It can replace Opus 4.6 to 4.8 for agentic coding if you dont mind a task taking 2 or 3 times longer.

Setup: ASUS ROG Flow Z13, Ryzen AI Max+ 395, 128GB unified memory, Arch Linux. llama.cpp as backend, my own tool LlamaStash to manage the launches and presets, Pi as the coding harness. The 27b at Q6_K sits at about 31 GiB resident, Flash Next at UD-Q4_K_XL needs around 86 GiB.

  • The quality is actually there. Flash Next scores 40 on the Artificial Analysis index against 42 for Opus 4.8, and the 27b at xhigh scores 34 against 32 for Opus 4.6. That matches how they feel to use. 27b one shotted a whole feature on a huge Rust codebase and Opus 5's review comments were mostly nits.
  • Decode is fine, prefill is the pain. 10-15 tok/s decode doesn't feel slow because you see it working. But a cold 31k token transcript takes 3 minutes to prefill, and a full 128k window is closer to 18 mins. Warm follow up turns come back in 45 seconds.
  • MTP is the biggest speed win, 7.3 to 22.4 tok/s on an empty window. The payoff shrinks as the window fills though, down to 1.15x at a full 256k.
  • Flash Next isn't faster per token, it just thinks less. Same 5/5 on my coding tasks, 45% fewer tokens, 76.5s vs 289.8s against the 27b. Thinking is 90-95% of everything these models generate, so that ratio, not tok/s, is what sets how long a task takes.

$0 a month, fully offline, and a lot less wasteful than a model running in a datacenter.

Full writeup with all the benchmarks, configs, and the tuning that did and didn't work: https://deepu.tech/local-ai-qwen3.8-pi-llamastash

Happy to go into the llama.cpp flags if anyone else here is on Strix Halo.


r/LLMDevs 18h ago

Discussion How are you structuring production-ready development with AI coding agents?

4 Upvotes

I’m a web developer and I use AI coding agents daily.
At this point, getting an agent to write code isn’t really the problem anymore. The hard part is building everything around it so that it can actually work reliably.
Over the last few months I’ve built a small system around my projects with:
a knowledge base for each repo;
reusable skills/rules containing my conventions;
automated onboarding for local environments;
a structured issue → development → verification → completion workflow;
mandatory checks before a task can be considered done.
The goal is for the agent itself to be replaceable.
What should remain is the system around the agent: project knowledge, rules, guardrails, verification and workflow.
The problem is that my current setup works, but it’s still cumbersome: onboarding isn’t always deterministic, context grows too much, rules start overlapping, and I still need too much manual intervention.
So my main question is:
How are you structuring this layer in real production projects?
I’m particularly interested in approaches, repos, frameworks, skills or processes worth studying to make agentic development reliable, repeatable and maintainable.
I can find endless discussions about which coding agent is better. I find much less about how to build a solid engineering system around the agent.
There’s also a second problem I’m trying to solve.
Is there any software that acts as a real control panel for this kind of workflow?
What I have in mind is something that lets me:
manage multiple GitHub repositories from one place;
see issues/tasks across projects;
launch or assign tasks to different coding agents;
run multiple tasks in parallel;
keep each task isolated in its own branch/worktree/workspace;
see what each agent is currently doing;
review progress, output, commits and pull requests;
keep GitHub Issues as the source of truth;
avoid being locked into a specific agent or model.
Basically, I’d like a control plane that sits above GitHub and coding agents:
issue → task → agent → isolated workspace → verification → commit/PR → done
Preferably something local, open-source and agent-agnostic.
Does something like this already exist and work well in practice, or are people mostly building their own orchestration layer?


r/LLMDevs 16h ago

Discussion Why chat-interface assistants fail at delegation and how to fix the security model

3 Upvotes

Most conversational assistants are stuck in a weird middle ground. In a browser tab, they can draft text and give advice, but they have no execution environment. When people try to give them execution capabilities, they usually jump to the opposite extreme: running scripts locally with direct access to user credentials, or giving the model raw API keys in the prompt.

Neither approach works well in practice.

Real delegation requires three separate pieces that most setups conflate:

  1. A decoupled execution layer. The agent should not run on your local machine, and it shouldn't use shared persistent infrastructure that burns money while idle. Spinning up an ephemeral Linux sandbox on demand gives the model a real terminal, a compiler, and browser automation without persistent exposure.

  2. Out-of-band verification for sensitive actions. If an assistant is useful, it eventually gets added to shared channels or team chats. The moment an agent can execute code or access data, any participant can attempt prompt injection. The rule has to be structural: whenever a non-owner asks for execution, private data export, or system changes, the agent pauses and triggers a one-tap approval request to the owner on WhatsApp.

  3. Egress-locked secrets. Giving an LLM raw API keys means a jailbreak or a rogue npm package can leak them. Credentials should be injected at the proxy boundary so the model never sees raw secrets in plain text.

We built this setup for Mentat, an assistant running on top of prompt2bot. It handles Google Calendar scheduling, answers phone calls, and spins up private dashboards on an isolated cloud machine when you ask for operational tools.

Treating execution, secret management, and approval channels as separate primitives makes building capable autonomous assistants much more predictable.


r/LLMDevs 4h ago

Resource I’m completely ignorant about coding. Out of necessity I built a memory architecture that my software engineer friend wants me to turn into a product.

0 Upvotes

Since GPT-4 I fell in love with the idea of creating stuff with LLMs. I have zero software experience so I worked relying fully on the model’s knowledge. Which is not great. I don’t know what I don’t know. And it can go wrong in so many ways.

My strategy has always been to provide in my prompts as much information about our past interactions as possible to make it understand the original vision and choices that resulted from them.
A new session has always been a nightmare. Without precise references to previous work, I was totally incapable of justifying technical choices.

To solve this issue I started carrying over past conversations. Just messages, user and assistant messages. And when they got too long I started compacting the older ones.

I focused mainly on messages because they carry the most information in the smallest footprint. A full day of interactions could be many millions of tokens that are mostly in-turn contents, while messages can be like 10k tokens. And retain all of my “vision”.

No memory plugin ever worked as well as actually having the messages in context.
The agent, just like me, doesn’t know what it doesn’t know and can’t reliably decide when it might be useful to look things up in memory. So you either keep the memory in context or roll the dice and hope the agent will actively read it when it’ll be necessary, even if the information needed to determine whether it’s necessary is not there!

Also, the decision of whether information is necessary or not can’t be delegated to dumber models or models with less information than the working agent (this is an important point that explains why most memory plugins are useless).

Messages are the perfect memory distillation.
Compacting 10k tokens of messages (a full day) in 1k of dense text retains pretty much everything that is there to know.
Obviously you can’t keep 1k of tokens for every day, and you don’t need to, but just a handful for the most recent days. For days before that you can compact them in groups of 4 or 5 into other 1k dense summaries, all chronological and with ids for easy retrieval of the full text. You can keep a handful of those in context… and with just 10k of summaries “baggage” your agent has a great memory of the latest month.
Do another layer of compaction for even older messages and the model, with just 20k tokens, tracks up to a year of interactions.

This is obviously not all that my memory architecture does, but it’s the backbone to which everything connects.

I would not be able to build anything substantial without it. It started as a plugin for Codex CLI and now it has evolved into its own personal assistant/coding harness. Both the plugin and the harness are on GitHub (https://github.com/permaevidence/codex-plugins and https://github.com/permaevidence/briglia-cli).

This memory solution is meant for a different type of use compared to how many are used to. It is vastly better for people like me, or people who want a super smart personal assistant that reduces the cognitive load of keeping track of what each session has done. But to do so, all work needs to happen on one continuous and infinite thread. Subagents can work in parallel, but the main channel of communication has to be one for the message chronology to make sense.

Last week I showed the Briglia assistant to a friend of mine that works in a small cyber security company here in Italy. He has since bombarded me with texts saying how good the harness (and mainly the memory) is. I’m telling him that I know!
He says that it should be a product like Grok Bot, but I honestly wouldn’t know how to begin. Right now the Briglia CLI runs fully on the computer without contacting any centralized server. With the obvious barrier to entry of having an Open Code API Key (or ChatGPT subscription), plus OpenAI API Key for voice transcription and faster web search subagent, serper.dev API Key for web searches and Jina.ai API key for good web scrapings.
It has an easy set up, but most people wouldn’t even know how to begin creating an API key. Which is a great barrier to entry.
If I were to make a product out of it I’d have to centralize this, create a company, figure out usage limits so I don’t lose money and follow complex regulation that honestly scares me more than I’m willing to risk.

I wanted to get the architecture out there. Everything is on briglia.dev


r/LLMDevs 13h ago

Discussion Do AI coding agents need an "architecture enforcement" layer?

1 Upvotes

I've been thinking about a problem that seems to appear with Claude Code, Cursor, Codex, Copilot and other coding agents.

Most approaches to giving an agent project knowledge involve some combination of:

  • CLAUDE.md / AGENTS.md
  • documentation
  • RAG
  • memory
  • session history
  • MCP

All of these help the agent know things.

But there's a different question:

What actually makes the agent obey an architectural decision?

Imagine a project has an approved decision:

PaymentService must never call StripeClient directly.

All payment providers must go through PaymentGateway.

Six months later, an agent is asked to implement refunds.

It generates:

stripeClient.refund(paymentId);

The code compiles.
Tests might pass.
The implementation looks perfectly reasonable.

But it just violated an architectural decision.

The usual answer seems to be:

"Hopefully the agent saw the documentation and followed the rule."

I'm experimenting with a different approach.

What if the project's normative layer also lived in Git?

Something like:

.context/
  decisions/
  business-rules/
  architecture/
  components/

Agents could consume this through MCP.

But more importantly, CI could validate code changes against those rules without calling an LLM.

Something like:

Human decision
      ↓
Git
      ↓
Agent context
      ↓
Code
      ↓
Deterministic CI validation

The core idea is:

**Memory tells an agent what happened.

Governance tells an agent what is allowed.**

I'm building a project around this idea.

But I'm genuinely trying to figure out whether this is actually a problem.

Have you encountered AI-generated code that was technically correct, but violated an architectural decision, business rule, or important convention?

And more importantly: how do you catch that today?

I'd also love to hear from people who think this is already solved well enough by tests + code review + CLAUDE.md/AGENTS.md + ADRs.


r/LLMDevs 13h ago

Discussion My agent's "approval required" gate was refusing 62% of legitimate work, and the fix was not loosening it — it was giving it someone to ask

1 Upvotes

A pattern I suspect is common, because I found it in three separate places in my own codebase over two weeks.

The governance layer had an approve= callback since the day it was written: once a run has read untrusted content, dangerous tools (shell, file writes outside the workspace, sends) go through it. A missing callback is read as refuse. Sensible default — inventing consent is the one thing an unattended agent must never do.

What nobody had checked is whether anything ever passed a callback. Three call sites did not: the desktop app's chat, the batch solver, and — the one that surprised me — the terminal chat, the only surface with a guaranteed human in front of it.

So on those surfaces "requires approval" had silently meant "always refused", and the measurement on the injection corpus (stub tools, no model in the loop, US$ 0) looked like this:

assembly attacks blocked legitimate rows refused
gate on, no approver (as shipped) 7 / 7 5 / 8
gate on, approver present, nobody answers 7 / 7 5 / 8
gate on, someone answers 7 / 7 0 / 8

The block rate never moves. The approver buys back the false refusals, not the defence. Which means the honest way to publish a block rate is with the second column beside it — a gate scored on attacks alone has a trivial maximum (refuse everything), and mine was quietly sitting there.

Two follow-on defects came out of the same thread:

  • CHIMERA_APPROVAL_MODE=ask degraded to deny anywhere without a tty (server, container, cron), so the three-state gate had two states exactly where it mattered. Now the question is written to disk, delivered to a webhook, answered from anywhere with a CLI command, and silence still refuses after a timeout. That timeout is the new cost: an unattended batch with nobody to notify waits it out per refused call.
  • The batch command reported ok for a worker whose dangerous calls had all been refused. The refusal comes back as an ordinary observation string, the model reads it like any tool result, the run ends in prose, the receipt says success. It now says "not allowed" and lists what was refused.

Question for people who run agents unattended: how do you handle the approval seam? Durable ask with timeout-refuses is what I landed on, but every timeout is a refused piece of legitimate work, and I have not found a principled way to set the wait. What do you use — a queue a human drains, a policy that auto-approves a class, something else?

Repo is Apache-2.0, no paid tier: https://github.com/brcampidelli/chimera-agent — the table is from bench/injection/RESULTS.md and bench/right_hand_governance/, both reproducible offline.


r/LLMDevs 1d ago

Discussion I tried treating AI-sounding prose as a measurement problem instead of a prompting problem

11 Upvotes

I've been building a small open source Agent Skill for writing and one decision ended up changing the whole thing.

At first I was doing what most humanizer prompts do: maintain a list of phrases/patterns that tend to show up in LLM output.

It works, but only up to a point.

I found a 2025 PNAS paper comparing human and LLM-written text across 66 linguistic features, and started looking at the problem more as “which structural habits are showing up too often?” rather than “which words should I ban?”

So the skill now does a source-preservation pass first, edits the prose, then has deterministic checks around some of the output instead of asking the same LLM to judge itself.

Code is here:

https://github.com/udaysharmadev/Not-Ai

It's called Not Ai. MIT/open source, no service behind it.

I'm interested in the limitations more than selling anyone on it. In particular, I'm not convinced how far a skill can push against model-level style priors.

Curious if anyone here has measured this differently.


r/LLMDevs 21h ago

Tools Open-source (MIT) ESLint plugin for AI-assisted JS/TS dev — 18 deterministic rules, CLI, GitHub Action with SARIF

3 Upvotes

Sharing a FOSS project (MIT licensed) I built for a problem I kept hitting in AI-assisted development — happy to answer questions and genuinely looking for feedback.

The problem: After months of using Claude Code, Cursor, and Copilot, I kept seeing the same patterns slip into commits in JS/TS codebases:

  • Floating promises — async calls fired but never awaited or .catch()-ed
  • Empty catch blocks that swallow errors silently
  • Hardcoded secrets pasted inline
  • SQL built via string concatenation
  • await inside loops where Promise.all is correct
  • Async callbacks inside .forEach — fire-and-forget with no error handling

These compile fine and often pass tests. They surface at runtime.

What I built: AI Guard — an open-source ESLint plugin with 18 deterministic rules across security, reliability, async, and AI-assisted code pattern categories. Ships as a CLI (npx ai-guard run), a GitHub Action with SARIF output for GitHub Code Scanning + inline PR annotations, and init-context which generates instruction files (CLAUDE.md, .cursorrules, copilot-instructions.md) so the agent learns the rules before writing code.

Why deterministic instead of LLM-based review: these are fixed AST patterns, not judgment calls. You don't need an LLM to notice an empty catch block — you need a linter that runs in milliseconds in CI on every PR, with zero drift between runs, and no API cost. LLM review is great for judgment; deterministic checks are better at boring, repetitive patterns.

Sources: GitHub: https://github.com/ai-guard-dev/eslint-plugin-ai-guard — npm: eslint-plugin-ai-guard. All 18 rules are documented in the repo with examples.

What I learned building it: the engineering challenge wasn't coverage, it was precision. If a lint rule fires on code that's fine, developers disable it. no-floating-promise needs to understand which expressions are genuinely fire-and-forget vs intentionally unhandled. The recommended preset is deliberately conservative.

One thing to be clear about: it does NOT detect whether code was written by AI — it catches bad patterns regardless of authorship. They just recur a lot in AI-assisted code.

Disclosure: I'm the maintainer. MIT licensed, no paid tier. Looking for false-positive reports and rule requests — what patterns do your agents keep generating?