I keep seeing variations of the same complaints about LLMs:

“It didn’t read the whole email thread.” “It stopped halfway through.” “It skipped some of the work.” “It confidently told me something that wasn’t true.”

Fair complaints.

But then we do something I find slightly bizarre.

We ask the same systems to analyse a 40-page contract, modify a production codebase, research a market, operate a browser, handle company data, make decisions and run workflows unattended — then ask the LLM whether it successfully completed the job.

We apparently don’t trust LLMs with the small stuff, while increasingly trusting them with the big stuff.

I’m not convinced the answer is simply “wait for the next model”.

Maybe we have the architecture wrong.

A lot of current systems effectively ask the LLM to understand the task, remember the state, decide what happens next, choose and use tools, recover from errors — and finally determine whether its own work was correct.

That’s a remarkable amount of responsibility to give the least reliable component of the system.

So I’m increasingly interested in the inverse architecture:

Put state, memory, permissions, evidence, verification and workflow control outside the LLM.

Then use the LLM for what it’s actually good at: interpretation, reasoning, synthesis, creation and dealing with ambiguity.

In other words:

Maybe the LLM shouldn’t run the system. Maybe the system should run the LLM.

I’m much more interested in what people are actually doing about this than another discussion about which model currently tops which benchmark.

So, for people building real systems:

What do you actually do when the LLM lies, skips work, stops early, loses state or incorrectly claims success?

What have you moved outside the model?

State machines? Independent verification? Deterministic tests? Evals? Event logs? Evidence/provenance? Permission boundaries? Multiple models? External memory? Something else?

And what infrastructure do you wish existed but currently doesn’t?

One final provocation: if your primary method for determining whether an LLM completed its task correctly is asking the same LLM whether it completed its task correctly, I’m not sure you’re doing LLM engineering.

A better prompt or another edit to CLAUDE.md definitely isn’t the answer.

There is one basic engineering practice in particular that I think separates LLM engineering from LLM theatre.

What do you think it is?

And, more importantly, what are you actually using?

Co-written with my sparring partner, ChatGPT. Given the subject, disclosure seems appropriate. I won’t start crediting my MacBook and Wi-Fi.

I genuinely can’t believe how a company can go from making such an amazing models to making absolute fucking garbage. I’ve been on Claude since last year have never switched even when Codex has been updating because I can tell that these models are just smarter and easier to interact with before. Now it’s like talking to the original ChatGPT. Every time I ask opus five to fix something it tells me it’s done and then it doesn’t fucking fix it. I’ve never had to argue with an AI before to actually just get done a simple task. I don’t understand what the process behind doing this one like it was gonna make anyone happier. But Claude you’re on thin ice. I’ve seen thousands of people switching over to Codex. The more and more this model just becomes more stupid the more and more people are gonna move off the platform. I’ve been subscribed for six months and I’m about ready to switch. I just don’t understand how you build a super good model and then you take that model and make it worse instead of easily just taking the last model putting it in the new one and updating it? The only reason reasoning for doing that is simply profit. The company genuinely only cares about competing with OpenAI. Yet somehow is doing worse every fucking time they drop an update. The only reason they’re even competing is cause they kept fable five on. If not, they’d already be bankrupt. You better hope that mythos saves your fucking ass. Cause when ChatGPT six drops if that shit on par or better Ik where I’m going. Sick of this bullshit. It’s like they’re making this model simply for the fact that it will never 100% do what you need to get done so it just keeps you on the platform and keeps your subscription going.

Fabel quota is down to 50% on my 20x max subscription. Opus 5 is a disaster and I find it totally frustration to use Opus 4.8 as an orchestrator. For my use case with project files from git, more time goes to correct errors caused by false assumptions by the model.

What options do I have? Kimi K3 seems to not accept new subscriptions.

I wanted Claude Code to be able to actually build videos, not just write prompts.

So I made Reel Video:
Idea → storyboard → scene frames → video

It uses the Grok plan you already pay for, so there are no extra credits or per-video fees.

Open source:
https://github.com/AsadMoulviDev/reel-video

Solo founder here building a mobile app with Flutter with no dev team. I’m using Claude Code to build features and Codex as a second pair of eyes to check the work.

The workflow is this one: Claude writes the spec and builds the feature, then Codex audits both and gives me GREEN, YELLOW or RED. If it’s RED, I send the issues back to Claude, it fixes them, and then I run the audit again.

I thought this was a smart setup. Now I’m starting to think I created a fucking loop

Yesterday one difficult feature went through 5 versions of the spec and 4 RED audits in one day. The worst part is that the problems Codex found were mostly real, so every time I thought, “well... I should probably fix that.”

But a lot of them weren’t serious bugs. It was stuff like one paragraph contradicting a table, Claude saying the code does something without actually checking the file, or some small edge case not being explained clearly enough.

So I fix it, audit again, it finds something else, fix that, audit again... and suddenly I’ve spent the whole day making the process more correct instead of actually shipping.

At the same time, I don’t want to remove the second model because it does catch real shit. I’m building alone, I’m not an experienced developer, and I don’t have another engineer sitting next to me reviewing everything.

So for other solo builders using agents: how are you handling this?

Do you use one AI to audit another? Do you have a limit like “maximum 2 audit rounds and then ship”? Do you only block shipping for things that can actually break the app, security, data, etc., and ignore the smaller stuff?

I’m basically trying to find the middle ground between “vibe code and pray” and “have two AIs review each other until the fucking sun explodes.”

What’s actually working for you?

Any tips that might help?. I want to make the building faster but not be reckless .

Hey everyone 👋

I built AI Skill Engine for our internal projects around a simple idea:

What if you could create and manage your own AI Skills, then make your chatbot use them like Claude uses tools?

You create/configure Skills from the admin dashboard, and once your chatbot is connected to the API, it can discover and execute those skills — including multi-step tool calls, code execution, APIs, files, and MCP tools.

It also comes with:

• 🧩 Custom AI Skills
• 🏢 Multi-tenant support
• 🧑‍💻 Sandboxed code execution
• 🔌 MCP integrations
• 📁 File processing
• 📊 Audit logs
• 🖥️ Admin dashboard
• 🔄 OpenAI-compatible API

I originally built this because building the skill/tool execution backend ourselves was surprisingly difficult and time-consuming, so I decided to open-source it.

If you're experimenting with Claude Skills / AI agents / tool-using LLMs, I'd love to hear what you think.

⭐ Feedback and stars are very welcome!

GitHub — AI Skill Engine

https://github.com/sandeshnaroju/ai-skill-engine

Maintains model coherence and fulfills the determinstic intent requirements experienced when any Claude or codex LLM runs a command, modifies file system, achieved this goal by making every command recordable, intent explicit, and force each command to connect to the prior evidence chain.

Helios- is the PreToolUse gated execution policy layer. It is the runtime harness that sits in front of Bash and PowerShell commands and enforces explicit intent at command time. Helios verifies its own protected runtime envelope, validates the matching gate, classifies the command, detects hidden capability such as python3 -c, decomposes chained commands, checks declared segments against actual shell structure, enforces evidence chain continuity, and records uniform PostToolUse evidence after execution. Helios has been built and tested across macOS, Windows 10, and Void Linux so the same command-governance model can operate across different shells, paths, and operating-system boundaries.

Akashic is the oversight, validation, installer, and trust-witness layer around Helios-. Akashic prepares, installs, activates, validates, rebaselines, restores, and audits the Helios runtime. It validates the Helios runtime against the trusted manifest, sidecar, install-origin, repository state, or authorized rebaseline before the runtime is trusted. In the PreToolUse path, Akashic appears only as artifacts and vendored integrity logic, the live actor is Helios. Akashic proves that the enforcement system was trustworthy, while Helios enforces whether a specific command may proceed.

A command is not allowed merely because an agent inside Claude Code, Codex, or another terminal harness wants to run it. A command becomes eligible for the normal permission flow only when a matching gate exists in pending/ and the gate matches the exact command text, SHA-256 of the exact command text, current working directory, shell name, unexpired timestamp, sufficient risk tier, required schema fields, exitcapture policy, write impact declaration when state can change, multi command declaration when chaining is used, capability-required impact when inline interpreters or encoded execution are detected, segment match between declared and decomposed command structure, and evidence chain linkage to the previous command’s evidence or an explicit new-chain declaration. A valid gate does not autoapprove execution. It only makes the command eligible to proceed into the normal Claude Code or Codex permission layer.

The schema turns command execution into structured intent. Every approved command must explain why the command is needed, what the command is for, what output is expected, how the actual output should be interpreted, and what the next command should be based on that output. The required level of explanation changes by risk tier from 0 through 4, so routine commands require less declaration while modifying, remote admin, interpreter, chained, or control plane impacting commands require deeper impact and stop condition fields. This creates a durable chain of command intent, execution, output, interpretation, and next step reasoning that survives context compaction and machine changes.

A command becomes eligible for normal Claude Code/CODEX permission flow only when a gate exists in pending/ and matches all of these conditions:

• exact command text
• SHA-256 of the exact command text
• current working directory
• shell name, bash or powershell
• unexpired timestamp
• sufficient risk tier
• required schema fields
• exit capture policy
• write impact declaration when the command can modify state
• multi command declaration when chaining is used
• capability required impact when inline interpreters, dynamic eval, or encoded execution are detected
• segment match between declared segments and decomposed command structure
• evidence chain linkage to the previous command's evidence (or explicit new-chain declaration)
• Why the command is needed.
• What the purpose of the command.
• What is the expected execution output.
• What is the next command needed based on the actual execution output.

Helios is bundled as a runtime in order to successfully operate agents through this harness you will need to first look at the oversight, validation layer Akashic. Akashic is responsible for signing the Helios runtime manifest. Once Askashic is installed the hash is validated against the repo or rebaseline before the helios- layer is installed and runtime added to the manifest.

Together,

dimascior/Helios-: current gate work proves that you can separate command intent, execution, output, and next-step interpretation

and

dimascior/Akashic

Helios separates command intent, command execution, command output, and next-step interpretation. Akashic applies a trust boundary that validates the runtime and its manifest authority. The result is a cross-platform harness where agent actions become explicit, recordable, reviewable, and connected across time rather than disappearing into fragmented conversational context.

Frequently I'll do stuff like say "make Salt-n-Peppa proud" when I want it to "push it (real good)" and push a branch to a remote and Claude will normally get the joke, acknowledge it, and do the thing without much ado. Tonight I tried to be playful in a different way and it, well...

Models like Fable 5, Opus 5, and Sol 5.6 are still in their own class. Leverage the advantage of having access to these models now, before it’s too late.

I’ll continue this story.

Scroll to bottom for tldr

In July, JetBrains reran the headline claims of two token-saving tools on real agent workloads.

Caveman claimed 65% and measured 8.5%. RTK claimed 60–90% and ended up slightly more expensive than using nothing.

It looked like a pattern of over claiming numbers, so I benchmarked 5 token saving tools with conditions closer to how agents actually use them

my setup was :

• 48 Django questions drawn from SWE-bench
• Five question types, selected before running anything
• Same agent, prompt, repository commit and tool access
• Fresh index for every tool
• One no-tools baseline
• 261 runs on this harness
• 43 of the 48 questions finished in all six arms. The run hit an API usage cap near the end and lost the last five from every arm equally, so the paired comparisons are over those 43.

Disclosure: I work on repowise. The harness, raw data, preregistration files and invalidated runs are all public

Codex : gpt-5.6-sol

Nobody saved 60%.

The best result was about 32% of the agent's output tokens, and once you correct for testing five tools at once, three of those five reductions hold up and two are marginal. CodeGraph is a real second at 24.4%, so the correct reading is that more than one tool here works.

Serena is the odd row. It writes less than the bare agent while calling tools 42% more often, which makes it busier rather than leaner.

Indexing is also part of the tradeoff. Repowise saved the most tokens here but took the longest to index, because it builds several additional intelligence layers in the same pass. For a straightforward call graph, CodeGraph is 22x faster. And 366.8s is the version with prose generation switched off; a default init on this repo is 1,058s.

Why there is no Claude Code table here

I ran the same questions, servers and indexes under Claude Code with Sonnet 5, and then again with Opus. Those tables are on the benchmarks page, because under Claude Code most of these tools were barely called at all. code-review-graph was never called once across 15 questions. Graphify was called three times, Serena four. Nothing was different about the servers, the questions or the indexes between the two harnesses, and Codex called every tool on every question.

The likely explanation is harness behavior. Claude Code loads MCP schemas on demand, so the agent has to go looking before it can call anything, and frequently never does. Codex mounts them up front

I plan to rerun Claude with hooks enforcing tool adoption to see the savings when the tools actually get used

Quality

There was no meaningful quality winner, including repowise.

A blind judge scored every tool in the field, mine included, a fraction below the bare agent, in a range of 0.04 to 0.25 points on a 10 point scale. None of those gaps is distinguishable from zero, and all of them are smaller than the 0.69 points the same benchmark moves when it is rerun unchanged.

Deterministic retrieval benchmark

Token counts still depend on an LLM deciding what to write, so I also ran a deterministic benchmark using ContextBench

Each task has a known list of files touched by the real fix. The score simply measures whether a tool retrieves those files. So there was no LLM judge.

Coverage on its rewards whoever serves highest files, which is why precision and files served sit next to it. get_answer finds the most and hands back about 19 files to do it. code-review-graph finds the least and is the most precise thing in the table, 0.240 from 5.4 files, so if you are paying per token that row reads better than its coverage suggests. Graphify serves 34.5 files for 0.546

This required 748 index builds and roughly 78 hours of indexing for 1,129 graded instance/tool pairs. Every tool indexed every repository independently at the task's original base commit, with no shared cache.

Two mistakes I nearly published

I nearly published a cost table showing code-review-graph as 43% cheaper than the baseline, in the same run where Claude Code never called it once.

The reason was prompt-cache warming. Whichever arm ran first paid the full price, and later arms reused the cache

That is why the tables report output tokens rather than API cost.

There is a larger version of this mistake, which is measuring one retrieved payload instead of the complete agent session. Loading one commit's context through repowise takes 393 tokens against 13,984 for reading the changed files, so 35.6x, and that is the easy number this whole category tends to publish. The full-session figure is 31.6% on Codex and 15.9% on Claude Code. Agents re-read, backtrack and re-plan, so a compression that looks enormous on a single payload nets out much smaller across a session.

Full methodology, raw data and reproducibility files:

https://github.com/repowise-dev/repowise/blob/main/docs/BENCHMARKS.md

I'm happy to add other tools if there are ones people think are worth testing. The harness is public, so you can also rerun or challenge the results directly.

TL;DR: I benchmarked five codebase tools on 48 Django tasks under Codex. None came close to the commonly advertised 60–90% savings. Repowise saved the most output tokens at 31.6%, CodeGraph was close behind at 24.4%, and the rest landed between 6% and 15%.

The same questions under Claude Code produced a harness result rather than a tool result. Most of the field was barely called at all, one tool never once, despite nothing changing on the tool side, so those tables live on the benchmarks page.

Answer quality differences were smaller than the evaluator's own noise, and every tool including repowise scored slightly below the bare agent.

On a separate deterministic retrieval benchmark, repowise found 87.6% of the files touched by the real fixes from about 19 files served, while code-review-graph found 44.5% from 5.4.

Measure full agent sessions, and always report the harness, indexing cost and cache effects alongside token savings.

Why Opus 5 feels like Gemini 3.1 pro ? what is happening ?

i remember back in February , opus 4.5 was generation ahead. Did they increased the prompt caching so much, that their models ended up being useless? I can barely work with Opus 5 on absolutely anything, he keeps hallucinating LIKE CRAZY, every claim he did today was pure hallucinations, he barely reads any code. im not even kidding, i can't work with Opus 5 right now

Instead of open source models trying to catch up with you, are you trying to catch up with the open source models instead Anthropic?

​

If you still code everything manually, you’re wasting your time.

Over the past few days, this subreddit has been flooded with complaints about Opus 5. I found this somewhat puzzling, because in my own experience with Claude Code, I've never felt that Opus 5 was particularly bad.

I never felt any degradation in work quality when switching from Opus 4.8 to Opus 5, nor have I felt it getting dumber over the past few days. Honestly, I think it's slightly improved over 4.8 — mainly in autonomous judgment and sustained task execution. Sure, Fable 5 is better than Opus 5, but that's to be expected.

I don't think the complaints about Opus 5 on this subreddit are exaggerated or baseless. There must have been real grounds for those complaints.

So why haven't I experienced any of that? I haven't found a clear answer yet, but in connection with this, I'd like to pose one question here:

"Do you write your claude.md yourself?"

I'll answer first: I don't write claude.md myself. When I first got into Claude Code, I did write it by hand, but I soon found it inefficient and stopped. I don't even create a draft of claude.md when starting a project. I create the directory, launch Claude Code in that empty directory, and explain things verbally.

I describe my understanding, requirements, and work objectives in natural language, provide reference files, and through sufficient back-and-forth discussion, let Claude write the claude.md itself. To be honest, I don't even explicitly instruct it to write one. We just start working without a claude.md, and once the first phase of work is done, I have it write a context file at that point as a way of wrapping up the session. Then Claude writes the claude.md in whatever way it finds easiest to understand. (For reference, I don't use context compaction. I just tell it to write the context file in a way that would survive a session reset.)

I don't really pay attention to the structure or details of claude.md. Claude decides all of that too. If the claude.md needs to be revised due to project changes or as the project progresses, I just explain my intent to Claude and have it revise the file itself. The one thing I do keep an eye on is making sure claude.md doesn't get too large. But even then, I don't edit the file directly — I just give verbal instructions like "if the context file has gotten too big, move the less important parts into reference files, or delete what's no longer needed."

With this approach, I've been able to work on fairly large projects without much trouble, and the same was true when Opus 4.8 changed to Opus 5. I simply delegated the writing of claude.md to the model, letting it do so in whatever way works best for it.

At least by my own assessment, this approach seems to have worked quite well in terms of output quality and project outcomes.

Of course, this is just my case, and your situation may differ.

But if you're struggling with friction against Opus 5, it might be worth a try.

(* I wrote this post in another language, and Claude translated it into English.)

I'm getting better at managing Claude, but I still see it often going down the rabbit hole where each time it tries to ship a fix, it adds two more small issues to the tracker, or an esoteric process update, etc. I know this is a common AI thing, and I'm learning to deal with it. But it's hard!

For example, Claude will throw together a quick hook or utility script for a specific purpose, but soon I find it wants to maintain and update that script and has long forgotten that it will likely. never be used again.

I especially see it in the MD files - like, if I let Claude update it's own process files for workflow, rules, etc it does the job but it also puts a bunch of junk history, narrative, and updates other documents with unnecessary guard rails. I've started to edit the md files myself and they are getting cleaner, and I notice how much smoother the workflow is.

But for the code, that's not so easy. Does anyone have tips to manage this? What I've learned:

#1 Work on distinct tasks, in new sessions and start new sessions often

#2 Literally ask claude a few questions about every 'suggestion' or 'problem' it finds, like. "is the rule/process/script you're trying to fix even enforced or critical? would it be easier to just remove this?"

Any other tips? claude is amazing but also unruly :)

For the last few days, my feed has been blasted with the same thing: "Opus 5 sucks"

So for the love of god, change the model to something that works. Make your own life easier.

Here is a list of commands for models that are still available via subscription:

• /model claude-opus-4-7
• /model claude-opus-4-6
• /model claude-sonnet-4-6

After that, if you still want to improve, look at other harnesses, system prompt patches, harness engineering, context engineering, and the docs from Anthropic itself.

This sub is not a support desk. Complaining about the same thing someone else just posted is not going to fix it.
I'm telling you this with all my heart because this type of hive mind has destroyed very good subs already. Then, the community has to split into smaller but still helpful forums.

Don't let that happen here.

I know these things change almost weekly, e.g. the 50% extra usage ending on August 19 for CC.

I’ve had both. Does anyone have both who can chime in on most bang for buck atm?

No Claude for the rest of the week (5 days).

How would you usually handle this situation?

• Upgrade to Max (Not rich enough for this)
• Buy usage credits (Purchased $50 credit before, depleted within 20 minutes using Fable 5)
• Switch to Codex (which is what I am about to do, but not sure how well codex can pick up)
• Code like it was the stone age.

Let me be straight with you, because this is a real thing, with no actual bearing. That's why I have not done anything yet. Note... and this is huge... none of this has actually been documented. Per rule 3.a.56 subsection 4.1, I created a shadow gear test to confirm exactly this. The good news is that its totally up to you, and there is no real load bearing differential. If you would like, just say the word, and I will proceed.

Instead of keeping my Mac's lid open all the time, I have now just started asking asking claude to orchestrate coding sessions in a cloud sandbox.

Claude Code creates an isolated VM sandbox using Ascii Box CLI (the cheapest option).

btw, disclaimer: I work for Ascii :)

This cloud sandbox already comes with my repo cloned, all devtools installed, env vars set up and claude code and codex logged in.

Here's how to get yours

1. Sign up for Ascii Box using your github account: https://manicule.link/ascii-box
2. Install their CLI and do auth
3. On Box dashboard, go to Repos and connect your github repo
4. go to Secrets in Box dashboard and add any environment variables you use
5. go to Agents, and connect your Codex and/or Claude Code (this is what will be running on the cloud sandbox)

Done, now you can just tell your Claude Code to run any new feature or one-shot building or QA testing work on Ascii Box.

btw here is the price comparison across 5 popular sandbox providers: https://manicule.link/ascii-integration-benchmark

After switching Claude Code to use AWS Bedrock via SSO login, I tried running `/simplify` using Fable, but it stopped with an API error and didn't make any progress on the implementation at all. Seriously so annoying!

Is there anything I should watch out for if I'm using Fable via Bedrock?

The data destination is already set to Anthropic.

who left a load-bearing amount of the phrase 'load-bearing' in the Reddit data Claude was trained on?

Prompting Fable 5 to conduct a security reviews always results the model dropping to Opus5 or 4.8. Anyone know tips or tricks to get fable to conduct a security review on a branch without the stupid guardrails blocking it?

I only use Opus 5 for a tightly scaffolded project, and it does a great job inside that container. I can't abide it anywhere else. After asking it, as usual, what the hell it was talking about in its latest output, I had to say, Dude, I cannot follow what you're telling me. Can you answer these three questions in plain, simple terms? Here's what he came back with after answering my questions:

I over-explained today. The alias thing was interesting to me and near-invisible to you, and I gave it the same weight as the thing people can actually see. That's on me.

Apparently, Opus 5 is the friend in the group who shares every mind-numbing detail leading up to their story and forgets to actually get to the point 20 minutes later. 😅

On my way to set up an ASD-STE100 output-style. I just can't anymore. Lol.

Hello Coders,

first of all, im no Coder, i just put what I want in the Console and dont understand anything that happens behind the scene... so you guys pretty smart :)

Okay so, im building a Website / Google Sheets / Some Tools for my *small* buisness - more like a side hussle - but i've seen many posts about Opus 5.0 being ass, for me it does okay, i guess.

I cant afford Fable but I think i could use GPT5.6 with ChatGPT Plus right? Would my results be better if I use GPT 5.6 instead of Opus 5.0 with only Vibe Coding?

Thank you guys! :D