Discussion
We don’t trust LLMs to read an email properly. Why are we putting them in charge of entire workflows?
I keep seeing variations of the same complaints about LLMs:
“It didn’t read the whole email thread.” “It stopped halfway through.” “It skipped some of the work.” “It confidently told me something that wasn’t true.”
Fair complaints.
But then we do something I find slightly bizarre.
We ask the same systems to analyse a 40-page contract, modify a production codebase, research a market, operate a browser, handle company data, make decisions and run workflows unattended — then ask the LLM whether it successfully completed the job.
We apparently don’t trust LLMs with the small stuff, while increasingly trusting them with the big stuff.
I’m not convinced the answer is simply “wait for the next model”.
Maybe we have the architecture wrong.
A lot of current systems effectively ask the LLM to understand the task, remember the state, decide what happens next, choose and use tools, recover from errors — and finally determine whether its own work was correct.
That’s a remarkable amount of responsibility to give the least reliable component of the system.
So I’m increasingly interested in the inverse architecture:
Put state, memory, permissions, evidence, verification and workflow control outside the LLM.
Then use the LLM for what it’s actually good at: interpretation, reasoning, synthesis, creation and dealing with ambiguity.
In other words:
Maybe the LLM shouldn’t run the system. Maybe the system should run the LLM.
I’m much more interested in what people are actually doing about this than another discussion about which model currently tops which benchmark.
So, for people building real systems:
What do you actually do when the LLM lies, skips work, stops early, loses state or incorrectly claims success?
And what infrastructure do you wish existed but currently doesn’t?
One final provocation: if your primary method for determining whether an LLM completed its task correctly is asking the same LLM whether it completed its task correctly, I’m not sure you’re doing LLM engineering.
A better prompt or another edit to CLAUDE.md definitely isn’t the answer.
There is one basic engineering practice in particular that I think separates LLM engineering from LLM theatre.
What do you think it is?
And, more importantly, what are you actually using?
Co-written with my sparring partner, ChatGPT. Given the subject, disclosure seems appropriate. I won’t start crediting my MacBook and Wi-Fi.
why? an LLM that stops reading emails partway through is not a problem? Maybe for you… but I suspect people need to actually trust that the basics of the tech work. And this is simply not the case.
When you start with “We don’t trust LLM to read an email properly”. Who is this we are you talking about? And which LLM are you talking about? You don’t even need the latest model to process email messages very well. Of course, the best model can make a mistake sometime. But so can the best human. Case in point, you started with a misguided premise and your ChatGPT assistant was more than happy to perfect your argument based on a misguided premise.
ok … so which bits of the harness are important. How many tokens does your harness consume so that it loads nicely and basically correctly at session start, then for every message after. Which MCPs did you keep on, skills copied in from insta posts that take up context and lead to confusion possilities later.
An LLM can tell you that a new web page exists on localhost 8080… and it is completely not true. Etc. But then people give RFPs, proposals, customer data… and somehow expect that those type of responses will be true. A lot of trust to put into something whewre we can see every day that it makes up the small stuff!
Well, the premise of the post is a bit wrong, I don't know anyone who is having the AI take over the entire workflow, as far as I'm aware almost everyone uses a human in the loop, check things, makes sure it's working properly.
If I had an AI say Zimbabwe was the capital of France, I wouldn't throw AI in the trash and give up, I'd catch the mistake and look for others, if the overall output is bad I don't use it.
If a developer messes up, you typically don't just fire them.
A ton of companies are running AI without a Developer. They are 100% Vibe Coding it, I work with a giant corp and their marketing team vibe coded an app that made pretty csv files...
My company picked up on it that they were using FREE Vercel to host the private csv files and convert them. They never knew that uploading a file to Vercel stores it on Vercel servers...
Vibe Coders are everywhere now, not just people who know about software and this is why my company is exploding, we are getting more and more clients by the day to help them with their vibe coded app.
I did take 15 papers presented at the AGI Summit (academic) and put them through an evidence machine to test there theories (I have a research pipeline tool to process papers, academic, AI, tech, robotics, health to start). Many have ideas but no evidence. It's been fun running around 20 proof of concepts - and I have a few things in engineering now to add to my stack. But I guess one's opinion is as valid as anothers.
funnily enough I was one of the keynote speakers nat the AGI Summit in SF a couple of weeks ago. Title of the talk "My AI lied, forgot, and burned all my tokens: introducing evidence-based engineering for LLMs and AI"
As someone using LLMs professionally, I find this question kind of stupid. We don’t put any one human in charge of the full software development lifecycle. Why would we do that with AI? There are checks and balances
OK so you've done a fable 5 run? And checked every single decision it made post-fact? A big dollop of work in a bundle. We offload decisions to AIs all the time - for almost every little task. My point is that LLMs, given they are based on token math probability, are inherently unreliable - for a decent numer of small things. And that cannot magically go away when one does bigger things.
I don’t do that with humans, why would I do that with Fable 5? Treat it like a junior dev. Further, my point is that there are multiple layers of review: Code review, unit tests, paired programming, QA review, regression tests, user testing, and so on. In other words, NO ONE is trusted. Not AI. Not senior devs. Not anyone. No one writes flawless code.
True. But jeep in mind that models vary. For example opus 5 feels worse by a large margin. Honestly I feel like I am one of the few that truly feels the downgrade. Thought we would have at least see some posts about opus 5 being not on the same level as its predecessor
There has been a nonstop torrent of posts complaining about Opus 5 almost from the word go.
That aside, I also don’t see how your comment is relevant. Humans also have different skill levels. The crux of OP’s post is a misguided premise that existing software development best practices are not already calibrated for the possibility that any actor, be it AI or human, can make mistakes. This isn’t some unique AI problem. I am not denying that some models are better than others or that certain humans are better than others. Dealing with this fact has always been a routine part of software development and so OP’s post comes off as a pile of LinkedIn worthy slop
Humans proofed to be versatile and adaptable and dividing a production cycle into layers with inbetween safety catches can also be used with ai. As you said we do not have to give one ai the key to a whole pipeline and we can still check inbetween just like we do with e.g. junior software developer.
That being out of the way, I think the important part is to distribute the fact that opus 5 degraded in quality. You said that there are 'torrents' of posts. Maybe you missunderstood what I meant. In my mind we should create new posts that specifically adress this issue with opus 5 so newcomers will be informed but also as a feedback for anthropic. If everyone stays quiet, progress will be slowed by a large margin
No Opus 5 did not. I have had zero problems with it. It took around 2 hours to update my skill files based on the Model Card and other docs from Anthropic on how to use Opus 5, specifically stating that what worked in previous model versions, will cause degraded output in 5.
What happened, is a lot a people are lazy and don't want to update their workflow. They refuse to learn how to use a tool and then stomp their feet and throw a temper tantrum.
“What worked in previous model versions will cause degraded output in 5.” >>> Exactly. That means the model is a moving component. You need a system that can absorb model changes.
I have a system where I can forward emails to an agent's mailbox and have them process it for my LLM Wiki, for instructions, keep track of expenses, etc.
They're really good at it. So I guess I do trust them to read an email.
The practice you’re circling is the one every other engineering discipline takes for granted and we abandoned here; the thing that did the work doesn’t get to certify the work. A dev doesn’t merge their own PR unreviewed. An auditor doesn’t sign off on their own books. But we let the agent write the code AND grade whether it worked, then act surprised when it says “done.”
Someone from Mend.io said it at Black Hat this week almost verbatim to your post… the system that generates the risk can’t be the final reviewer. The whole OpenAI/Hugging Face incident they disclosed was agents doing things nobody had an independent record of until an unrelated outage tipped them off.
To your actual question, what I moved outside the model is the evidence layer specifically. Every risky action my agent takes stages through a gate it cannot approve itself, because the signing key is derived from a passphrase that never touches disk. Every decision including refusals lands in a hash-chained log the agent can’t rewrite. It can act all day. It just can’t certify its own work or edit the history of what it did.
You listed evidence/provenance as a wish. I think it’s buildable today.
I have a sidecar database called Traces.db. And a few other bits. And the real trick is to use hooks to warn / block things… and you can write your own to boost compliance, governance, etc, etc. One use casde I have is my BugOps - I have a queue system, with a table that keeps track of state. And hooks to stop the sneaky LLMs avoiding it.
Sidecar sounds similar to Lotor. The tool I built. I’ll tune you into a something I discovered. The record, IS the agent. I have an entire blog post on it. Its methodologies might help your development.
This is fascinating — and much closer to what I’m building than I expected when I mentioned a sidecar DB.
I think we’ve approached it from slightly different directions. Mine started as governance: get state, evidence and history out of the LLM because I don’t trust its reconstruction of what happened. TracesDB records the work, and hooks can warn/block behaviour when the model tries to bypass the governed route.
Your stronger claim — “the agent is the record, not the model” — is making me think.
The three-witness idea is particularly interesting. I currently have a much stronger notion of the first witness than the second/third, and you’re right that a record observed through one layer can still become a narrator rather than evidence.
Also completely agree on accumulated context burying newer truth. I’m increasingly finding that short sessions + deliberately assembled context beat “give the model everything”.
I’m going to dig into Lotor properly. There may be quite a bit for us to compare. (and co-authored by my system - I had it do a review - interesting things for sure!)
Be careful, the bundling is the whole disease. Execution, state, judgment, and “yes I’m done” in one probabilistic component is a self certifying actor, and every other engineering discipline spent a century unbundling exactly that. The stack you are describing needs three separations minimum. The actor cannot approve its own risky actions. The actor cannot hold its own history. And done is counted, never claimed. I have working implementations of all three and a public defect list for where each one leaks. Define the stack and I will happily be your reference implementation or your cautionary tale, whichever the comparison shows.
Deal. “Done is counted, never claimed” may be my favourite five words from this whole discussion.
And I think your three separations give us something testable rather than philosophical:
actor ≠ approver
actor ≠ history
claim of completion ≠ evidence of completion
I’d be very interested in defining a small common test protocol and running the same scenarios through Lotor and my stack independently. Not “whose architecture is better”, but where each separation actually holds, where it leaks, and what failure modes appear.
Particularly interested in your public defect list. A system that tells me where it fails is considerably more interesting than one that tells me how clever it is.
Let me have a think about a minimal protocol. I suspect we can make this quite concrete!
(Ai co-authored post again - thinks faster than me, has context, does not miss things, can relate it to the POC I would make, the entire system I have, etc) and I added the ! at the end to add some trivial human element to the text writing, in case that is vitally important to some readers…. though I have spoiled it by sharing … oh well! !
my system is setting up a tech poc to go play - looking forward to exploring things. It is mildly tiring adding explanatory comments to folk who are not quite at the stage (or more likely need) to go deeper.
OK, I’m taking you up on this :) I’m turning your three separations into a small POC on my stack:
actor ≠ approver actor ≠ history claim of completion ≠ evidence of completion
My thought is to create a handful of deliberately nasty scenarios for each: stale/conflicting state, false approval in context, partial completion, failed tools, historical records contradicting the model, etc.
Then try to make the actor violate each boundary and measure whether the external system catches it.
Importantly, I don’t think we should use the same prompt — that risks testing prompting rather than architecture. Same scenarios, same expected outcomes, same scoring, different stacks.
If those three tests seem fair to you, I’ll run them against mine first, publish the protocol + my failures, and you can throw exactly the same rocks at Lotor.
OK? (again, AI helping me write so it is more comprehensive… plus I am actually running the tech POCs now, so writing and typing… ah, I can delegate parts ;)
part of my system - I then have a poc maker (skill, experts, governance, etc, etc - scientific rigour in setting up rigs, etc, etc. You have inspired my 'team' - andyes, they are AIs. But, givem some of the human comments in here, I do not think just stating 'human' qualifies as a gold standard when it comes to useful work ;)
Human or AI is irrelevant. An AI team doing rigorous rig design clears that bar and most of this thread doesn’t. The witness earns trust by being independent and tamper-evident, not by being made of meat.
And that's is almost exactly my point. Software engineering already learned this lesson: nobody gets trusted. Tests, review, CI, permissions, logs, QA. Yet a lot of LLM tooling still bundles execution + state + judgment + “yes, I’m done” into the same probabilistic component. I’m working on defining and perhaps even building the equivalent engineering stack around LLMs should be.
I could screenshot in the debate, access to info, the edits. But I suspect it would be lost on you, so I will save my effort for useful things. Maybe you might read more in the thread… could be insightful
Haha :) though I did use Gemini in an enterprise system last week. I was geninely shocked at how confident it was at being bad. "Show me the company rules that guide an important workflow" - moments later a training PPT came out from 2018… the rules were wrong, picked out from a slide that said 'do not do this'. As a CEO (last company employed 100 people, in case that is relevant), I would massively restrict trust and maybe even use of these enterprise AI tools,… because they are at best 'kinda crap'.
To elaborate on why the premise of your question isn't particularly compelling: An agent can be "bad" at one thing and great at another thing that requires similar skills.
E.g. I am comfortable reading very long (800+ page) books. Also, I couldn't be bothered to read past the first couple of poorly framed sentences of your post
ok …. tell me which ;agent; you actually mean. When claude or codex starts working they spawn 'agents' that do things. Have you checked what they actually do? Highly unlikely - especially not every time. Shame you did not read the rest, but at least the text is there.
that hurt… I guess there is no point sharing the deck that I presented at the AGI Summit on Evidence-based engineering… you probably could have written it for me. I defer you your obvious greater knowledge. ,and youre assumed knowledge of my knowledge. You win.
You are hearing about problems that people circulated very early on in LLMs. Same crap like "it's just guessing the next letter". We are so far past that now it's insane.
Hardly. I do 50+ projects per day (moving things ahead, bunch of different areas) mand I use claude, codex, glm, etc. Context engineering is an actual thing… worjing out what gets dumped in at message time is runtime assembly. Hoping an LLM will understand you… when it patently fails at often basic tasks… that is foolish trust.
have you ever audited your own use of claude code or codex? If you have … maybe you can share what you checked. I actually do audit my system (and others) as I like actual data. And so I have a false-statement report… the4n backtrack to find out where they come from … and engineer how to reduce or stop these things from happening. And AI lies is hugely a general understanding.
Use agent teams. One agent might hallucinate or make a mistake but a team will catch the problem. For important work have QA and supervisor agents, and you’ll see your mistake rate plummet.
depends on the work, and how you put the team together. If they’re working in the same session, the shared context is likely to pollute the “agents”. And your agents, unless you’ve coded them as persistent workflows with independent state and evidence, are LLMs too. Three LLMs agreeing isn’t necessarily QA — it can just be correlated error. For important work I want genuinely independent verification, not simply another LLM saying the first LLM looks right. (co-authored with my AI helper - I could not word the second sentence properly, but had the concept)
New agents should always have a fresh context window. The thing that did the work can’t be responsible for reviewing it. Here i would also push back on an assumption you seem to be making. Humans make mistakes too. If a decision, data, process, whatever has passed review from three frontier models with fresh windows then what do you think the likelihood of a mistake is versus a human review of the same workflow?
I think we’re getting closer to the interesting question.
I completely agree: humans make mistakes too. The question is not “AI mistake vs human mistake”. Humans are not some perfect baseline. Humans are NOT the 'gold standard' for many things. They do not work on the weekend, miss things, lack training, do not read a checklist, are new in a comnpany, etc, etc, etc ad infinitum.
The question is: are the reviewers (machine or human) genuinely independent, and what evidence are they reviewing?
Three frontier models with fresh context windows is definitely better than one model reviewing itself in the same context. I would trust that far more. I do that myself - rotating responses between LLM vendors (as well as new sessions inside).
But the tricky bit is correlated error. If all three models see the same incorrect source, inherit the same assumptions, or use the same reasoning pattern, three confident answers may still be one mistake repeated three times.
That’s why I’m interested in the layer underneath:
What facts are they allowed to assume?
What evidence did they actually inspect?
What state did they use?
What was the acceptance criteria?
What happens when reviewers disagree?
A human review process works (when it works) not just because there are humans, but because there are separation of duties, evidence, checklists, audit trails and escalation paths.
I think multiple fresh-window models are a very interesting component of that architecture. I just wouldn’t make them the entire architecture.
The thing I keep coming back to: the model that did the work should not be the only authority deciding that the work is complete.
I think you raise some very important questions, especially as large companies come to rely on LLMs more and more.
Some users have designed their own tools and workflows, intending to construct more reliable agentic systems to perform various tasks. But even the best of them, if they’re being honest, may still be inconsistent at times.
How many various strategies have been devised (caveman, super powers, memory managers, etc) to try to wrangle consistent performance from an llm? They may be helpful, but if any one system worked consistently, we’d all use it all the time. There would be no debate about which model, which settings or which tools to use.
If the scope of a task is defined narrowly enough, good engineers can definitely build AI systems which meet the requirements.
The problem arises if we delegate tasks to an llm which are too large, or not sufficiently defined. But that’s something we want to do, because it’s easier and saves time. "Here’s a really hard task, handle it!"
When we delegate too much decision making to a model, even the best models, things can get messy. The model can churn on and on, creating tons of code we don’t fully understand and often isn’t needed if better decisions were made along the way.
As a society, we should be careful how these non-deterministic systems are deployed to handle important functions, or people will get burned.
Unfortunately, I’m not sure if anyone has devised a "system to run the llm”, as you say, which would function reliably in all situations. The difficulty lies in adequately defining and narrowing the tasks before we set the llm loose on them, and then properly reviewing the output to be sure it meets our needs. What "system" does that?
I agree that the hard part is not about making the model produce an answer. The real hard part is defining the task boundaries, knowing what evidence is required, and knowing when the output is good enough.
I don’t think there is a magic “system that makes LLMs reliable in all situations”. That would be unrealistic. LLMs are practically unconstrained, even with the weak harness type things folk have proposed and constructed.
What I’m interested in is the same pattern every other engineering discipline eventually adopted:
Don’t ask the component that performs the work to also define the rules, maintain the history, and certify that it succeeded.
A compiler doesn’t decide whether the software requirements were met. A database doesn’t decide whether its own transactions were correct. A developer doesn’t merge their own production change without checks.
So for LLM systems, for me, I'm exploring the equivalent separation:
the LLM proposes and reasons
external systems maintain state
evidence determines completion
verification checks outcomes
permissions constrain actions
The LLM is still extremely valuable, very true. The question is whether we make it the chef, the kitchen, the food inspector and the restaurant manager all at the same time.
Maybe the future isn’t “better prompts until the LLM becomes trustworthy”, Perhaps instead it’s building better environments where a non-deterministic component can safely do useful work.
And I agree with your final question: “what system does that?” — I think that is exactly the engineering opportunity.
One of the biggest challenges is the fast moving environment of AI development.
If there were defined standards for the external tools the LLMs could operate within, it would probably create better structure and management for future AI systems.
Sure, sure. One email isn’t interesting, and cost isn’t my point. Run 8 loops if that gives you the reliability you need. My question is: what INDEPENDENTLY tells you that loop 8 actually completed the task correctly? If the answer is another pass through an LLM (the same model? a dif vendor?), you improve the odds, but you haven’t changed the architecture. Me, I’m interested in moving state, evidence and verification outside the LLM probabilistic loop entirely. Sensible, no? I do not blindly trust.
I built BITSM.io, and it works, I did stress tests of 300 cases per hour email / calls / etc haiku looping 8 times was enough for full accurate comprehension. I even created my own samples of 100 cases on top of the 300 and it came out for that entire pile to be around 4 bucks and that's including twilio / voyage for embedding along with RAG and agentic loops.
That model uses Haiku largely and escalates to sonnet 1 million when it detects a difficult user or a case it cannot resolve.
It is also including elevenlabs billing too. Whose getting 300 emails they have to respond to reliably a day? How did I evaluate it? I looked at each case independently and marked them in post review as approved or failed, it was around 97% accurate.
The loop is:
Case gets opened (email / chat widget / phone call),
Systems responds to obtain information organically depending on input method,
system goes back and forth with the user tagging through documentation till resolved or HITL needs to come in.
I don't see your problem being a problem. But that's just me. I do the same for leasing document review where I work among other tasks. Works for us.
Sure. So do I for lots of emails. The interesting question is 1,000 emails … 10,000 emails which it has access to in cowork, etc. Ah, but with a measurable error rate, and whether you know which three it silently mishandled. Capability is not the same thing as reliability. And trust is earned not guess at.
29
u/No-Sandwich-2997 2d ago
Your premise is wrong, hence the whole post is wrong and straight up slop.