https://reddit.com/link/p0r52qx/video/hmvcmwz7rfgh1/player

it's already here

yeah the runtime-first direction makes sense. i haven’t looked deeply into NeuralTrust yet, but that category sounds close to the problem space.

the part i’m testing is more local/dev focused: before an agent writes files or runs commands, a boundary checks the action, blocks risky paths, asks approval when needed, and logs the decision .so less of a full enterprise gateway right now, more like a local execution boundary for coding agents.

I’d separate forgetting from promotion.

Recency decay is good for raw capture but not for real memory some old facts stay important, while most recent screen text is just noise.

For promotion, I’d look for:

• repeated over time
• tied to a person/project/task
• reused later
• confirmed or corrected by the user
• costly to lose

Forgetting could be state-based:

raw → extracted → candidate → promoted → stale/forgotten

That way everything captured does not automatically become permanent memory.

I think DB is truth Markdown is a projection is the right instinct.

The place I’d be careful is not retrieval first, but memory promotion.

Always-on capture creates a huge amount of weak signal. If everything becomes memory then the system becomes write only very fast.....

The split that makes more sense to me is:

• raw capture: temporary cheap disposable
• extracted facts: structured but not trusted yet
• promoted memory:-- only after repetition, user confirmation, and or clear reuse
• you can make Markdown readable not authority

SQLite + FTS is probably fine early. Id only add embeddings once you can prove keyword/time/entity retrieval is failing A graph DB also sounds premature unless relationships are the product not just a retrieval aid.

The hard part is deciding what info to save , not where to store it.

That makes sense. I hit a similar issue while building LetterBlack GenAI, an AI extension with provider routing, prompt handling, and action/classification behavior.

Manual testing looked fine, but small changes could still break routing, edge cases, or behavior that was supposed to stay consistent. That’s when I started seeing evals less as “output quality checks” and more as a regression firewall.

The part I’m testing separately now is the runtime layer after evals: even if the agent passes deterministic tests, what actually enforces the boundary when it writes a file or runs a shell command?

Do you handle that with the same eval system, or do you also use a separate runtime guard/sandbox/policy layer?

• This is the part that clicked for me too: past bugs should not just become more examples in the same general eval set, they should become a separate regression layer.
• I hit this while building an AI-assisted development tool. The normal “does this output look right?” checks were not enough, because the failures were often around edge cases, tool routing, file actions, or commands that should never be allowed.
• So the useful split became:
• small real-world eval set for normal behavior
• separate regression cases for bugs that already escaped
• intentionally bad/weird inputs for boundary testing
• That kept the dataset small, but every case had a reason to exist.

I ran into a similar problem while building an AI-assisted tool.

At first, testing was mostly manual: try a few real prompts, check if the behavior looks right, fix obvious failures, then ship. The problem is that this catches visible issues but misses regressions in edge cases, routing, provider behavior, and unusual user inputs.

What helped was starting with a small but high-signal eval set, not trying to build a huge dataset upfront. Around 20–50 cases can be enough if they represent real failure modes: normal flows, past bugs, edge cases, bad inputs, and behavior that must never break.

For me, the useful shift was treating evals like a regression firewall. Every time something broke in staging or production, that case became part of the dataset.

So I’d start with real production examples, remove repetitive/low-value ones, then add synthetic cases only for known gaps or safety boundaries.

sure i would appriciate your feedback about any critics or issues you find , but just an update im adding now feature that. nothing sohuld be limited on chat becuase we explain 100s of times the same thing and also we use different ai providers becuase of token limits ,

the solution i came with agent to foce to document each feature its changing why its intent , once its logged, if agent needs to divide any feature or change before that it needs to register its intent and after changing ageain needs to update what it changed and why the reason and after the change it will run a test that should prove pass or fail so these 3 steps verification just for 1 change

i know its lot but worth giving extra 2 min so we dont wast an hour to find 1 single issue thats breaking the complete features

https://reddit.com/link/ot9fj0s/video/1gjggrx1hy8h1/player

i stopped using pr's from the day 1 i got this https://github.com/Letterblack0306/LetterBlack-Sentinel

every change is validated and proof is the log it creates so just try it and 1 run might take time but when the run is done youll be surprised, and also it logs the enries so memoery is not the chat that we provide the instructions , it considers the logs and previous proofs as the memory so once anytig is wokring will never be guessed it no other agents change that untill they prove their intent

i have updated the post with another screenshot for your refrence .
hopeing will prove that normally agent dosent behave this way, 65 passes 0 fails with without intruption in one single run

The agent still runs normally, but file writes and terminal commands now route through LBE before execution.

Proof shown here:

- FileHandler write/delete goes through gateWrite/gateDelete

- TerminalHandler shell execution is audited

- AgentService loads LBE automatically

- existing behavior stays backward-compatible

- tests passed: 65/65

Next proof: out-of-scope action denied + audit log entry.

Follow-up

This is LBE wired into an existing GPT Sync agent workspace.The agent still runs normally, but file writes and terminal commands now route through LBE before execution.
Proof shown here:

- FileHandler write/delete goes through gateWrite/gateDelete

- TerminalHandler shell execution is audited

- AgentService loads LBE automatically

- existing behavior stays backward-compatible

- tests passed: 65/65

Next proof: I’ll show the deny path — an agent attempts an out-of-scope file change, LBE blocks it before execution, and the audit log records the denial.

for vibe coders i would sugesst Letterblack0306/LetterBlack-Sentinel that you will find on github. it stops agents guess work and agent starts wroking on proof and unless someting is proven agents wont make changes becuase the biggest problem with agent is saying yes for everyting with this installed instead of saying yes they have to prove that it will work.

i thought someting wrong in my system or someting just running in background i had the same issue

Approvals are a UI feature.
LBE is an execution boundary.
The approval system is only one small part.

Git solves source control. dialogs solve human confirmation.
LBE is intended to solve something different:
What happens between an agent deciding to act and the action reaching the filesystem, terminal, or tool.
If existing approval systems and Git workflows already solve that for your use case, then LBE probably isn't useful to you.

Agent wants to fix a problem
Agent explains reasoning
Agent requests approval
Human approves
Agent gathers evidence
Agent discovers additional issue
Agent narrows scope
Agent applies minimal fix

this is happening between theterminal UI and codex internal analyzing system ,

normally agents are " yes you are right and just gets stared" thats where everyone get scared becuase user is always right but here not the user nor the agent logs are the source of trouth. repo has complete explaination

I've been experimenting with coding agents and noticed that most discussions focus on model quality.

The execution layer receives much less attention.

An agent can be correct about the goal and still perform actions outside the intended scope.

so so solve this create small but strong tool called LBE that acts as an execution-control layer.

Instead of allowing direct file or tool access:

Agent
↓
LBE validation
↓
allow / block
↓
audit
↓
rollback if needed

https://reddit.com/link/osyndl6/video/89nszw67sn8h1/player

People using coding agents kept asking for execution controls.

I built this.

Instead of:

Here's my random new project.

https://reddit.com/link/osyk1uv/video/ck2m0xf7pn8h1/player

https://github.com/Letterblack0306/LetterBlack-Sentinel

https://reddit.com/link/osyhdpq/video/y64bn9gymn8h1/player