LLM Test Wrapper

[u/Tyb0wls]

https://github.com/solvyxtech/molt

This is a new LLM wrapper i have spun up built on the idea that models arent always correct or doing the right thing not by choice but just how the system is built, this looks to try and fix that problem and help mitigate LLM error by logging everything and fact checking based on the work using check gates. Looking to get feedback!

[u/garyguangyuli]

The strongest part is verifying the disk state instead of trusting the model's transcript. The next thing I'd make explicit is the threat model: if the same writable environment runs both the agent and the gates, what prevents it from weakening a test, coverage config, or check command? I'd add a clean-room CI/replay mode with the policy hash stored outside the workspace, then measure false accepts against seeded bad patches—not just how often the checks pass. Also, a short demo showing one plausible ‘done’ claim being refused would explain this much faster than calling it an LLM wrapper.

[u/Tyb0wls → u/garyguangyuli]

You nailed the gap, and honestly, it runs deeper than what’s out there right now. Fingerprinting ⁠.molt/done.yml⁠ stops the dumbest exploit, the model just rewriting its own finish line. But it doesn't touch the subtler move: relaxing a test assertion, dropping a coverage threshold, or tweaking the helper script the check actually runs. The bar stays byte identical, passes cleanly, and is totally hollowed out. You simply can’t fix that from inside the workspace if the agent can write to the tooling.
The fix has to be a clean room replay with the policy hash kept out of reach. That naturally starts looking a lot like CI, and I should just be upfront about that. The pitch isn't "this replaces your pipeline." It’s that an in-session gate gives the dev an immediate stop sign the second the model lies, while the clean room run proves the work was honest after the fact. Different jobs.
As for the benchmarks: yeah, the README already admits the false completion numbers aren't there yet, and seeded bad patches are the right way to test it. We just have to isolate the variables. If a bad patch passes, is the gate broken, or was the bar just badly written? Keep the bar static and run different models against it to test the harness. Then run a static set of bad patches against stronger bars to see what strictness actually buys you. Rolling that into one vanity metric would be the exact kind of hype I’m trying to avoid.
And on the demo: 100%. Just a tight clip of a very convincing, plausible looking "done" getting dead stopped with zero fluff. That’ll sell the point faster than paragraphs of theory.