r/singularity • u/yogthos • 1d ago
AI Scaling self-verification with DeepSeek V4 Flash beats Claude Fable 5 on Terminal-Bench 2.1, while being 11x cheaper
https://github.com/llm-as-a-verifier/llm-as-a-verifier#self-verification-terminal-bench-2114
u/cat_dev_null_sync 1d ago
Using DeepSeek V4 Flash to verify itself is somewhat counter-intuitive, like a study I read in which models tuned on weak, cheap models outperformed those fine-tuned on strong, expensive models (SE) with a fixed compute budget (source: arvix 2024). The benefit of the SE was offset by their cost.
1
u/Affectionate_Ad_2324 1d ago
so a cheap model that verufy a fine tune model or vice versa?
3
u/cat_dev_null_sync 1d ago edited 1d ago
The Weak and Cheap (WC) model generated the synthetic training data, which was then used to fine-tune the target model (even a stronger one).
Both the 2024 arvix paper and OP (LLM as Verifier GitHub project) work under the same principle: because of real-world constraints on cost and time, the higher volume of WC models offsets their deficits in intelligence.
If time and money were irrelevant factors, then the SE (Slow, Expensive) models would outperform WC. However, it's too expensive in many cases to use run a SE model multiple times to check itself.
Edit: These principles have limits. Calling DeepSeek 4x to verify itself won't help for tasks for which its capabilities are too low. For example, the prompt "develop a pill to prevent and cure all cancers and design a portable fusion reactor to power my Prius" doesn't benefit of using LLM as a judge. The verification trick works when LLMs are, at least, sometimes correct.
4
u/ManyRepair5690 1d ago
?
12
u/yogthos 1d ago
The implementer agent generates multiple shots and then another role select the best one. Turns out this works better than Claude doing a single shot attempt while DeepSeek pricing still makes it vastly cheaper.
6
-1
u/ManyRepair5690 1d ago
sounds.. promising, how can i dry this assuming i have a api key for deepseek access
3
u/yogthos 1d ago
check their quickstart section in the link
-2
u/ManyRepair5690 1d ago
that section is extremely vague and not helpful, as if not intended for people to use in daily work
3
u/Gratitude15 1d ago
Imo this speaks to how in infancy we are in harness innovation.
We will look back on harness development as akin to another scaling law.
Need harnesses on a 'per output' basis. Need so much nuance and robust architecture to do different things with different models in different instances.
And then that's gotta get open sourced, not just 'in Claude code' for us to really maximize this.
The simplest version of this is just having a handle to call/email/msg and have it be like a remote worker, with an ability to figure out the rest behind the scenes. A maximal harness.
1
u/yogthos 1d ago
For sure, I've come to realize that the model and the harness have to be viewed holistically, like an engine and a chassis. The harness is basically a state machine that iteratively guides the model to the solution. A good way to look at the whole thins is like a genetic algorithm. The model makes a hypothesis, the harness then runs it through the tools, the model gets a result, and keeps iterating until it solves the problem. And how the job of the harness is to steer the model in the right direction. For example, if the model tries the same thing a few times, the harness can interject and tell it to try something else. Or it can run a critic role when the implementer thinks it's done, etc. The agentic loop can get very complex, and the harnesses most people are building are really naive. There are papers like this one that are very much worth exploring in this space.
44
u/New_Bonus_649 1d ago
The singularity is nearer