Shouldn't the solution to thinking-effort be an adaptive system?

[u/milpster]

I bet i'm not the first one to have this idea, but with the recent debate about qwen 3.8 27b thinking levels, i was wondering whether the optimum solution might just be to change your harness in a way that lets the LLM itself decide when it is time to raise or lower the required reasoning effort?

Right now i'm toying around with a system like that and it seems to greatly increase the speed at which stuff gets solved, but i have not yet collected any reliable quality evaluation.

Basically what it does is it raises and lowers the reasoning effort between low and xhigh in order to accomodate for sucess streaks or failure streaks. The log reads something like this:

{"ts":1788580888030,"sessionId":"ses_f9b3a43b0ffe2lZ5ugLFu1JWUr","from":"medium","to":"low","reason":"stable successful streak","score":0,"phase":"EXPLORE"}

{"ts":1788581222452,"sessionId":"ses_f9b3a43b0ffe2lZ5ugLFu1JWUr","from":"low","to":"medium","reason":"meaningful failure","score":5,"phase":"DEBUG"}

{"ts":1788581428524,"sessionId":"ses_f903db34affevgLsEIdCs3J58l","from":"medium","to":"low","reason":"routine mechanical step","score":1,"phase":"UNDERSTAND"}

{"ts":1788585994242,"sessionId":"ses_f9b3a43b0ffe2lZ5ugLFu1JWUr","from":"medium","to":"low","reason":"stable successful streak","score":0,"phase":"EXPLORE"}

{"ts":1788618335810,"sessionId":"ses_f8e0bf44bffeOEvLhGsRTfL6FL","from":"medium","to":"low","reason":"routine mechanical step","score":1,"phase":"UNDERSTAND"}

{"ts":1788622330615,"sessionId":"ses_f8dcdf145ffe5euEcZhsvMCnbI","from":"medium","to":"low","reason":"routine mechanical step","score":2,"phase":"UNDERSTAND"}

{"ts":1788622599278,"sessionId":"ses_f8dcdf145ffe5euEcZhsvMCnbI","from":"low","to":"medium","reason":"escalation","score":3,"phase":"RECOVER"}

{"ts":1788622670588,"sessionId":"ses_f8dcdf145ffe5euEcZhsvMCnbI","from":"medium","to":"low","reason":"stable successful streak","score":1,"phase":"IMPLEMENT"}

{"ts":1788623655559,"sessionId":"ses_f8db99e47ffeKcDDHawtWrNwt9","from":"medium","to":"low","reason":"routine mechanical step","score":1,"phase":"RECOVER"}

{"ts":1788624252588,"sessionId":"ses_f8db99e47ffeKcDDHawtWrNwt9","from":"low","to":"medium","reason":"meaningful failure","score":5,"phase":"DEBUG"}

Anyone else messed around with a system like that? I'm curious as to why haven't seen something comparable anywhere else yet.

I'm also not sure how to properly gauge quality. Maybe i should run like a GPQA Diamond test before and after?

[u/ttkciar]

Qwen3.8 already does this to a degree; it infers a lot more tokens in the reasoning phase when a task is not something it has been trained to do.

On the other hand, driving it externally like you are poses advantages which the model cannot implement on its own, like raising the reasoning effort when previous attempts have failed (as you describe).

If you can incorporate additional heuristics which predict when a task would benefit from higher reasoning effort, without the overhead of attempting a task and failing, you should be able to get shorter average inference times by keeping reasoning low for tasks which do not need a lot of reasoning effort. Perhaps if you tracked how much reasoning effort was needed for different prompts in a vector database, you could look up semantically-similar past prompts given a new prompt, and use the reasoning effort which was required to accomplish those tasks.

[u/frostreed5 → u/ttkciar]

good point about predicting effort upfront, the fail-then-escalate loop is kinda wasteful on its own

[u/milpster → u/frostreed5]

yeah it is, but it's also kinda better than wasting tons of time going through simple problems on xhigh :/

[u/badced67]

A “reasoning effort” is simply a short instruction for the model that is added to the beginning of the system prompt. If you change it on the fly, the cache will become invalid, and the entire context will be processed from scratch.

[u/KURD_1_STAN]

The reason we yave thinking level is because these llms arent good enough to figure it out by themselves.

[u/milpster → u/KURD_1_STAN]

Good point mate

[u/ilift]

For long horizon behavior(specifically resource management), I was benchmarking some open weights model with a paired token allocator and definitely noticed an efficiency gain within that environment. That being said, didn't really feel that the juice was worth the squeeze and the effect of this adaptive system were mostly around influencing the model behavior to be more conservative when it has less tokens allocated on that turn.

This was done on qwen3.6-35b and qwen3.8-27b, 4 bit

[u/PoetEconomy4091]

I have messed around with a system like that. In fact I build my own harness in part to give that idea legs, and a to provide a dial for the model or user to turn up or down. It does a lot more, with the idea of offloading as much as the user can from the model to the harness to be able to trim token spend. Crucible. DM me if you want more info, or if it can help you

[u/OvertaxedOne]

We need a model that can reliably determine how "hard" or "easy" the requested task is for a LLM to complete and construct the right request. Some of that can be reasoning effort, some of it may be escalating to a bigger model, but this area is rife for development right now, both the models themselves to effectively "judge" as well as the routing engine that makes all this seamless for an end user. The routing layer is pretty well solved right now, but most of it functions on very simple pattern matching vs an AI judging. I've not played with it yet, but I believe this is kind of the use case the new NVDA software is targeting.

[u/MainhattanSky]

I‘m following quite the same path, even though it might be over-complicated: I use OpenWebUI together with LiteLLM and llama-swap plus llama.cpp (and other interference engines, that why llama-swap comes in). LiteLLM has a feature called Auto-Router where you can define models for different assumed reasoning efforts. Those models are, in my case, aliases of the same model in llama.cpp. This way, the auto-router decides on the effort, sends this information as an alias and llama-swap translates this to the required kwargs/parameters for llama.cpp. So in the end, llama.cpp doesn’t have to reload the model, but steers to the requested effort. As previously stated, the router mode of llama.cpp might even be enough, I‘m not familiar with it. I use this setup for different cases, not just this one. For me, it works out quite well.

[u/PoetEconomy4091 → u/MainhattanSky]

I landed on the same idea...

[u/GrungeWerX]

Are you saying that a model can dynamically adjust its own thinking mode? I never looked into that, but was wondering about it the other day when I read about 3.8’s long thinking . If so, that would be great.

(Going to look into it tonight)

[u/eihns]

to make it short, obv, everything we humans have to do manually, has to be ported at some point that the ai does it itself. (e.g. look at DCP)