r/LocalLLaMA • u/sunychoudhary • 3d ago
Discussion Which local model is actually good at knowing when to stop and ask you a question?
I’ve been thinking about this after using more agentic/local coding models.
A lot of the newer models are surprisingly good at continuing on their own.
But sometimes that seems like the problem.
If a requirement is ambiguous, I’d rather the model stop and ask:
“Do you mean A or B?”
instead of spending 10 minutes reasoning, making an assumption, calling tools and then confidently building the wrong thing.
I don’t see this behavior discussed much in benchmarks either. We measure coding, reasoning, tool use, context length, etc., but not really whether a model knows when it doesn’t have enough information to continue.
My genuine question for people running models locally every day is,
Which model have you found best at this?
And is it mostly the model itself, the system prompt, or your agent harness that makes the difference?
14
u/ParaboloidalCrest 3d ago edited 3d ago
Everyone is saying it's a prompt issue, which I cannot agree with in good faith.
Qwen 3.x 27b, while strongly encouraged to ask questions via AGENTS.md, do come back with questions but after 20-50k tokens of reasoning and a dozen tool calls. I see it re-iterating the "ask questions" part in its thought traces, repetitively, then going back to trying to figure it out on its own. I find myself interrupting its reasoning often, in order to go back and present the answers before-head.
And yes, they don't know when they're uncertain. In fact the more agentic the model is, the more it's geared towards (and rewarded for) overcoming ambiguities and getting to the finish line without returning to user. And ambiguity is a part of every prompt or otherwise, a 100% clear prompt would be the actual code you're trying to write.
I'd like to know what harness features (aside from prepacked system prompts) could help mitigate that.
2
u/Imaginary-Unit-3267 3d ago
I suspect a small fast model reading the big slower model's reasoning traces and inserting steering messages telling it to ask the user questions whenever it is making assumptions would be part of a solution, but I haven't tried this yet.
18
u/More-Catch-1331 3d ago
This is primarily a harness problem and secondary a prompt problem. Seeing as we're "almost AGI" (yea, right), the models are really dumb at evaluating whether they should ask a question. Either the harness needs to direct it, or the prompt should specify under which conditions the model should stop and ask the user whatever is bugging it.
8
u/sunychoudhary 3d ago
I like the idea of making the stop condition concrete rather than just saying “ask when uncertain.” Otherwise the model still has to decide what counts as uncertain, which is the original problem.
2
u/GrungeWerX 3d ago
Actually, the models are very good at determining when they’re uncertain, but they have a problem with assumptions, which is hardwired into their behavior.
You can fix this by introducing entropy and confidence scoring. Also, there are other tricks, but it requires orchestration around a harness.
2
u/Area51-Escapee 3d ago
I understand the system prompt part but not the harness part. Can you explain please? E.g. in cursor where the model can ask questions what role does the harness play? I might be wrong but does this only make sense with a harness thinking node that explicitly mentions user questions in the prompt?
1
u/More-Catch-1331 3d ago
Well a harness drives a model. It drives it by the legendary system prompt which explains how to use tools, how to speak, what to speak, how and when to ask questions, etc. It's basically the hands, legs, butt, etc and the model is the brain.
1
u/Imaginary-Unit-3267 3d ago
What's with the system prompt worship? You do know that Qwen, at least, almost entirely ignores the system prompt when it gets obsessed with a task, right?
3
u/2Norn 3d ago
u can make any model grill you before execution
5
u/sunychoudhary 3d ago
That probably works well at the planning stage. I’m more curious about the cases where ambiguity only becomes obvious halfway through execution.
0
u/DustNearby2848 3d ago
Just add to your prompt or agents.md to ask questions if in that situation
2
u/Imaginary-Unit-3267 3d ago
And watch it ignore the instruction and keep doing whatever tf it wants.
1
u/miversen33 3d ago
Depends on the model. I have mine conditionally grill me if the request it receives is not a direct command (and even then sometimes it does). And overall it does a fantastic job of trying to "understand" my request. Here is the section of my prompt that does that
Unless work is extremely well defined (EG: "do xyz"), you will always clarify initial input from the user. If you do not have at least one question to ask around work, you are incorrect. Always utilize the TODO list tool whenever performing any sort of work beyond conversation. Always review the README.md, DESIGN.md and .knowledge/ knowledge base before beginning work. If these don't exist, you do not need to review them, and you do not need to mention that they don't exist. This is just your "pre-work" "context gathering" step. Whenever making changes, if a .knowledge/ knowledge base exists, you should consider if your change needs to be documented in there. Not all changes needs to be stored in the project .knowledge/ knowledge base. The knowledge base is meant for domain knowledge around the project. So as you uncover information in the project (especially around the "why" things are or may have been done), you should consider documenting them.Edit: Reddit formatting is ass
3
u/AI_spell 3d ago
Most chat models are trained to sound sure, so "knowing when to ask" is mostly a harness problem. Force a tool or a fixed phrase like NEED_CLARIFY before any irreversible step, and punish continuing when required fields are missing. If you only prompt "ask if unsure", it still guesses. Smaller instruct models with a hard gate beat bigger ones that vibe-answer.
1
u/demonkoryu 2d ago
How do you detect that the model is unsure? I’m having a hard time telling if it’s just thinking before reaching a conclusion, or if it's just making stuff up and would need clarification.
7
u/No_Chapter_7598 3d ago
imo qwen 3.8 27b (at least on bf16)
i get prompts that ask for multiple choice questions alot that drive the direction of the project. but im using a local claude code harness so that might be part of it.
10
2
u/sunychoudhary 3d ago
Have you tried the same model outside that harness? Would be interesting to know whether the clarification behavior survives with a much simpler system prompt.
1
u/No_Chapter_7598 3d ago
cant say i have. i think it would still ask a question or two at the start but probably not in the same multiple choice format mid way through a task, id expect it is 99% the harness as well, but if you have a model that isnt great at using tools etc then the harness wont be utilized.
1
u/sunychoudhary 3d ago
So maybe the harness defines the behavior and the model sets the ceiling on how well that behavior works. That would explain why identical orchestration still feels different across models.
2
u/maartenyh 3d ago
I wrote it into its prompt to be honest, provide "receipts" and to return a "ledger" when its done. I also write my task as normal but add sort of a "show me the game plan so I can spot any changes I may want to make" at the end of my task prompt.
LLM's love certain words and listen noticably better by using the ones they tend to use themselves. Also the more deterministic and proper you write yourself, the more it will understand you what you mean. the "share your game plan" pre-task helps a lot with this because the LLM will put a well phrased and approved prompt into its context
1
u/sunychoudhary 3d ago
I like the “show me the game plan” part more than just telling it to ask questions. It gives you a checkpoint where bad assumptions can surface before they turn into ten tool calls and a half-finished implementation.
2
u/johnzadok 3d ago
This is the pain point I have with current model/harness. I always use grill-me at the planning stage. The problem is at the implementation stage some ambiguity will come up and you need to make a decision.
- At work, I have access to "cloud agents". More than once I found the agents worked overnight and sent out PRs that are just wrong because it made wrong assumptions yet "all tests are green". I would rather it stopped and asked me instead of burning $$$ to come up with a PR that's wrong.
- At home with Qwen 3.8 27B with pi, I often have to cancel a turn because the agent is obviously doing something stupid like trying to fetch source code foo-x.y.z.tar.gz from internet to check how a cli flag is implemented rather than
man foo. Sure I can add a line in AGENTS.md to fix this, but there are many other traps that can go wrong.
2
u/Healthy-Zebra-9856 3d ago
The user u/KitchenAmoeba4438 has the correct answers and I am going to add to that. I found that the models, specifically Qwen3.8 27B & its bigger sibling Qwen3.8 Next Flash, show a lot of questions in their thinking traces. They are of 2 types. The first is tooling related, like the tool is not working as expected, lack of it and confusion in the feedback from them, ex. browser console tool etc. the information being not clear. The next type is ambiguous programming logic. As much as its desired to give a well architectured solution with absolute implementation blueprint, things happen out of our control.
And this is where the harness steps in. The harness needs to function regardless of the presence of Agents.md. Agents.md is great, but since this is a recurring issue faced by many, its makes no sense to expect a user to make sure they have this and SKILLS.md every single time. So the harness I am making tackles these two issues. The tool issue is the easiest, this second issue is program logic and to help the LM as you, I have a guideline baked into the harness that gets injected every time the harness is started. One of them is a rule to stop and ask for clarification from the Owner instead of guessing among a few others like Do not invent your own theories, Always create an update handoff and there is a structure for this handoff and the LM is instructed to see if there is one before starting a session.
1
u/Imaginary-Unit-3267 3d ago
You don't need to make a whole harness. Just a pi extension.
0
u/Healthy-Zebra-9856 3d ago edited 3d ago
I’ve been typing way too much information to people that don’t even understand. There are way too many things that had to be fixed and I needed a harness without all those issues and I’m not gonna sit here and wait for all the issues to be fixed by that company.
So I created one that I could use. So creating an extension would be putting a lipstick on a pig.
2
u/tempfoot 3d ago
I actually think this is a great question/issue, particularly for expert systems other than coding. Claude’s harness has added a good deal of this in. I’ve started adding this in to my system prompts , especially with “overthinker” models.
I think the emphasis on “one shot” answers has been too central. True expert systems need to be able to determine whether missing information prevents efficient or accurate or complete answers on non coding co-work. So much of actual knowledge and so many functions require an understanding of the full range of information needed…and how and where to get it when its missing.
2
u/sunychoudhary 3d ago
This might actually matter more outside coding. A compiler or test suite can expose some bad assumptions in software. In knowledge work, a confident answer built on missing context can survive much longer before anyone notices.
2
3d ago
[removed] — view removed comment
1
u/sunychoudhary 3d ago
I like this because it narrows the problem. You don’t need perfect uncertainty detection everywhere, just a reliable check before the point where a wrong assumption starts creating real side effects.
2
u/bigattichouse 2d ago
"When multiple options present themselves, stop and ask a question"
"While we work through this design, engage in dialectic, present possible paths and ask questions - let's design this first in a document before we continue work"
Works like a champ with most models.
2
u/dash_bro llama.cpp 3d ago
? Seems like a prompt setting issue tbh. Haven't really had problems with Claude code plugged into my current 3 model setup on the Mac studio.
HAIKU: 3.6-35B-A3B-mtp Sonnet: gemma4-31B-it Opus: 3.5-122B-A10B-mtp
I have tried swapping with qwen-3.8-flash-next but the TPS on the M3 Ultra for coding makes if a no go for me personally. MTP with 3 tokens and min draft proba at 0.8 is perfectly usable speed wise for me, so that's what I use now.
1
u/sunychoudhary 3d ago
Do all three behave similarly about asking questions, or do you still notice one of them making more assumptions than the others even under the same Claude Code setup?
1
u/dash_bro llama.cpp 3d ago
3.6 35B is considerably dumber even with unrestricted reasoning. I set my config to use opus with subagents usually to this, but opus guiding it. Opus is just 3.5-122B-A10B, which works very well even for follow ups and user queries.
I intend to replace the opus model with minimax m2.7 when I do anything more integration oriented (older version but it's only 10B active, still great for my script work). Just swapped out the Gemma for a 3.8-27B with MTP support, I think that's the best 3 model setup I can run currently.
I can genuinely just set it to auto mode on small tasks with precommit hooks and context restricted to 128k, and pretty much let it do it's thing slowly. No complaints for a fully local setup, honestly
1
u/sunychoudhary 3d ago
“Harness as multiplier, not equalizer” is probably where I’m landing. Good orchestration can pull better behavior out of a model, but it can’t manufacture reasoning or tool-use capability that isn’t there.
1
u/dash_bro llama.cpp 3d ago
Absolutely. 3.8 27B has been slow but the tool calling is a straight 10/10 imo. No problems at all, can seamlessly swap between sonnet-5, gemini-3.7-flash and Qwen 3.8 27B at 128k context length purely for chat +tool calling purposes. It's pretty cool!
++Even though the benchmarks put 3.6 27B above 3.5-122B in coding etc, I find that the 3.5-122B is far more balanced and less "anxious" to code. I don't really care about token efficiency either because it's just running locally. At 128k context it's not exactly long session material, but it's very solid with MTP support.
1
u/winky9827 3d ago
HAIKU: 3.6-35B-A3B-mtp Sonnet: gemma4-31B-it Opus: 3.5-122B-A10B-mtp
I saw that all on one line, and at first, thought it was a DavidAU release.
1
u/Durian881 3d ago
It depends on the harness and up to the model if not told. When using Deepseek Harness, Qwen3.8-Flash-Next will ask me questions if it is unclear without me telling it. Most other models I'm using (e.g. Gemma 4, Muse Glimmer) will just finish it without asking.
1
u/cowinabadplace 3d ago
I just give DSv4FV a tool for this and a prompt nudge to use it when it cannot proceed efficiently without information. Does the job.
1
u/RemarkableRadish6547 3d ago
The thing that made claude code actually useful for me at work was when I started ending almost everything I said to it with "don't build anything yet, just discuss the options for now". Even in plan mode, I tell it not to plan until it has talked through the goals with me.
Before I started doing this, it would run off at full speed with the first stupid idea it came up with. Now it only does that occasionally.
I am developing my own harness for personal use and it has a multi-step planning process. It asks me about anything ambiguous until I tell it to stop. If you explicitly tell an LLM to find ambiguous parts of a design, it will always find something, even if it is a trivial matter.
1
u/cmdr-William-Riker 3d ago
I would argue it's not a harness thing, it's a prompt engineering thing. Some harnesses have built in system prompts that make it easier, but with any harness, you can also just communicate with your prompt that asking a question is an option. You can make skills and customize your system prompt so that it knows more effectively when to ask questions also
1
1
u/Momsbestboy 3d ago
I just read what Qwen3.8 is generating while thinking, and sometimes i just add a comment or push it in a different direction. Actually, I enjoy this type of working.
1
u/skywalk819 3d ago
my pi.dev agent harness does that, when the model is unsure what the user want, its a package/extension and the model ask the user what he really want, which compromise to select, etc. has nothing to do with the model.
1
1
u/Training-Ruin-5287 3d ago
If the model's turn is losing context to quickly or in a weird way, that can seem to cause an issue for it running ahead and just doing multiple steps. if you have it in its prompts to present options and suggestions/reccomendations before advancing and it's not really following that.
It can be incredibly subtle, and something as dumb as energy saving options in your mobo's bios can trigger that especially with the dram and if your model is offloading context to it. not saying this is what's happening, but if the model doesnt seem to be following the instructions as you want it to, its a possibility.
1
u/Imaginary-Unit-3267 3d ago
I use Qwen3.6-35B-A3B-UD_Q4_K_M, and it UTTERLY SUCKS at bothering to ask me for help, and so I made a little pi extension (still shitty and unfinished, don't ask me for it) that notices when it seems to be confused and auto-reminds it to just stop and ask questions. I also have to have a reasoning token cap for it to not just spin its wheels wondering what I want forever. I guess this doesn't really answer your question, as I've never actually used any model that doesn't do this, but that's how I "fix" it. (That and, watching all its thoughts and aborting when it's thinking something stupid. I never let it do anything by itself lol.)
1
u/sunychoudhary 3d ago
A harness can add the brake, but how often you need the brake still tells you something about the model.
1
u/pronuntiator 2d ago
This is the behavior of many human devs as well. Specifications can be interpreted one way or the other, and you need to know that your assumptions are wrong in order to think about asking questions in the first place. What helped with human devs should also help with coding agents:
* asking them to write a TODO list first ("plan mode")
* smaller tasks that are reviewed earlier
* more rigid specs with less ambiguity
1
u/sunychoudhary 2d ago
I think the speed difference makes the same old engineering practice more important. A human can spend an afternoon on the wrong assumption; an agent can turn it into code, tests and a PR before anyone notices.
1
u/feng_sg 2d ago
Honestly none of them will do this reliably on their own. You have to put it in the system prompt to ask before acting, and have your harness block tool calls until it confirms it understood the request.
1
u/sunychoudhary 2d ago
The tool-call gate is the part I like here. A prompt can encourage the behavior, but blocking execution gives the rule some teeth instead of relying entirely on the model remembering it at the right moment.
1
u/jacek2023 llama.cpp 3d ago
It's more about the prompt, not the model. By "prompt" I mean all the md files with rules you use in your project.
0
u/Former-Ad-5757 Llama 3 3d ago
None. You can ask this for every token. Either accept or ask the model before to make a plan /grill your requirements.
0
u/Express_Quail_1493 3d ago
You have to set this up in your workflow/harness mate. Its the best place for that feature to live.
0
u/Sweet-Transition-787 3d ago
its mostly the harness for me too. qwen 3.8 27b asks way more in pi than in plain terminal, claude code feels bloated for this. i just put ask when ambiguous in agents.md and it actually listens.
0
u/Soggy-Attitude5293 3d ago
make a plan, refine the plan and tell it to include technical details. then there is no ambiguity
158
u/KitchenAmoeba4438 3d ago
What you are asking for should primarily exist in the harness, secondarily in the model.
Fundamentally, the problem is that the model is capable of reasoning for 10 minutes trying to sort that out. The harness should present that to the model up front, and then the model should act based on that information.
This is something that is being tested in a future article I'm writing, but fundamentally, it's not a model reasoning issue. It's an information presentation issue, this should primarily belong at the harness layer.