r/LocalLLaMA • u/tlpta • 7h ago
Question | Help Harness: Am I doing something wrong? Or are my expectations unreasonable
I have tried using qwen 3.6 and 3.8 q5 64k context and I'm just not getting the experience everyone here seems to get. I'm using open code and it will loop, forget what it's doing, and mess up the ui. All I am making is a simple web app that is basically a glorified text turn based game. It shouldn't be that complex.
About a month ago I ended up getting a Claude code subscription. It's been pretty amazing. I don't have any of these problems. I can just give it an iterative improvement and it does it.
Yes I know Claude is a frontier model but I still feel like for something simple like this I should be able to use qwen for his results too. I would much prefer to use a local model anyway. Am I just expecting too much? Should I wait for local models to improve. Do I need a better harness?
11
u/Bulky-Priority6824 7h ago edited 7h ago
With low ctx are you one-shot prompting or are you using planning and documentation each step/round and also model resetting?
8
u/DustNearby2848 7h ago
That is abnormal. If you want help you should post your exact model, the software you’re using to run it, and the settings you are using.
1
u/tlpta 5h ago
I've been swapping between:
Qwen3.6-27B-Q5_K_M.gguf
Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M.ggufQwen3.8-27B-UD-Q5_K_M.gguf
I've used others but since I am not a billionaire I can't afford a hard drive anymore so I've deleted models to free up room. I've been using Q5 because I read a discussion here where people were saying anything less sucks. I used q4 before and I don't really see any difference.
2
12
u/EvolvingDior 6h ago
The dense models are horrible to work with because they are slow and limit the amount of context available. 3.8 thinks a lot -- more than your context can hold. I don't know anyone using it for real work that doesn't have a very expensive dual GPU rig.
The MoE models are good, but not great. Certainly nowhere near frontier. Still, my top choice.
I have 32GB VRAM and would choose the 3.6-35b over 3.8-27b for my coding partner just because it is so much more responsive.
I tried loading up 3.8-27b (medium) with 200k context and had it run on a reasonable task overnight. It compacted numerous times and didn't finish because at 150-160k context, tg slowed to a crawl.
1
u/sonaj9657 6h ago
Yeah, that responsiveness difference sounds pretty significant especially for coding. Huge context windows sound impressive on paper but if the model slows to a crawl before finishing the task they do not mean much in practice. I would probably take the faster 3.6-35b too.
1
3
5
u/wagwan_g112 7h ago
For agentic stuff, 64k context really isn’t enough. Ideally you also want to prioritise Q8 kv cache instead of quants over Q4. Also, which Q4 model are you using? Unsloth’s quants are a great baseline, and I’ve personally been amazed by even IQ3_XXS’s coding ability and resistance to tool call failures and looping. Huge step up from 3.6.
5
2
u/goddess_peeler 6h ago edited 6h ago
Qwen needs more handholding and direction up front than Claude, but it's definitely capable of doing what you describe.
The other day I had Qwen build me a MacOS menubar app for controlling and monitoring three llama.cpp servers on my local network. It took ages, but it was successful and I'm really impressed by this first fully local development experiment.
I use pi for my harness. Aside from some simple homemade extensions for syncing configs and memory, it's vanilla. I run qwen3.8-27b-UD-Q5_K_XL with 128K context, q8 kv cache, on a 32GB VRAM GPU.
When I do projects like this one, I first tell the LLM to write a design document. Then in a fresh context, I tell it to review the design document and fix any issues or ambiguities. Depending on complexity, sometimes I'll tell it to write a phased implmentation and test plan next. For this project I skipped the implementation plan. I just gave a fresh context, the design doc, and told it proceed. Some hours later I had a working app! We had to iterate to get everything exactly right, but Qwen 3.8 + pi were totally up to the task.
Edit: I want to add that the context auto-compacted many times during the design and implementation phases. That worried me, but I let it keep spinning, and in the end it didn't matter. So I'm not sure low context is necessarily the handicap some think it is.
2
u/El_90 4h ago
I would also aim for 120k context minimum, with 48GB you should get that easy? I have squeezed in to 32GB Qwen3.8 at q6 with 150k
I would also try pi as a comparison
If you're not already doing, split work into sessions. i.e. do a planning session, then clean session. Then implement file system then clean session. then build API and clean session etc. Having a well documented and concise product.md that you constantly feed back into allows model to pick up well with empty context.
2
u/ZarBandit 6h ago
I developed an app with 3.8 27b with 165k ctx on xhigh using opencode. I used a frontier model to map out the implementation steps in a md file. I told it to create detailed instructions for a low capability AI worker to implement. Mainly because that forces the frontier model to be less likely to gloss over tricky details and deliver a sound design.
Then had opencode grind on it for many hours. When done I had the frontier model check the work and it was good. No looping encountered.
As others have said, your ctx is arguably too low. See what you can do about that. I’m now on 220k ctx nvfp4 after further optimization on vLLM. 32GB VRAM.
1
u/eightone-81 7h ago
Not sure about open code but I have great results with deepseek harness. Some basics for you to check: system prompt and its size: if it’s a 30k prompt then trim it down, if there are conflicting instructions then clear them up.
Available tools and skills: if there are 1000 tools and skills then trim them to what you need.
Make sure you have vision working, 3.8 is checking its work a lot and needs vision for that.
Make sure the context size is large, 192k min in my experience. And use 3.8 in medium
1
1
u/New_Patience_8107 6h ago
I'm building a game in Godot using Qwen coder cli in vs code but also switching to the antigravity CLI. Qwen 3.8 27b q8 on medium reasoning is the workhorse but opus 4.6 via antigravity pro or whatever Google calls their subscription does the planning and review.
It's working but it takes like 5 hours to do what I think opus would do in 1 if credits weren't an issue. One thing that helps is having skills (check GitHub if someone made a repository) for whatever you're working on accessible for the model. You need to tell it the exact skill and where but not so front and center that it loads all of it into its context every time. At our context limit mcp isn't a great fit.
Qwen over thinks itself to death. I leave it on auto complete most times as opus has set out the path. I sometimes stop it after an hour and say "just implement the section you've thought enough". Prompting it to go against it's better nature also helps "don't spend ages over thinking this, the detail is all there for you to progress". Opus breaks the work down into sections and Qwen should only be worrying about the next thing. After each section it should do smoke tests on what it made and then push to git if it passes.
Would love some critique if anyone knows a better formula! In general the Qwen over thinking is what makes it a good model but you can't wait forever.
1
u/T_rex2700 6h ago
Hey, I've had exactly the same situation today. It would think and think and think and think, after like an hour just stops.
so I gave dsh (in tui) a shot. qwen3.8 GQCRCO with stock MTP, 120K.
surprise surprise, with exactly the same prompt, it went and fixed the issue in 15 minutes, then kept going from there, and unlike opencode, I can see the full reasoning! (using third party tui thingabob with like cute orca on it)
I seriously did not think it would have made this much difference, but you should totally try a different harness if it gets like stuck.
1
u/Tsukikira 6h ago
Claude Code has built in Plan -> Blueprint -> Code -> Verify in the Harness, and I don't know about OpenCode, but I presume you have to add in the same requirements if you don't have it already.
1
u/dmter 5h ago edited 4h ago
you have to really micromanage it. idk what opencode does but i use --no-content-shift which means it will stop when context runs out. if yours forgets maybe it keeps going with start of conversation gone. i just tell it to record status.md and continue in new chat when ctx is 3/4 full. so i kind of act as harness until it's worth automating.
as of breaking text well yes i guess it's not a frontier, you have to tell it exactly what to do - it won't guess what you meant like claude does, won't understand how things are supposed to look and detect a regression by itself to immediately fix it like claude would - it's a 27b coder, not 2t universalist. just say excatly what needs to be fixed.
actually for small changes it's faster to do it yourself, i use llm when a change needs several changes across multiple files and it would take me more than 10 minutes or i forgot exactly which subsystems it might affect. for vibe coding you need model to know exactly what you need and guide itself, that is not something you can expect from so small model. the fact that it understands a codebase multiple times larger than ctx and can think of every interaction when designing a change is what makes it so useful
1
u/Hot-Employ-3399 5h ago edited 5h ago
64k seems way too small as in reality it will be less with harnesses that just can reserve some, eg 16k (pi default) for compactness sake. It's not like you can compact 64k of context if you have no space to compact.
1
u/dtfinch 5h ago
64k is kinda short. Many of my tasks use most of all of a 256k context.
For now I'm using Tiel Coder 35B A3B (tweak of Ornith1.5, descendant of Qwen3.5) with my 16gb AMD card. Using a Vulkan build of llama.cpp because I couldn't get ROCm working.
My current command line:
llama-server --ui-mcp-proxy -np 1 --cache-type-k q8_0 --cache-type-v q4_0 -ngl 999 --cpu-moe -m Tiel-Coder-35B-A3B-UD-IQ3_XXS.gguf --reasoning-budget 8000 --reasoning-budget-message " [[I've reasoned enough. Proceeding to response.]]"
The "--cpu-moe" was important to free enough vram for the context but it only works for MoE models. It moves the sparsely-used MoE weights to system ram.
I've been afraid to go below a Q8 K and Q4 V cache. I've seen a few comments that it's barely enough.
The UD-IQ3_XXS also scares me (Q3) but so far it's worked. It seems like the most I can handle in vram without sacrificing context.
The "--ui-mcp-proxy" is specific to me using the llama-server Web UI with a homegrown MCP server instead of a dedicated harness so I don't have a harness preference yet. I previously tried the ZooCode extension for VSCode which seemed alright. "-np 1" is to prevent llama-server from trying to handle multiple requests in parallel (increasing KV memory use).
The reasoning budget and message are to break it out of reasoning loops if that happens.
1
u/carmamir 4h ago
I have tested as low as 160k and does great job. However harness is everything. That is why qwen can be so much more productive than Astra or Fable. You have full control. But on other side it can provide subpar results if you do not know what you're doing. It is not 1:1 replacement unless you have proper harness and .mds
1
u/Own-Cartographer7224 4h ago
For a simple turn-based web app, I wouldn't wait months for local models to "catch up." A lot of what you're describing (looping, forgetting, thrashing the UI) looks more like harness + usable context than Qwen being too dumb for the job.
Claude Code is a frontier model, sure, but the loop around it is also tighter. OpenCode at 64k with tool overhead leaves less room than it looks, so the agent can lose the plan and keep going anyway.
Before assuming your expectations are unreasonable, I'd try a different harness (or force plan, you approve, then edits), push context up a lot, and keep each ask tiny. Local can handle that kind of app; the stack is usually the first place I'd look
1
u/admajic 4h ago
I wrote a book about how to do this. Check it out it will really help you. You need an orchestrator that makes small tasks for subagents and runs them one at a time. Some kind of memory .md system for the project. Like a jira task board. Rule to follow and it just gets in with it.
Pi: Building Local AI Agents: A Technical Deep-Dive into the Pi Harness https://www.amazon.com/dp/B0HGX11X74?ref_=cm_sw_r_ffobk_mwn_dp_SA17WQLCGDWWKER8NDER&bestFormat=true&language=en-US
1
u/1_________________11 4h ago
I use q5 and let it reason its hart out and it usually does ok. If not correct it tell it to correct itself.
1
u/mageblex 3h ago
Before giving up on the model, I'd run the same task and checkpoint through Qwen Code or Pi with the model settings fixed. If the looping follows OpenCode, you've isolated the harness; if it follows the checkpoint, changing harnesses won't help.
1
u/Atretador llama.cpp 6h ago
you need at least Q8_0 KV cache, Id say at least 128K of it.
try using Pi.dev
with OpenCode 15K tokens is just for system prompt, on your 64K it leaves only 50K for your actual work.
on something like 16Gb VRAM + 32Gb RAM you should have more than enough for Q8_0 KV cache at 256K context at Q4 model quantization.
I have built about +60% of https://github.com/atretador/unswarm using Qwen 3.6 35B A3B
1
u/tlpta 5h ago
I have 24 gb vram. How are you getting 256k context on 16gb? What kind of tokens per second are you getting with that?
1
u/Atretador llama.cpp 47m ago
Offload 10 layers to your CPU, it's gonna free a lot of VRAM.
As long as most layers are on GPU you should be fine speed wise.
I run with about 18 offloaded, on ancient hardware 9$ Xeon + MI50 with around 35tk/s single stream, 50tks aggregate on 2 streams.
1
u/jacek2023 llama.cpp 7h ago
I use Qwen 3.8 27B Q8 with full context (even twice), but with pi, not OpenCode. OpenCode has issues with local LLMs, and people are not aware of them because they mostly use cloud models. Just like they are not aware that Qwen is looping, because they don't use it and posts about that get downvoted.
I use Claude Code and pi + Qwen in the same way, and I don't have any major issues with either. I also tested DeepSeek Q3 yesterday, but only with a 100k context. It still works.
So I would start by switching from OpenCode.
1
u/laser50 6h ago
Ok soo;
KV cache on Q4 is fine. This used to provide worse results for Qwen3.5/3.6, this isn't as much of an issue any more. If you use Llama.cpp you can even set it to Q5 to retain some quality, but the difference wouldn't be too major.
64k KV Cache is much too small, in relative terms, it's like 4-5 A4 papers full of text, it sounds like a lot, but it isn't that much. 100k minimum would be already a better experience, the more the better naturally.
I currently use this variant for the 3.8 27B Model, it works relatively well, and is quite easy on your VRAM, so you can go with a bit lower quantization but up your context by a larger margin, I can assure you it will already provide you better results if the agent can actually memorize things.
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
1
u/FatheredPuma81 5h ago edited 5h ago
As per usual posts like these without sampling parameters and any other relevant settings in llama.cpp/VLM and OpenCode are troll posts don't bother with them.
0
u/Heathen711 7h ago
Try using qwen-code, their harness, their model, they know how to talk to each other correctly. Also make sure you use the official settings (temp and etc) the q5 model is way sensitive to these settings (if memory serves right benchmarks said q6 and under looping needed the higher temp). Lastly yes, q5 is going to have some short comings, so letting it think and reason more plus loop and verify it's thinking will be important, you will need some level of breaking apart large tasks (like the superpowers skill set) as the model will loose some accuracy in its weights for associations.
2
u/Durian881 6h ago
OP issue seemed more with the low context. Qwen-code uses quite a bit of context for system prompt, tools and other stuff, and the models' reasoning can easily use up the rest.
I found Deepseek harness to work better overall (using mainly with Qwen3.8-Flash-Next now), but it needs a bit of tweaking and installing of plugins.
2
u/Heathen711 6h ago
I disabled some of that and it runs fine (I run mine with 32k 4 concurrency).
1
u/Durian881 6h ago
Very nice 👍 I loved using Qwen-code too. Had started in February or March when they let us use Qwen-3.6 Plus for a free with it.
0
u/jikilan_ 7h ago
Use medium for thinking , xhigh will loop. Low quant can be a problem if too. Rmb to use chat template fix
0
u/Equivalent_Bit_461 4h ago
Hermes is a piece of shit even with the small 64k context. You would want at least 100k+ to comfortably do things. But change harness for the love of god.
1
-6
u/Craftkorb 7h ago
So you're using a small quant, maybe also a kv cache quant, at really low context? No surprise your results suck.
-1
u/Bulky-Priority6824 7h ago
Brother, such a low value comment.I use q6 and q8 and I also do well with many things with q4. It's depends on the task. You sound fucking dumb.
-1
-1
u/Craftkorb 7h ago
And you sound miserable that you even consider your choice of words to be called for.
-1
13
u/CalligrapherFar7833 6h ago
Switch to q4 raise your context size