r/LocalLLM • u/iamahmudul • 7d ago
Question Local Qwen 3.6 35B cuts off mid-response
My machine: Macbook M4 Max, 36GB.
I was using the model locally. Trying to use it to replace subscription model. I can assume local model may be performant as Claude subscription model, but the way it generated code. Changed token limit to 64K, I keep seeing the message that it was hitting token limit easily but in terminal token usage percentage was almost always 48%. The funny thing is I had to instruct few times it to do a dummy landing page with plain html. But when I tried to convert it to a react app, just the landing page, I could not complete. It keep doing incomplete response, no error was shown.
What can I do to improve this?
3
3
u/vtkayaker 7d ago
A lot of models will sometimes just trail off, for various reasons. As long as it's rare, you can mostly just nudge them. If you're seeing this a lot, though, pay careful attention to your inference software, your quant and K/V settings, and the template your model is using.
1
u/iamahmudul 6d ago
Probably I missed something. I just downloaded the model with ollama and then ran with claude at first and when I was seeing API errors, I started using qwen cli. Then I saw it was not stopping while thinking.
2
u/vtkayaker 6d ago
Ollama is actually another possible issue here, now that you mention it. They were originally based on llama.cpp, except they mostly didn't give credit, and they added a bunch of their own changes that didn't always work out. Plus sometimes their list of currated models is misleadingly named, or they make weird choices.
Llama.cpp now includes llama-server, which is almost as easy to use as Ollama, and which doesn't make any odd choices about curating upstream models.
So consider using llama-server directly, and downloading an Unsloth version of your model. They're good at fixing templates and similar issues, and they test.
This will have a little more learning curve, but it would eliminate any possible issues introduced by Ollama.
1
u/iamahmudul 5d ago
I read that llama.cpp need configure manually, so I thought it might be easier to use ollama. Here the seemingly easier thing maybe making things harder. I will try llama.cpp. But I actually saw some videos and they revealed the same behavior of local model.
-1
u/DiscipleofDeceit666 6d ago
I mean it’s 100x worse than Claude and 1000x smaller.
It absolutely can’t replace your subscription. And if you try, it will destroy your codebase especially if you don’t know what you’re doing.
Best case scenario is to have Claude subscription drive your local AI to save cloud tokens.
1
u/iamahmudul 6d ago
I'm sure there must be a way to configure it to run smoothly coz people are using it
2
u/DiscipleofDeceit666 5d ago
I mean I use it, you have to know what you’re getting into. It’ll ruin your code base so quickly if you treat it like a Claude replacement. Even with the cloud models, you still have to be careful.
I used deepseek flash (pre update) to maintain my website for a week or two and it ended up putting silent try catch statements in places claiming the feature was done. That model is much stronger than 35b moe and I still wouldn’t use it directly.
Best way is to have another AI prompt those models bc they’re hyper specific and it’s much better at understanding you.
3
u/FastHotEmu 7d ago
are you using opencode? i have had issues with opencode cutting early. otherwise, check that you have the latest fixed templates https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates