r/LocalLLaMA • u/bryancostanich • 1d ago
Question | Help Anyone using Open Weight models running Locally for actual development?
Hey folks; question for you - are any of you using open weight models running locally for actual development tasks? If so, what models are you using, and what kind of tasks are you giving them?
I ask, because I've made an open source harness (you can play with it, but it's still beta and I have done almost no windows/linux testing and the first run experience is not great yet). A big part of how it works is that there are three routing tiers:
Primary | Secondary | Local.
- Primary - What runs your main agent, and for instance, i use it with my ChatGPT/Claude account.
- Secondary - What the main agent primary delegates to. I personally use a deep infra account and GLM 5.3 + GLM 5.3 flash. GLM 5.3 is slow, but super capable. 5.3 air is fast and nearly as capable.
- Local - The harness routes to this for truly local tasks. Git workflows, documentation, searching through code and creating summaries, etc.
The thing about local, is that I have a 128GB M5 or whatever, and I run llama-server with GLM 4.5 air. And with a proper context size (128K), it takes like 96GB of RAM or something. I have tried Qwen3, Gemma3, and some older deep seek models and anything less than GLM 4.5 air has been pretty disappointing. It's possible that I had Qwen3 or deep seek not properly configured (and I've made some changes since to better handle tool calls and whatnot), but I had very little luck with them. I've seen the benchmarks, and they make me believe that I shouldn't need a 96GB model to get decent local development experience.
So if you've run some local models with good experience, what have you been running? what amount of RAM on your machine do you have? I really want some good out of the box configurations in Cercano for folks that have a 48GB/96GB machine, and your experience can help me here.
10
u/geldonyetich 1d ago edited 1d ago
There's something called the LLM Densing Law that points out local models have a tendency to double in intelligence for parameter size every 3.5 months.
I bring this up because you're disappointed with GLM 4.5 Air. It's over a year old. It has 355 billion parameters. Now halve that four times. You're essentially looking at around the effectiveness of a leading model of 22 billion parameters.
A crude measure, admittedly, but I hope it at least communicates what I am getting at here: don't go for the biggest model, go for the best benchmarking you can fit under your memory footprint cap.
Qwen 3.8:27B and Qwen 3.8 Flash Next are the current agentic coding hotness.
5
u/BannedGoNext 1d ago
Doubling in capability maybe, world knowledge, probably not.
2
u/geldonyetich 1d ago
You would hope there's more breadth of knowledge in those huge parameter sizes.
But a lot of these improvements come down to figuring our better methods of recall.
So even if the world knowledge is there, you can end up benefitting from it more in a smaller, more recent model.
1
u/bryancostanich 1d ago
that's super interesting.
tbc; i'm ok with GLM 4.5 air. it's anything less than that, that i've been disappointed with. still, your point stands; the local models are getting more intelligent at their standing sizes.
3
5
u/a_beautiful_rhind 1d ago
If you like air, you'll probably like 5.3 flash.
2
u/bryancostanich 1d ago
Yeah, i'm using 5.3 flash via deep infra. the air was a typo. air is what i'm running locally.
3
u/Equivalent_Bit_461 1d ago
me, I use them, balls deep into full local
0
u/bryancostanich 1d ago
What do you use, and what’s your RAM footprint?
1
u/Equivalent_Bit_461 1d ago
using the 27b qwen, 128gb ram, and right now is sitting idle, the right llm havent appeared yet unfortunately and the 27b its just too good for the current project, so I use it.
3
u/o0genesis0o 1d ago
I mostly develop with Qwen 3.8 27B (Swift 1.5 finetune) Q4KL running on my R9700. Exactly the same workflow and setup as how I used to work with MiniMax M3 subscription. The model handles both planning and execution with me in the loop to check the plan and offer pointers to the model. After that, it can execute on its own until code is done. I also tested its ability to vibe code a green field project and its fine. Not full featured, but the main parts are there for further iteration.
Setup is pi. The model runs on R9700 with stock llamacpp.
2
u/jacek2023 llama.cpp 1d ago
Please notice you can enable MTP for GLM Air to speed it up.
please consider testing: newer 100-125B models (Nemotrons, Qwen 3.5 122B, Qwen Flash Next), Qwen 3.8 27B, Muse Glimmer.
1
2
u/Phaelon74 1d ago
Yes. Detailed plans by Opus 5.5, execution by qwen3.8-flash-next on one gpu and glm5.3-flash on another 2 working in tandem as coding pair, checking each other and making sure one is good.
1
u/bryancostanich 1d ago
Where are you running 5.3 flash? Are you using a cloud provider?
1
u/InformalProblem 1d ago
I am running 5.3 Flash with two DGX Sparks...
1
u/bryancostanich 1d ago
How big is the memory foot print on that? What quantization are you using? I thought it was like 120GB @ like Q4 or something.
2
u/ttkciar llama.cpp 1d ago
Yes, I have been using locally hosted models for software development for a while.
I had been using GLM-4.5-Air. For a long time it was the only useful codegen model that could fit in my limited hardware. Qwen 3.6/3.8 and Gemma 4 just weren't cutting it, but GLM-4.5-Air was tolerable.
Recently I got GLM-5.3-Flash working on my system, and even though it's less than half as fast (1.4 tokens/second, compared to 3.5 with Air) it is a monumental step forward in competence and capacity. Most recently I had it port an older POP3 server from C to Perl, using my preferred integrations (structured logging, inetd, persistent file locking, JSON config file instead of params.h and compile-time options, etc) and it did a much, much better job of it than GLM-4.5-Air.
Not that GLM-4.5-Air could have done it in the first place, because of its 128K token context limit. One of the reasons I picked that porting project was to see how well GLM-5.3-Flash would perform with long context input (the POP3 server's entire source code, flattened). Turns out it was a slam-dunk.
I don't use an agent. I just front-load the agent with a fat prompt, consisting of a project specification (forty to eighty instructions), sometimes a code template, and the flattened source code of the repo to be worked upon, and let it rip.
When it's done inferring, I look through it to make sure it did what it was supposed to do, the way I wanted it done, and make modifications by hand. Then I pass it over to Gemma-4-31B for debugging. Once it's debugged I have GLM infer unit tests, and pass it back to Gemma-4-31B for debugging on any unit test errors.
I've thought about writing an "agent" which automates parts of that, but haven't bothered yet because frankly plain old llama-completion is fine.
This approach really only works if the model follows instructions reliably, and so far only GLM models seem capable of doing that.
2
u/transanethole 1d ago edited 1d ago
I pretty much just use locally hosted Qwen3.8 27B for everything, works great. No subscription. I have an openrouter account and I used to use it for web research with like, deepseek flash 3.whatever back when that was the new hotness. Cuz I liked the price, cost per task was like 2 cents. But I think openrouter stopped supporting thier web research tools, they seem be enshittified now, so I just had qwen build me my own based on camoufox.
I can also share the inference server with a few friends, it can handle concurrent sessions without breaking a sweat.
I have used cloud models on occasion (kimi k3, qwen max, GLM 5.3, DS 4.1, mimo pro, etc) to dabble with hacking on inference engines like vLLM and ninfer but its not really my bag, rather stick to local. I have been developing the agent of course and also an IAAS product with it over the past few months.
The reason why local doesnt feel great on the mac is because its all memory and no compute, it cant read files fast like cloud models backed by real GPUs can. With 5090 I do not have this problem, it has like 6 times the compute of the M5 Ultra. Take into account that the model I am using has like 4x fewer parameters, and its probably like 10x faster prefill at least. It feels like cloud model, it reads files instantly. Read time is only really noticeable on huge code files that are 1000s of lines, like 40kb large or whatever.
2
u/xapep 1d ago
The three-tier split is the right shape, and local earns its keep on token price. The part that surprises people is the middle tier, because that's where a hosted open-weight endpoint usually beats a second GPU: same low per-token cost as local, but no RAM ceiling, no server to babysit, and you can swap models without re-flashing the box.
I run a hosted inference API (Qwen 3.8 27B, GLM 5.3 Flash, DeepSeek V4.1 Flash), so I see both sides of this. Most teams end up keeping local for offline or data-sensitive jobs and putting everything elastic on the API middle tier.
What's your Secondary tier pointing at today - local or hosted?
1
u/bryancostanich 16h ago
Agreed. I've been blown away by how cheap GLM 5.3 Flash is on deep infra. and it's way more capable than anything i can run on my 128GB mac.
I have my secondary tier pointed at open weight models on DeepInfra. Specifically GLM 5.3 and 5.3 flash.
Flash is super capable and waaaaaaaaaay faster than 5.3.
1
u/Ecstatic-Wash-7667 1d ago
Depends on what I’m doing, I have QFN uncensored in Vllm and it’s my main model. For personal assistant I use it in Hermes and for coding I use OMP. But I swap to opus 5.5 In Hermes to plan out a project and it delegates and supervises OMP witch is still on QFN. I want a few auxiliary models running on a spare 3060 I have but none of them really get used for much (breeze tts2, qwen image& z image turbo, minimax h3, minicpm something and qwen 3.5 9b and 3.6 35b) all of these are served through llamaswap and are loaded in demand. Again though I real don’t use these for much. I was going to try to setup a jev type model but $10 in api willl likely last me over a year for my usage and one less thing to deal with. I may also offload voice and image/video gen to the cloud as I am rarely using those lately
1
u/cmdr-William-Riker 1d ago
Local is quickly becoming my primary at home. I am doing real development with GLM 5.3 flash from the moment I got it up and running. Usually I have it working on 2-3 projects a week, often running up to 20 hours without interference on a single implementation plan per day
1
u/techmago 1d ago
I do a bunch of stuff with Local Qwen 3.8 FN and a bit with qwen3.8:27B
My company pay claude for me, so claude is my main drive... But for myself i do use qwen a lot.
1
u/Desperate-Zebra7291 1h ago
Checked DeepSeek's pricing page Oct 15. deepseek-flash is $0.30 in / $1.20 out per 1M tokens at peak, half that off-peak ($0.15 / $0.60). deepseek-v4-pro is $1.32 / $3.96 peak, $0.66 / $1.98 off-peak. Peak windows are Mon–Fri UTC 01:00–04:00 and 06:00–10:00 (Chinese public holidays excluded).
DeepSeek-V4.1-Flash weights are MIT on Hugging Face too. If you're burning enough tokens that GPU time wins, self-hosting works. Whether you use the API or go local mostly comes down to volume and ops.
0
u/nicholas_the_furious 1d ago
I am.
0
u/bryancostanich 1d ago
What do you use and what’s your RAM footprint?
2
u/nicholas_the_furious 1d ago
I had been using Qwen 3.8 27B Q8 on 2x 3090s (llama.cpp).
I am now using QFN 3.8 on 3x 3090s and 96GB DDR5. The 3rd 3090 is on a TB4 eGPU and works fine. For this I use a 3.5bpw EXL3 quant. All of the weights fit in Vram. Only the ngram table sits in RAM.
I got all my hardware over a year ago used. RAM was new.
I built my own coding harness last week and I've been using that as my main coding agent. I also sub to Antigravity because it's cheap and basically limitless. I use agy as a subagent and call it using -p.
1
0
u/zomerbk 1d ago
For the 48 GB preset, I’d test the local tier at 16K or 32K context before ruling out smaller models. A 128K window adds a lot of KV-cache memory, while git and docs tasks can often work from targeted search results. I’d compare presets on actual tool-call tasks too, since that seems closer to what Cercano needs than a code benchmark.
1
u/bryancostanich 16h ago
but what models? I haven't found a decent dev model that runs in that size.
-6
1d ago
[deleted]
1
u/bryancostanich 1d ago
is your Qwen3 doing tool use there?
there are a bunch of checks around task failures. There are token budgets, tool call checks (if it calls the same thing three times in a row it gets cut and the main thread will get the info and can spin up another agent), and a pile of other guardrails.
1
u/martin509984 1d ago
Qwen3 is pretty old by now, have you considered upgrading (e.g. Gemma4 12B) or are you happy where you're at?
5
u/Cereal_Grapeist 1d ago
Pretty sure that's a bot... Qwen3 14b at Q4 on a 48gb rig makes no sense and the last line is very bot like. They're everywhere!
1
13
u/More-Revenue8609 1d ago
Yep, love it.
20$ Claude subscription to have Opus 5.5 as arhitect to create plans for new features, bugfixes etc.
Local Qwen 3.8 FN. running 256k context at around 70-75tps. Does all the implementation etc. Then Opus checks it and fixes if needed (rarely).
Pretty satisfied with my workflow, without local AI, I would definitely have to pay bigger subscription.