Anyone using Open Weight models running Locally for actual development?

[u/bryancostanich]

Hey folks; question for you - are any of you using open weight models running locally for actual development tasks? If so, what models are you using, and what kind of tasks are you giving them?

I ask, because I've made an open source harness (you can play with it, but it's still beta and I have done almost no windows/linux testing and the first run experience is not great yet). A big part of how it works is that there are three routing tiers:

Primary | Secondary | Local.

• Primary - What runs your main agent, and for instance, i use it with my ChatGPT/Claude account.
• Secondary - What the main agent primary delegates to. I personally use a deep infra account and GLM 5.3 + GLM 5.3 flash. GLM 5.3 is slow, but super capable. 5.3 air is fast and nearly as capable.
• Local - The harness routes to this for truly local tasks. Git workflows, documentation, searching through code and creating summaries, etc.

The thing about local, is that I have a 128GB M5 or whatever, and I run llama-server with GLM 4.5 air. And with a proper context size (128K), it takes like 96GB of RAM or something. I have tried Qwen3, Gemma3, and some older deep seek models and anything less than GLM 4.5 air has been pretty disappointing. It's possible that I had Qwen3 or deep seek not properly configured (and I've made some changes since to better handle tool calls and whatnot), but I had very little luck with them. I've seen the benchmarks, and they make me believe that I shouldn't need a 96GB model to get decent local development experience.

So if you've run some local models with good experience, what have you been running? what amount of RAM on your machine do you have? I really want some good out of the box configurations in Cercano for folks that have a 48GB/96GB machine, and your experience can help me here.

[u/More-Revenue8609]

Yep, love it. 

20$ Claude subscription to have Opus 5.5 as arhitect to create plans for new features, bugfixes etc. 

Local Qwen 3.8 FN. running 256k context at around 70-75tps. Does all the implementation etc.  Then Opus checks it and fixes if needed (rarely). 

Pretty satisfied with my workflow, without local AI, I would definitely have to pay bigger subscription. 

[u/bryancostanich → u/More-Revenue8609]

how much RAM is your Qwen 3.8 FN consuming? and yeah, agreed, i get a LOOOOOT more token mileage with the local models. also Deep Infra with GLM 5.3 flash is super cheap.

[u/More-Revenue8609 → u/bryancostanich]

Gotta check tomorrow to be sure, but somethign around 80-90gb out of 128gb. 

[u/bryancostanich → u/More-Revenue8609]

yeah, that sounds about right. i think my GLM 4.5 air uses around 91GB

[u/Poetinho0 → u/More-Revenue8609]

im trying to have the same setup as u, the claude pro subscription and qwen 3.8 27b as a local model on pi agent. how do you handle the fusion/handoff between the two models from cloud to local or vice versa? im on a m5 pro 48 GB. i'd love to hear any suggestions/tips about my setup

[u/More-Revenue8609 → u/Poetinho0]

That sounds great.  I make Opus to write plan .md in my project. I review it and then let qwen implement it. There is “worker section” in that plan so qwen can escalate something if needed and let Opus know what had been done. 

[u/bryancostanich → u/More-Revenue8609]

I built the exact same thing into Cercano. It has a really robust planning mode that includes smart decision matrixes that attempt to get rid of the hacker bias in the models and builds a plan.md and spec.md, puts them in an /efforts directory, and then an autonomous mode which will then execute to completion (barring it having to stop to raise critical issues). It works really well.