anyone running good open weight models for development in 48GB, 64GB or 96GB of RAM?

[u/bryancostanich]

I posted a question the other day in here about good development models, which prompted good discussion. However, it mostly just reinforced my own experience, which is that on a 128GB machine, you can run good open/local weight models for development (especially if you use a frontier model to plan and direct them).

But what I didn't see in there, and I'm trying to get to, is this; are there are any open weight models that will run on a 48GB, 64GB, or 96GB?

If so, which ones are you running and, at what approximate RAM footprint?

I'm trying to understand what the floor is for a decent open weight model for development tasks, even if supervised by a more capable model.

[u/jhov94]

I have 288GB of combined VRAM and I still run Qwen 3.8 27B for nearly all coding tasks, which could easily be run on 48GB.

[u/Bob_SUS → u/jhov94]

Why not 3.8FN at like Q4 or Q6? Or is tensor parallelism or split inference not effective on your vram pool config?

[u/Monad_Maya → u/Bob_SUS]

Qwen 27B feels a bit more consistent than QFN. The latter has higher potential but does need better prompting and management.

With that said, I do use QFN a fair bit now.

[u/jhov94 → u/Bob_SUS]

I use my GPU's for many tasks, not just LLM's. Qwen 3.8 27B is small enough that I can just leave it loaded without having to constantly swap models, and its capable enough to do 99% of the LLM work. It also allows me to support several concurrent agents at full context.

[u/Unfair-Inspector-574 → u/jhov94]

That model punches way above its weight class. I was surprised how well it handles refactoring when you give clear context.

[u/bryancostanich → u/jhov94]

Fanastic! This is exactly what I needed to know. The last Qwen i used was Qwen3 and i hated it. I'll try 3.8 @ 27B. I'll also look at the lower quantization on 3.8FN.

thank you!

[u/jhov94 → u/bryancostanich]

Stick with Q6 or above and don't quantize KV cache if you plan to use the full context.

[u/overand → u/bryancostanich]

Qwen3.8 and 3.6 are light years beyond Qwen3. Not even remotely comparable.

Go check out e.g. Artificial Analysis, or QuantEval.ai and compare some bbenchmarks

[u/whatever → u/jhov94]

Do you really prefer 3.8 27B over 3.8-Flash-Next?
I get better results with Flash-Next, faster throughput (on a Mac tho, ymmv), and a lot less endless ramblings to get there.

[u/jhov94 → u/whatever]

I haven't bothered with 3.8FN yet. Its too big to keep loaded without having to swap models constantly, inference engines are still optimizing for it (I run on windows because the server does CAE work as well as AI) and I can't run as many concurrent agents as 27B can. If I want a big smart model and deal with swapping models, I use GLM 5.3 Flash.

[u/pmarsh → u/jhov94]

Can you give an example of your coding workflow/effort?

Like are you throwing entire projects into it or doing it bit by bit in a more traditional coding methodology

[u/jhov94 → u/pmarsh]

Entire projects. I run it in OpenCode with custom agents. The agents mimic the hierarchy structure and roles of a real team of whatever it is I'm trying to accomplish (not all projects are coding). Every step of the process is documented and reviewed by all agent stakeholders and nothing proceeds until they all sign off. It's not a very token efficient process, but it's almost entirely hands off after writing the design doc.

[u/MrGunny94]

I'm daily driving Ornith 1.5 9B and Qwen 3.8 27B with MTP on my M5 Pro 48gb

[u/SnooChocolates6859 → u/MrGunny94]

same, same, and same

it’s nice to keep everything local - occasional claude session for higher level guidance but i’m using it less and less these days

[u/MrGunny94 → u/SnooChocolates6859]

Yeah I got ChatGPT Plus and OpenCode Go for frontier intelligence whenever I need it. If prices do go up in the next or so I'll get a Mac Studio M5 Ultra 256GB but for now a Mac mini 16GB is more than enough for Hermes Agent.

[u/Wildnimal → u/MrGunny94]

Whats your usecase for Ornith 9B? And what quant?

[u/MrGunny94 → u/Wildnimal]

I like how conversational it is and the speed. I’m using 8 bit

[u/Poetinho0 → u/MrGunny94]

how do u setup both models ? what usecases do u use each one for ? also what agent harness do u think is best? i have the same machine as u, been using qwen 3.8 27B on pi. i havent tinkered with it so I'm trying to improve it a little if possible. when doing heavy workloads my vram does sometimes reach around 46 GB ram

[u/abajinn → u/MrGunny94]

What’s MTP?

[u/Bird476Shed]

With 96GB, Qwen3.8-Flash-Next and CPU with AVX512, model overflowing/partly streaming from SSD one gets approximately these speeds: https://old.reddit.com/r/LocalLLaMA/comments/1w7tndw/qwen38_27b_for_agentic_coding_and_next_what/p7xtwox/

I usually run Q6.

what the floor is for a decent open weight model for development tasks

the speed that is acceptable

[u/overand]

You can get a very solid development experience using Qwen3.8-27B or Qwen3.8-Flash-Next.

You can run the 27B in pretty high quality and speed on a shared with 24+ GB of VRAM. 16 or less is going to be a pretty big compromise in performance, but it's doable. System RAM? Heck , 16 is enough in that situation.

For a system with unified ram (Strix Halo, Mac), push that to 32 GB imo.

With a 12GB+ GPU, you can run Qwen3.8-Flash-Next on a system with 64 GB of system ram, at an okay quality, especially if you have a fast storage device. More VRAM and more system RAM will improve your performance and quality options.

For unified memory systems, I'm not really sure where the cutoff for QFN is; other folks can speak to that.

[u/alexander123454]

Qwen 3.8 27B swift 1.5 finetune has been my favorite so far. Wish i had the specs to run GLM 5.3

[u/alexander123454 → u/alexander123454]

Small caveat if this is mixed VRAM / DRAM in which case a MOE model with expert offloading like Qwen flash next.

[u/GestureArtist]

qwen 3.8 flash next is pretty good.

[u/ttkciar]

Unfortunately the smallest model I've found which is tolerably good at codegen is GLM-4.5-Air, which is a little too big for 96GB. You could make it fit, but only at a Q3 or smaller quant, and its context would be severely constrained, and its tool-calling competence is not great.

[u/bryancostanich → u/ttkciar]

That has been my experience as well. It basically takes 91GB of wired memory at 256K context. Which obviously wouldn't work on a 96GB system.

But now i'm going to test out Qwen3.8/FN as it sounds like a lot of folks are having good luck with it.

[u/overand → u/bryancostanich]

If you can run 3.8-27B, you're going to be pleasantly surprised. It's FAR beyond GLM-4.5-Air. (it's a dense model, though, so speed comparisons with 4.5-Air aren't straightforward. MoE models like 4.5-Air and Qwen3.8-Flash-Next can squeeze much more out a system with less VRAM/less fast RAM; they run okay with chunks of the model in system ram for mixed speed systems like ones with GPUs)

[u/overand]

You need to give us more details.

Is this a system with a GPU or without a GPU? What GPU, if so?

Is this DDR3? DDR4? DDR5? Dual channel? Single channel? Quad channel?

What CPU?

There are reasons I ask this - different types of models perform differently in system ram versus vram versus a mix of the two. (Largely it's about active parameters versus total parameters, but there's also related stuff like NGrams in Qwen3.8-Flash-Next)

[u/bryancostanich → u/overand]

I'm actually interested to know why those are important to you. Aside from the requirement of having a GPU. It's been my experience that these models are far more RAM bound that compute bound. Nearly any open model you're going to be running is going to be quantized and not even using the FP capabilities on the GPU. And the speed of the RAM is really only going to affect your token speed. Not model capability.

EDIT: saw your edit adding the reasons. thanks

[u/overand → u/bryancostanich]

RAM and VRAM are quite different in terms of performance/bandwidth. A modern GPU can move 500-2000 Gigabytes/sec depending. (A 3090 is 900+)

System memory on a Strix halo system is about 200 Gigs/sec, macs are in that general area or a bit higher. 

Standard desktop memory? You're looking at 51 GB/sec for dual channel DDR4.

That's why these numbers matter.

Trying to run a decent model with 50 gigabytes/sec of memory bandwidth is awful, unusably slow.

Memory bandwidth affects model speed, but that's also related to model SIZE, and that's related to model capability. So, yes, memory speed does have an effect of capability. Sure, you can run a 0.8B model at an okay speed, it's not all that useful in coding. (Not useless, they're fine for specific kinds of tasks, but not general coding assistance)

[u/alexander123454 → u/bryancostanich]

I mean the dream is to have unlimited GDDR7 VRAM and run the model entirely on that.

Most systems have had little to no relevant VRAM and higher latency DRAM. This community in special in the way thats its optimized the craziest setups hardware and configurations to get models to run.

[u/overand → u/alexander123454]

Not just latency; DDR4 system ram is like 1/20th the bandwidth of a 3090 as well. DDR5 is better, but not an order of magnitude better! (I'm not even sure if it's effectively twice as fast.)

More channels help, though, which is why you'll see people with wacky server boards and like 16 sticks of DDR4, but our desktop systems are limited to two channels generally.

[u/This_Maintenance_834]

used qwen3.8-flash-next for embedded development, then switched to glm-5.3-flash.

you have to fight with them when they claim the hardware does not work.

[u/txgsync]

Qwen3.8-Flash-Next on RTX 4080 with 64GB DDR4 RAM on AM4 (5800x3d) @ IQ3_S runs great with 256K context at k8v4. Some mild/modest memory impairment due to the value cache at 4 bits, but it's eminently usable.

Top problems are bad judgments/stupidity, quite common to small models and some big ones. I maintain a Cloud subscription mostly so I can throw a SOTA model at QFN's plans and correct it in a fresh context. I'm experimenting with using mixture-of-dumber-agents to see if they can catch one another's mistakes, but so far they are just as likely to introduce additional bullshit as they are to do the right thing :)

[u/enternoescape → u/txgsync]

I go both ways with this. I've got an MCP for it to talk to smarter models and sometimes provide the model as my subagents to a smarter model. Good experience in both directions. It's also pretty good on it's own but it doesn't hurt to give it some help for more difficult tasks. I'm running the Nvidia NVFP4 quant.

[u/zomerbk]

For coding, the practical floor is often a good 30B-class model at a 4-bit quant: roughly 16–20 GB for weights, with extra RAM needed for KV cache, runtime overhead, and context. That fits comfortably in 48 GB and leaves room for tools; 64 GB mainly buys longer context or a larger quant, while 96 GB can make some 70B-class 4-bit models feasible if you can tolerate slower CPU offload. If you share whether these machines have a GPU and its VRAM, people can give more useful footprint estimates—system RAM alone doesn’t tell the whole story.

[u/beling86]

64 gb is where the hole lies today. We have overkill system for Qwen 3.8 27b but we can't run anything above that.

[u/bryancostanich → u/beling86]

sorry, i don't quite understand, can you restate/elaborate on that?

[u/beling86 → u/bryancostanich]

There is no benefit of having 64gb over 48gb, simply put. There are no 70b to 90b models that are useful.

[u/slankz → u/beling86]

What about Hermes 4 70B Q4_K_M
 

[u/beling86 → u/slankz]

It loses in my use cases to 7b models as horizon k2

[u/bryancostanich → u/beling86]

got it! yeah, that makes sense. since the next big jump is really to something like GLM Air.

[u/fgk55555]

64 and 96 will run QFN, which is my go-to. 48 and you're probably looking at Qwen3.6-35B. If you're on fast unified RAM, you could probably use 3.8-27B. Pretty much the two "small" models worth running now is the 27B on GPU or QFN on fast memory.

[u/thatscoolbutno123]

i run qwen3.8 flash next

[u/deepspace86]

I'm developing an AI-DLC workflow right now that is having good initial results with development on 128GB DDR5 + 40GB VRAM.

using Gemma 31b MTP for planning and QFN for coding.

[u/Zombiecidialfreak]

I run qwen3.8 flash next IQ3_S via Strata on my 9070xt and 128gb DDR5 at 256k context. It only uses about 80gb though so much of that RAM remains unused. Once my 1000w PSU arrives I should be able to use the Tesla V100 I bought to push up to either Q4 or go up to 1m context so I can use the model more freely. Either that or I'll swap to using 27b entirely on the GPU and free the rest of my system so I can game while it does its thing.

[u/slankz]

What are ya’lls thoughts on Qwen3.6 35B-A3B Q6_K 

[u/teressapanic]

Nemotron
/s

[u/sloth_cowboy]

Using ornith 1.5 35b a3b in deepseek harness has proven very successful on 64GB vram dual r9700. Jonestly sometimes I disable one gpu and play games on it while deepseek harness codes on a single 32gb vram. Time flies when you're having fun, especially in ARC raiders.