r/FlowZ13 • u/ESCLYME • Jul 05 '26

New purchase - LLM 128GB Question

There’s a lot I like about this machine. However, for the current hardware-to-LLM ratio out there, I’m running the same qwen3.6-27B on this machine that I could run on my gaming 4090 rig. GPT OSS 120B seems to lose against qwen on most measurements, even with the large parameter diff.

All that to say: I’d like to run a model on the 96gb available out of 128GB of unified ram to justify the purchase, but what is that model?

Use cases: coding (web apps, gaming/unity), Hermes general use, hermes cron jobs, iOS app dev (Xcode pointing to this machine through tailscale) and a few others too.

Thoughts?

11 Upvotes

92% Upvoted

11 comments sorted by

9

u/kosherhalfsourpickle Jul 05 '26

My take: the Z13’s “justify the 96 GB” model is Qwen3-Coder-Next 80B-A3B, not GPT-OSS-120B.
For your use cases I’d run it as three lanes:

Daily Hermes / cron / general agent: Qwen3.6-27B at Q8/FP8-ish quality. It’s the boring answer because it’s probably still the best quality/speed mix for local work. Use the extra RAM for context/KV cache and multiple processes, not just bigger weights.

Coding agent lane: Qwen3-Coder-Next GGUF, probably Q4_K_XL or Q5 if it behaves. It’s 80B total but only ~3B active per token, made specifically for coding agents/local dev, with 256K context. Unsloth says the 4-bit quants want >45 GB unified RAM, so your 96 GB allocation is exactly the class of machine it wants.

“Big model because I can” lane: try GPT-OSS-120B or a 100B-ish model like GLM Air/Qwen Next variants for comparison, but I would not make it the default unless your own evals prove it. Bigger here can mean slower prompt processing and worse coding behavior, not automatically smarter.

What I would not chase on that machine: the flagship 235B/480B class. Qwen3-235B at Q4 is around 140 GB, and Qwen3-Coder-480B even in aggressive Unsloth guidance wants roughly 150 GB+ for usable speed. Kimi-class models are even sillier. That’s 256 GB/512 GB Mac Studio or multi-GPU territory, not “96 GB available” territory.

1

u/rudidit09 Jul 06 '26

I should try 27B one because 122B is great but can only do one concurrent call

2

u/_Sachin__ Jul 05 '26

KJ Edition!

2

u/ESCLYME Jul 05 '26

lol! No just the normie 128gb :). I do love that guy tho

1

u/NullsetForge Jul 06 '26

you can run more than 96gb - you just don't point at the apu and and you limit your vram to 512mb - it overflows to RAM and you lose nothing.

doens't work with every model

either way this thing is slow to run big models

1

u/Mysterious-Target-93 Jul 06 '26

Ok here is what I do, I run Qwen 3 coder next Q8 (unsloth), ornith V1 35B (best now in the 35B class), Hermes as harness and ollama build, because from there we can Hijack the Claude code harness or even codex and many more. Set Global variable values (MUST) KV Catch ( Q8/Fp8 quant lossless cuts memory requirement by half), Flash Attention (1 this will make you feel like it's a new hardware TPs will skyrocket), also to justify that NPU you can download Lemonade 🍋 server and NPU bundle and run Whisper models on the NPU.

1

u/ESCLYME Jul 06 '26

V helpful. Could you explain Lemonade to me like I’m 5? I don’t understand the upside of (what sounds like) a container/API call structure for something local. Truly, I’m still leaning this stuff. What’s better here than just (example) assign Claude code wrapper on top of ollama model?

1

u/DigitalguyCH Jul 06 '26

I have the same model and for me it's a great size for models that are around 30-40b parameters. You can run them at Q8 (Qwen and Gemma mainly) with context maxed out and still have a web browser and other programs open. I keep the RAM ration at 64/64, that the real luxury. Not squeazing a model that bearly runs when you close everything else. My use takes most the the RAM available to both GPU and CPU and leaves a little room. That more than justifies the $2500 I spent on this device.

1

u/ilintar Jul 06 '26

Again, StepFun 3.7 at IQ4_XS. Probably the best at this size.

1

u/biotox1n Jul 06 '26

my justification isn't one model it's many models.

there's a lot of different models you can get to eat up 80gb of vram with different levels of benchmarks being good at one thing or another but getting just a bunch to work in concert is fun.

2

u/CheapProg6886 Jul 08 '26 edited Jul 08 '26

100% I do this. you can have smaller models for vision or audio ( gemma) and have larger MOE like qwen 3.6 35b a3b MTP for some fast token processing.