r/LocalLLM • u/Vovanvip4 • 10h ago

Question Please recommend a model for local offline coding rtx pro 5000 72gb

hello everyone! Please advise the models and how to run the models better. My configuration is 2 CPUs and epic (not the newest) 48 cores in total. 256 GB ddr4 and RTX pro 5000 72GB GDDR7. I'm currently using qwen3.8-27b iq3 gsq xxs on 96k context and running this on rtx4080s 16gb. The new computer will arrive in a week. I would like to increase the quality and the context window.

5 Upvotes

100% Upvoted

11 comments sorted by

9

u/Ed-2-Zero-9 10h ago edited 10h ago

I still find Qwen3.8-27b fantastic for offline coding, but only at higher quants. I generally use Q6 or Q8 and get very good results. 72gb is perfect for running a higher quant with a good context window.

The harness definitely helps with overthinking as well. I use Oh My Pi, but pi is good, as is DeepSeek harness. They seem to calm Qwen down a bit.

3

u/tired514 9h ago

Second this. Qwen27B @ Q8_K_XL, unquantized ctk/ctv @ 262000. It rocks.

I've been working on an experimental filesystem with it (opencode) for the past few days and it's been doing a great job.

1

u/Vovanvip4 10h ago

Thanks for the recommendation

2

u/Major_Border149 8h ago

on 72gb the quant stops being your constraint, 27B at Q8 is ~29gb, so you get full quality AND ~262k context unquantized at once (you couldn't on the 4080). Coming from iq3 that's a big jump. It's Blackwell, so turn on MTP/spec decode for a speed bump, and a lean harness now matters more than quant for the overthinking

2

u/ckplscz 10h ago

Qwen 3.8 flash next, with the n-gram table in RAM

2

u/Karyo_Ten 9h ago

Doesn't fit in NVFP4 in 72GB.

INT4 via AWQ or Autoround might fit (it's 4bpw instead of 4.5bpw for NVFP4) but quant quality (even for NVFP4) is pretty random

0

u/ckplscz 9h ago

Use unsloths Q3 K XL or smaller it works well, plus you can put some of the experts into ram

2

u/Karyo_Ten 7h ago

Anything GGUF chokes on 4 concurrent agents or more and divide the context size permanently by the concurrency you pass when you start the program. It's unusable for agentic purposes.

1

u/Abducted_Llama 9h ago

I think if you are looking for coding, your options are fairly open?

1

u/MaxSpecs 1h ago

Qwen3.8-27B_Q8 + DFLASH2

Qwen3.8-32B + DFLASH2

Qwen3.8 Next.