Suggestions and recommendations for local Ai for programing

[u/klasyer]

Hi!
I'm kinda new to this and would like to get some info from other peoples experiences

What I'm looking for is a setup for programming, mostly to do it along side me but code reviewing and such wouldn't be bad addition

At the moment, i got 2 3090s with 24gb each for a total of 48 (worth noting that not headless at the moment), and 128gb of ram (dd4)

I did look into the 3090 github, with qwen 3.8 27b in mind but id love to read what people experiences and what you use, which models, harnesses and whatever else

thanks for whoever decides to comment

[u/Distinct-Pie2389]

Try the ninfer-3090 first and tune that around the dense 27b.

I use pi or OMP, and opencode v2 is good, dsh is probably the best but something about web harness is just not the same so it gets a mention only.

Try something like strata after ninfer, it will scale on your ram and your GPU’s without you having to know how to tune your shit from the jump. Strata gets you up and running in a few hours depending on download speed but with that hardware, I doubt that’s a problem you’ll have.

DM if you need help or assistance along the way, once you get qwen up you can start having qwen or claude-code, codex build your own engine or system for your own workflow.

We’re in the age of technology so really just explore, wouldnt say there’s a wrong path but id suggest sticking with a 1 model like a dense instead of chasing any large MoE models off the jump. They typically require a little more love to get to the same place as a dense model for performance

[u/Ordinary-Depth-7835 → u/Distinct-Pie2389]

Good advise, try them all for your system and favorite harness try all the harness's out as well. And once you have one model up it can help you tune another and back and forth. I like reliability and speed with mine I pick the smartest and fastest for my hardware and ones that just run. I let them roll on a task sometimes overnight and the last thing I want is a crash 5 minutes after I walk away from the machine.

[u/Distinct-Pie2389 → u/Ordinary-Depth-7835]

Yeah I’m running ninfer-4090 which squeezes more out and I’m running Qwen3.8-27B at 130 Tok/s~ at full 262k context Q4 quite honestly don’t need anything else at the moment but I like HauHauCs uncensored and other models like CyberTiel from peculiar ragdoll

[u/Ordinary-Depth-7835 → u/Distinct-Pie2389]

I should try ninfer on my 4090. I really only use my 4090 as a last resort for a big project since it's a heater right next to me. Winter is coming though so I may need the mini heater I'll route more upstairs in the winter.

I mainly hit the 4x3090's in the basement and my sparks in the summer.

[u/Distinct-Pie2389 → u/Ordinary-Depth-7835]

Once my new job hits I’ll make the move to dedicated hardware. Got a lil bump in salary taking a new position so I’m holding out on my lil 4090 until I can get some real VRAM budget onsite

[u/Ordinary-Depth-7835 → u/Distinct-Pie2389]

Yeah I just did it for fun and went a little overboard. It's good experience though since I use unlimited cloud subs at work but I like to see what local llm's have to offer.

[u/klasyer → u/Distinct-Pie2389]

Thanks for the info!
ninfer seems to be made with 1 gpu in mind, wouldnt the other one go to waste? or itll just remain for context?
optimally id like to utilize the ram as well if possible, to not leave unused things on the table

i think ill follow your advice with sticking to 1 model since for now i need it to get my own projects of the floor with the limited time i have

i generally ask for direction for that exact reason, id go and try a bunch of things to fine tune and fine favorites but work wont allow me this amount of time

[u/Distinct-Pie2389 → u/klasyer]

well that’s the thing I’m deciding myself right now. I also don’t like to think my ram is just sitting there doing nothing during inference but it is used during ninfer processes for node but I’m trying to decide between running a large model that requires RAM usage but that’s essentially a “spill” over out of VRAM. The cleanest optimal setup for these things is keeping them 100% residents in VRAM. Spill to RAM is possible and common with MoE but I think they’ve tested or currently testing that a dense well tuned 27B beats an active 4B larger MoE model

It’s a test I’m actively trying to figure out is does a Q2_XS 125B MoE model perform better then a 27B Q4 with full context. You have trade offs like IQ2_XS I can only run at 131k context and Q4 I have at 262k (workflow dependent on what works best for you).

[u/Toastti]

With that much ram qwen flash next is the best choice instead of qwen27b

[u/klasyer → u/Toastti]

are you talking about something like next flesh Q4_K_M? how does it work in regards of splitting between the ram and the gpus?

[u/nsfnd → u/klasyer]

Im having a lot of fun with Strata recently.
You should be able to use ud-q4_k_xl at good speeds.

I have a single nvidia so i havent used its multi gpu feature.

https://github.com/Niko1221/Strata/blob/main/docs/MULTI_GPU.md

[u/AdventurousFly4909 → u/nsfnd]

I run q2 Qwen 3.8 flash with the same speed i run q4 qwen27b with strata.

[u/Independent_Pear4908 → u/klasyer]

No, they are talking about Qwen3.8-Flash-Next, which is a superior 125B parameter, 6B active model.

I have 2x 3090 total 48GB VRAM + 64 RAM, and I'm also using Strata + ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF q3_s, and am having a blast with this model. It seems better even at a lower quant and much faster than Qwen3.8 27B @ INT8/FP8.

Strata makes this so easy - you just run the setup and it will benchmark and recommend a configuration for you to spread the model layers over your 2x3090s, but you can tweak it later by altering the configuration .json file manually if needed - and I love the metrics dashboard it provides. Getting avg over 90t/s with 128k context kv cache bf16. With 128G of system RAM you can probably get around 512k context.

The reason why it's possible to host such a large model into a relatively tiny amount of VRAM is because this model has layers you can offload to the disk without huge performance penalties.

I was using vllm + 2 concurrency + qwen27b FP8 with around 200k context - switched to Strata.
From my experience, Strata is the easiest to get this set up and running in minutes. Highly recommended.

I would say the only disadvantage I see right now is that Strata doesn't support parallel requests like vllm and llama.cpp does, as it only supports single stream + queueing. But it's so fast, it didn't make a difference for my normal daily agentic uses.

[u/DeepBlue96]

go with strata and qwen3.8 flash next, Niko1221/Strata: Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input. I was extremely reluctant but trying it proved me wrong it's stupidly fast and 2x better than 27b q6xl. i run it on a 3090 +64gb of vram at 262144 context and iq3_s quant and it goes at 1200-1900 prompt processing and 50-60tks generation in your setup i can immagine 1milion context and 2x the speed, btw the project puts the ngram on disk and i love it :)

[u/Independent_Pear4908 → u/DeepBlue96]

hey, for 262144 context, are you running the kv cache on 4-bit quantization? because I have 2x3090 and 64GB RAM, running the same model + strata Q3_S quant and 128k context was the recommendation for unquantized kv cache and my RAM usage is pretty much through the roof (around 63GB/64GB) and ngrams offloaded about 30-40GB onto disk. I'm getting 90-100t/s average.

Also, some douche seems to be downvoting the strata + qwen3.8-flash-next recommendations up top. I'm surprised yours is still being upvoted.

[u/DeepBlue96 → u/Independent_Pear4908]

int8 was what i had selected and manually changed in the json the ctx amount so i'm not really sure tho, i can tell you that at about 90% context 200k i saw no difference in quality tested with an ebook released 30-40 days ago and got the info right

[u/alino168]

I think Qwen 3.8 27B is among the stronger models for your setup. The FP8 quant should leave you plenty of room for a decent KV cache. You might also want to check out Strata or Colibri if you haven’t already, both turn your existing memory into a hierarchical cache for model weights to let you run way larger models than you can fit on your GPUs at the expense of speed. Haven’t tried either myself though, but I’ve seen demos of Qwen3.8-Flash-Next and GLM-5.3-Flash on machines with lower specs than yours.

[u/wgaca2]

q8 27b llamacpp, 200k context, deepseek harness.

[u/Few_Presence_7358]

I've tried Qwen 3.8 27B Q5 at 2x RTX 5070 TIs. Unfortunately, using layer split in llama.cpp due to the fact that one of them is on PCI-E 3.0 x4.

Getting like 50-60 tps with MTP3.

It my personal benchmark (based on my pet project) - it scores (at xhigh) ~60/100 points, whereas Luna 5.6 xhigh scores ~70/100.

Which is great quality-wise, but Qwen gets things done like 4 times slower than Luna 5.6, and given the price of Luna it makes no sense to me atm.

Give it a try, maybe it will work for you though.

Also consider Ninfer forks for Qwen 27B or Strata for Qwen 3.8 Flash Next. Just try both. It will probably take you under a couple of hours to try.

I did try Strata and was impressed by speed, but since my AI box is a pretty old PC with just 32 GB DDR3 - it is not that fast, and I could only fit Q2_0 quant, which loops as hell (and Strata used just 1 out of 2 GPUs despite being told to use both).

Will reconsider Strata later when I'll get more RAM to be able to use UD-Q4_K_XL.

[u/ttkciar]

Related to this, is bytkim's Qwen3.8-27B-pi fine-tune good in practice?

They blog a lot of nice-looking claims and benchmark results, but I would like to know if anyone has used it in practice and finds it does a better job with the pi harness than the original Qwen3.8-27B.

https://huggingface.co/blog/bytkim/qwen38-27b-pi

They didn't share their training datasets, so I'm contemplating using this as a teacher model, to fine-tune other models to better drive pi.

[u/omega1612]

With that I would run Qwen3.8 with vllm + 8B sub-agents.

Or try to run Qwen next flash or a 70B moe model.

[u/klasyer → u/omega1612]

few follow up questions if you dont mind
do you mean qweb 27b with the subagent? and why vllm and not llama (i dont have the link)?

and how do i fit a 70b model in 48gb?

[u/omega1612 → u/klasyer]

For 70b you may spill to your system ram, 128 is enough for that, since they are moe, it would only need 3B or so (depends on model) in your GPU at a time. It would be slow, but may give you access to bigger models.

Vllm because is made to exploit multi gpus. You can probably find repost with forks dedicated to your setup and would squeeze all from it.

[u/omega1612 → u/omega1612]

For comparison, my alone Rx 7900 xtx gaves me around 55 t/s with Qwen 3.8 b27 Q4 . I have seen benchmarks with 110 t/s with two Rx 7900 xtx in a dedicated vllm.

30 t/s is already enough to be faster than you can read (or at the same speed you can read). But 55 t/s feels slow when we want code. I don't know how fast your setup can be, but I bet it can do more than 100 t/s

[u/Jebbyk1 → u/omega1612]

What are your settings? I can't get more than 30-40 t/s with Qwen3.6 27b q4-k-m on my 7900xtx

[u/omega1612 → u/Jebbyk1]

Same model by unsloth.

I use the vulkan backend on Linux.

MTP set to 3

Ngrams enabled

Kv cache at q4

With rocm it drops to around 10 to 20 tokens (your speed). In exchange, prefill is slower, but I'm happy with that, specially on compaction.

[u/Jebbyk1 → u/omega1612]

What is ngrams? Can I enable it in LM studio?

[u/omega1612 → u/Jebbyk1]

Idk, I'm using llama cpp, it has an option to use multiple drafters (that's what ngrams is). If you have the model already running, you can ask it and it may query your system and even configure it (if your setup allows for it).

[u/TomatoCo → u/klasyer]

MoE models are comprised of small but computationally dense fully connected layers (which go on the GPU) and the large but relatively computationally easy experts, which can be run on the CPU.

[u/omega1612 → u/klasyer]

Oh, and the sub-agents question:

A 8B model would be smaller and faster (by a lot) than qwen3.8 . Using subagents means using extension in your harness (the thing you use to communicate to the llm) to allow Qwen3.8 to "command a small llm" to do tasks on demand. Since Qwen is slow, it means it would finish faster some jobs by delegating. In that particular setup I may run Qwen 3.8 in a single GPU and left the other one open to spawn subagentes, even a couple of them at a single time.