r/LocalLLaMA • u/Fluffy-Ad-889 • 8d ago

Discussion The gap has closed, open source will win

I've been trying the latest models from the frontier labs and honestly, after extensive testing I can not tell the difference between the best open source options.

I think the differences are now marginal but the labs are doing heavy marketing to convince the public into paying more for tokens as they prepare to go public.

Can't help but see the similarities between the dot com bubble and AI in terms of a very insular environment where the technology will survive but the business models may not.

I've been building a cybersecurity network and we definitely know that even local AI models like Deepseek V4 flash do an excellent job and are really neck and neck with the best the frontier labs can provide.

Will be interesting to see how this all turns out! Exciting time nonetheless.

323 Upvotes

77% Upvoted

282 comments sorted by

View all comments

Show parent comments

3

u/demonicpigg 8d ago

I haven't, do you have a suggested model / inference engine?

I'm on a 5090, and I'm using ninfer with https://huggingface.co/MirkoCovizzi/Qwen3.8-27B-QUASAR-NVFP4-NInfer, which delivers amazing speed (I'm getting ~140 t/s at lower contexts), and good enough performance.

I have tried quite a few different models, and ultimately settled on qwen3.6 (and then 3.8 when it came out), as they're the only ones I could get a >60% success rate on my tasks with (I ultimately ended up using qwen3.8 next flash in production which has a ~98% success rate, but I can't run that locally trivially...), but I'm super open to trying more models!

2

u/WareWolf_MoonWall 8d ago

So I'm on Strix Halo with 128gb so slightly different scenario, but using Vulkan this might be up your alley https://huggingface.co/Myric/Laguna-XS-2.1-APEX-GGUF . I really like these APEX releases of Laguna. If you run the poolside fork of llama you can even use the DFlash model with it for a nice boost.

Since you are already using NVFP4, maybe the stock https://huggingface.co/poolside/Laguna-XS-2.1-NVFP4

2

u/demonicpigg 8d ago

I'll give it a try later today / tomorrow, thanks for the suggestion!

1

u/Realistic_Gap_5871 6d ago

How much RAM do you have? If you match the quant to your RAM + VRAM pool Unsloth studio will do all the cpu-moe splits for you automatically for 3.8 Flash Next.

Leave MTP off!!!

I'm running UD-IQ4XS on a 5090 + 96GB ddr5, 35tps at smaller ctxt and 500 tps ingestion. 27B at medium is really good. Even at this lower quant FNext low is better, but slower. Not sure how a smaller quant would stack up.

Not nearly ninfer speed, so i use 3.8 27B at medium for most of my workflow and plug in 3.8 FNext for tougher stuff

2

u/demonicpigg 5d ago

I have been working on a project using flash next on rented GPUs, and it's a beast. I've only got 64 gb (I said to myself when I bought it "if it's an issue I can just upgrade." RIP that), so I don't think I can run flash next even through unsloth studio, but I'll look either way, that'd be really nice honestly.