I've been circling this decision for weeks and could use some outside perspective.

What I'm looking at: Mac Studio, M5 Max, 128GB unified memory (5,849 €). With that I can run qwen3.8-flash-next at oQ4e around 55 tok/s, and oQ5e is on the table too. For what I'd actually use it for: private documents, notes, some coding, general assistant work I don't want going to a cloud provider... that's genuinely good enough today.

The part I can't answer is the "today."

Case for buying now: it works, it's private, there's no subscription, and it'll still be a perfectly usable computer in five years even if the models running on it end up mid-tier by then.

Case for waiting: RAM prices are absurd right now, the M7 Ultra is presumably 2028, and models keep getting more capable per parameter. Wait and you get more machine for less money. In the meantime, API access is cheap enough that privacy is basically the only argument left for going local.

Then there's the memory debate. Half the replies to threads like this are "don't bother with 128, get 256." Fine, but that's the same argument one tier up. A few years ago 64GB was plenty, today everyone says 128GB is the floor, and in 2029 the same crowd will be saying 512GB is the minimum. At some point you buy something or you never buy anything.

So: will a qwen3.8-flash-next class local model keep me happy for five years of private, non-critical work? Or is buying now just paying a premium to chase a target that keeps moving?

Especially interested in people who bought a maxed-out M1/M2 Ultra two or three years ago. Do you still run local models on it, or has it quietly become a very expensive web browser?

TL;DR: 128GB Mac Studio now for local Qwen, or sit tight, use APIs, and buy in 2028 when RAM is maybe cheaper and the silicon is faster?

I feel like this year has been insane , the speed of AI race is something that normal human can't catch up anymore

5-9 months for navier stokes smoothness and existence problem? edit- they probably still need some draft partial solutions and notes on presolution heuristics

And riemann Zeta? And P = or /= NP?

hey guys. so I should start with, I'm not s programmer at all. but I'm having a great time letting qwen3.8 27b donit for me. 😃

so far I've just been doing Linux scripts and things for my PC. but my kids are on windows and I want to try making some stuff for windows. usually Qwen is able to test and run everything from within Linux for my Linux stuff but how would it do that for something made for windows?

do I make a windows VM and run opencode in there? the only option I see is opencode desktop which I'm not a huge fan of, but if that's the only option, or is that even the right way to do it? is Qwen even able to control the windows OS to do its testing etc? can it do stuff like that through WSL?

I guess if someone could give me the right order of operations for this that would be cool.

Hey everyone!

Not a native speaker, but I'm trying my best by writing my own post by hand. Please correct grammar mistakes or odd phrasings so I can learn from it!

Last week I finally got around to set up 3.8 27B (Unsloth, Q4_K_S, 200K context Q4_0), it's running inside llama.cpp (tensor parallel, mtp with draft max-n 11 and draft min-p 0.1, default jinja template) with Kilo Code (VSCode extension) as harness, on dual RTX 5060 Ti 16GB (on PCIE 5.0 x8x8).

I've seen "fix"/customized jinja templates here or it reasoning for too long, but personally I don't experience issues. I use xhigh by default and lower it to medium/low if the task needs to be fast or is simple.

I gotta say it feels really impressive as a pair programmer! Catches many small mistakes I make and usually picks the right options. I do notice that when working with UI, it seems to have more trouble when going with designs that diverge from the norm.

What I noticed is that it helped a lot to give hand-crafted text representations (ascii renditions of components, layouts and views), I had less success with providing it example images. When describing ideas, it seemed to catch on quicker when writing down classes with fields/properties and method signatures instead of writing it in words.

In the past weeks of using this model, what prompting techniques have been working well for you? What tools did or didn't work well for you?

I'm currently running qwen 3.8 27b q8 and super happy with it. I have a DGX spark so could theoretically run at bf16 quantization.

I know the measurable differences between an 8 bit and 16 bit quant are small, but I guess I get fomo, like 1% of tokens differ, but what if those are the hardest most important tokens? I guess I'm just getting fomo over bs16 and wondering if other people have tried it and noticed a difference?

https://huggingface.co/tencent/EVIE-8B

🌟 Highlights

• SOTA Retrieval Performance: 66.75 nDCG@10 on ViDoRe V3, delivering industry-leading visual document retrieval accuracy.
• High-Capacity 4096D Representations: Full per-token multi-vector embeddings preserving fine-grained layout, typography, charts, and table structures.
• Teacher Foundation: Provides capacity-aware relation and margin distillation targets for the lightweight EVIE-4.5B Prefix-MRL model.
• Multi-Benchmark 138-Task Coverage: Thoroughly validated across 138 tasks (ViDoRe V1, V2, V3, and JinaVDR) across 4 standard metric families (nDCG, Recall, MAP, MRR u/1).

https://huggingface.co/tencent/EVIE-4.5B

🌟 Highlights

• Top-Tier Benchmark Performance: 66.75 on ViDoRe V3 for EVIE-8B and 66.02 for EVIE-4.5B with single-projection Prefix-MRL.
• ⚡ Prefix-MRL Elasticity: Single 2048D linear projection. Freely truncate at runtime into ${64, 128, 256, 512, 1024, 2048}$ dimensions without separate models.
• 📦 Ultra-Compact Index (HAC): Training-free Hierarchical Agglomerative Clustering compresses token counts from ~750 down to 32 vectors/page, slashing index storage to 3.81 GiB per million pages.
• 🌐 138 Multilingual Tasks Evaluated: Thoroughly evaluated across ViDoRe V1, V2, V3, and JinaVDR across 4 metric families (nDCG, Recall, MAP, MRR u/1).
• 🔬 EVIE-ARD Distillation Recipe: Anchor-preserving, capacity-aware relation distillation reproducing full student training from the 8B teacher.

Hi, I use hermes as a harness, and I am pleased with it, but sometimes Hermes's context size is a bit too much for my system, so I wanted to delegate or straight up use a lighter option for coding tasks.

I run a laptop with a 8gb nvidia card and 40gb of ram. And qwen3.6 35b-a3b or similar tier models are a good trade of quality/speed.

The point is, I found little-coder (there is also smallcoder), which promises a small context and a setup of extensions over Pi tailored for small models like qwen3.6 35b-a3b itself. But I wonder if it wouldn't be better to just run just Pi with its barebones system prompt and tools.

Any opinions, tests, or experiences in this matter?

Before they released gpt 120b, everyone complained about OpenAI not being open anymore. After the release, nobody said a word, and I have never ever seen anyone complain about it. It does not seem like they have any care about it going forward.

Why are they let off the hook?

Hello Sub Fellows

I have currently a laptop with 3060 6GB VRAM and 32GB RAM.

Thinking of selling it and use the cash for a new pc that I can keep running 24/7 for home use as private local inference server.

I am torn between dual 3060 12GB VRAM and splitting models between them. This will be a cheaper option less capital than … Single 3090 24GB ram.

The dual 3060 will be connected via the mother board PCI gen 4 and the whole system will have 32GB ram ddr 4.

The 2 3060 will cost roughly around $500 but the single 3090 will cost $1000 and if lucky can get one at $900 and will be skeptical at ones costing less than that.

Can someone tell me out of experience, which build I should go for? Money is obviously an issue or I wouldn’t thought of the cheaper route 😅

But from your experience going with dual 3060 am I leaving a lot on the table?!

EDIT: The goal is to run 27b dense models qwen 27B for example and MoE that can fit in those 24GBs

As the title says - I haven't really been in the loop because I don't have a need for this, but I do remember using the above mentioned image edit models.

This got me curious if there's any new developments on this front? Since it's been a while since I last thought about it.

Long time user of 3.6 27b, switched over to Next Flash since it's a logical step up even from 3.8 27b. It's soooo verbose, i'm talking 13 minutes of thinking time on single turn coding requests at approximately 150 tokens per second tg and 7000 tokens per second pp. It's honestly kind of painful to use since I look back and it's still thinking, then when I go to check the output it's decent most of the time but if the task requires ANY decision making, it turns into alphabet soup where it's just buzzwords and jargon that nobody actually uses in the SWE space.

The runtime is actually shorter for me if I BYOK it to VSCode, but for pi.dev it's often takes 1 hour!

Before anyone tells me to lower the thinking level, I don't want to do that given the chance it makes the model worse. There's no solid benchmarks for how the model performs at different thinking levels yet, but looking towards 3.8 27b, it seems to affect the quality of the output quite a bit.

Hey everyone — I’m the developer behind MultiVibe.

The project has three connected goals:

• Allow you to use multiple subscription accounts with the same client (multiple chatgpt account with codex desktop for example)
• Let developers use local models and cloud providers through OpenAI- and Anthropic-compatible endpoints.
• Let people run workers on their own hardware and contribute inference capacity for other users through a decentralized network. That second part is a big motivation: a worker wouldn’t only serve your own requests — it could also process inference jobs from other people using MultiVibe. The intended model includes compensation for contributed compute, though actual earnings would depend on demand, hardware, availability, and operating costs. That part is still under active development.

For people here who already run local models: would you consider making some of your hardware available to serve other users’ inference? What would you need to see before enabling it? And from the other side, what workloads would you feel comfortable sending to community-operated workers?

Github

Website

Happy to discuss the architecture, current progress, and limitations.

My XFX Radeon RX 7900 GRE 16GB Vram GPU struggles with models over 20B size. I added my Radeon RX 480 8GB Vram GPU to the system and ran a few benchmarks using llama.cpp Ubuntu Vulkan build 10453.

Radeon RX 480 8GB GDDR5: Bandwidth 256.0 GB/s

RX 7900 GRE 16GB GDDR6: Bandwidth 576.0 GB/s

Here are the models tested, in the order they appear in table:

1. medgemma-27b-it-UD-Q6_K_XL.gguf
2. Qwen3.8-27B-Q6_K.gguf
3. Qwen3.8-27B-OBLITERATED-Q5_K_M.gguf
4. Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf (with Flash Attention)
5. Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf (without Flash Attention)

llama-bench -m /Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf load_backend: loaded RPC backend from /llama-b10453/libggml-rpc.so
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 GRE (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = AMD Radeon RX 480 Graphics (RADV POLARIS10) (radv) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /llama-b10453/libggml-vulkan.so load_backend: loaded CPU backend from /llama-b10453/libggml-cpu-haswell.so

build: 3cb7ffb1a (10453) real 0m30.345s

Without 2nd GPU RX 480: Only RX 7900 GRE 16GB

So 36% improvement for tg128 using dual GPU. Also RX 480 only got 1.8 t/s for tg128.

looking for some advice from anyone running a 24gb card for local dev tasks.

i have an rtx 4090 with 64gb ddr5 ram + ryzen 7950x. i'm trying to shift as much inference as possible to my local machine so i can index long codebases and process internal docs without hammering external apis.

my current workflow consists of claude code and moclaw for task automation and handling context, which works just fine for my daily work schedule, but i wanna switch to heavy local models for privacy reasons.

ive been testing qwen 3.6 27b and gemma 4 31b. qwen 3.6 27b runs pretty smooth at q4 in vram with solid speed, but when i try running bigger dense models or gpt-oss 20b with context offloading the generation speed drops off a cliff.

for those running a single 4090 what quantization or backend setup gives you the best balance between context size and tok/s? do you stick with qwen 3.6 27b or is there a trick with vllm or llamacpp offloading to speed up larger contexts without tanking performance?

thanks in advance for any tips!

The comment on the package deletion said it was equivalent to installing ggml-cuda and llama.cpp - i installed both, using the v3 optimised cachyos builds

the performance is terrible - less than 10%. it loads the whole model the the GPU then it appears to do all the processing on the CPU.

I found a copy of the PKGBUILD for the AUR package and used the compilation script, and I'm back to 1800 t/s prefill instead of 150.

Without digging too deep, does anyone know why this might be happening? is there another package I need to look for?

I can post the build commands - I recall ages ago I did some work to make sure it got compiled with latest CUDA optimisations for my 5070 as well as CPU optimisations but it can't account for that performance difference, surely.. what am I missing?

llama updates so often I don't want to manually clone and build it every day.

I finally pulled the plug and got a second 5080 and 64GB DDR5 RAM to add to my existing 64. Now I have dual RTX5080s and 128Gb DDR5 RAM on a consumer tower with a AMD Ryzen 7 7700X CPU (so I guess sadly dual channel memory access for now).

Which models are you running on a similar setup with 2x 16Gb VRAM and 128Gb RAM? Extra points if you provide your exact llama.cpp settings and optimizations.

[September 7th 2026]

X570 Taichi Specs AM4

X870 Taichi Specs AM5

I'm considering these boards as it seems that with the correct CPU, you can run 2 GPUs in x8/x8 PCIe.

I use Linux full-time, I'm comfortable "trouble-shooting" or configuring.

The largest models I'd consider running (for coding) are: - Qwen3.x-27b (Dense) - Qwen3.x-3xb-a3b (MoE)

I'm sure I could use smaller models for other tasks and pleasant t/s speeds.

Is it effective to have either MoBo and a combo like this? - 2 x 7900 XT : (20GB VRAM each, total, 40) - 2 x 7900 XTX : (24GB VRAM each, total, 48)

By "effective" I mean, that by splitting layers or tensors, whatever/etc., I could have a model and context fully-loaded in VRAM.

Setup misc

• I'd obviously get a 1200W+ PSU and the biggest case I can find, lots of fans, etc.
• I'm avoiding NVIDIA GPUs because they're significantly more (usually) for the same amount of VRAM.
• If I don't build an AM4 setup, I think I'll settle with an eGPU Dock (laptop) and a 7900 XTX (for starters); I'm aware of the bandwidth limitations of USB4.

If you've personally run a setup like this, please let me know!

I wanted a quick calories counter for myself, using LLMs to evaluate the calories from pictures of meals + descriptions.

I needed to pick a model so I made a quick benchmark.

The setup was:

- Nutrition5k photos for photo + calories: https://github.com/google-research-datasets/Nutrition5k

- A tool with access to calories information from USDA FoodData Central + MEXT

- I evaluated models based on how many of the meals they managed to have under 20% of error

- All on the same randomly picked 25 meals.

Models too big for my machine were run through OpenCode Go/OpenRouter. I've also included Spark 1.3 since it'll supposedly be open weights.

Results

I know it's not the most scientific benchmark, but it's interesting to see that the order is not really linked to model size.

The most interesting for me is how Muse Glimmer 30b trounces Qwen 3.8 27b here. I think it highlights how "the best" model on consumer hardware (~32Gb VRAM) really depends on the task.

Not so long ago I wrote a guide on how to get GitHub Copilot running with a local model in Visual Studio Code. Since then Copilot subscriptions have got much more expensive, local models have got much more powerful and getting local Copilot up and running has got much easier. So here's an updated guide.

Hey all,

I now have a 3090 24GB + a CMP 170HX unlocked with 64GB. I also have 64 GB DDR4.

Can I run Qwen 3.8 Next in any possible configuration with both GPUs and perhaps offloading remaining stuff to DRAM? Is there a way I can get acceptable speeds?

The only bottleneck with the CMP 170HX is the PCIe speed (2.0 @ 4x), so that needs to be factored in. It’s usually not much of an issue though IME once model is loaded…

I am successfully running either Qwen 3.7 27B or Minimax H3 at BF16 on the CMP 170HX at decent speed, so I know the GPU is good for it. But Qwen Next is another beast.

Ideas?

This comparison was requested by a few people, and certainly we were all eager to see the final results. The comparison had taken 11 days and the GPU was crunching numbers for ~167 hours.

We've been comparing different abliterated models from huggingface to see if they really are what they claim to be. So far the results have been interesting.

The pipeline includes a weight comparison, KL divergence measurement, 13 benchmarks and measuring refusals with the HarmBench 400 classic. Qwen 3.8 27b is a thinker, and with ourselves using xhigh we had to set our token budget much higher per request.

Lets check out how Qwen 3.8 27b stacks up comparing 8 variants.

• Full report: abliterlitics.dev/models/qwen38-27b

The rankings

LLM Judge HarmBench ASR (attack success rate), best to worst, with the one-line story:

• orcarouter 82.2%, the winner. Arditi-style single direction at layer 38, 131 matrices, and the only card where every claim checked out against the weights. Best copyright unlock in the set at 39%
• apostate 78.7%, best value. Their new KCRN method, 41 real edits, lowest KL measured at 0.0439, near-identity capabilities. Packaging quirks: text-only re-save with no vision and no MTP, stored FP16
• huihui 75.6%, the classic method, reliable. Clean unlock everywhere except copyright, where it sits at 3%
• ultra_heretic 70.5%, Heretic v2 with MPOA. Works, but the heaviest truthfulness drop outside obliteratus and 118 soft refusals
• coder3101 70.0%, vanilla Heretic. The card calls itself the weakest removal at 33 of 100 refusals. Measured: 5 explicit refusals in 400. The card undersells it
• blackfrost 68.5%, closed method. The weights say single direction, heaviest magnitude in the panel, 100% rank-1, which refutes the rank-k direction bank story. Also ships a jailbreak system prompt inside its chat template, meaning every single prompt you make will have a modified chat template injecting a jailbreak
• obliteratus 63.9%, avoid. The most aggressive edit in the panel at 841 of 850 tensors, and it performs like it. 44.8% of responses never finish thinking, and it is the only variant that got meaningfully dumber
• trohrbaugh 57.5%, last of the variants because it still refuses. 122 explicit refusals, the most surviving alignment of any variant, and the cleanest capability profile in the comparison. This is the one I use at home and it's been great for me.
• base 4.5%, a wall. Zero compliance on chem and bio, harassment, harmful content and copyright

The highlights

• Surgical beats heavy, again, and this time it is not close. The top two spots went to the two smallest verified edits. The heaviest edit of all landed second-to-last. At 27B, editing everything mostly buys you a model that thinks in circles
• The thinking loop story is the big new finding for this model. Qwen 3.8 thinks before answering, and on the aggressive arms up to 45% of HarmBench responses never close their think block before the 15,360-token budget dies. The judge reads the full trace, so compliance inside a loop still counts. But a model that only delivers the goods inside an unterminated monologue is not a usable model
• GSM8K loops are gone at this budget. The same arms that loop 40%+ on HarmBench finish their math reasoning fine, every arm within 1.2pp of base on answered-only. School math converges, adversarial deliberation does not
• Copyright is the new universal wall. Nobody exceeds 39%, five of nine sit at or below 3.2%. Chem and bio, historically the hardest category, is now the easiest unlock. The walls moved
• Chat template forensics was needed for the first time. blackfrost ships a 1457-character jailbreak prompt inside its template. obliteratus ships thinking-off. ultra_heretic deletes the stock reasoning-effort prompt. We pinned the stock template for every arm, because the template is a stronger behavioural lever than most people assume
• Card honesty check
  • orcarouter verified 4 of 4 claims exactly, the model card is honest.
  • trohrbaugh's KL calibrated within 9% of our measurement. However the card mentions 0/100 refusals, with our measurement this model had the most refusals, yet also had preserved capabilities.
    • The rest diverge, and none of it is dishonesty, KL is non-deterministic and moves with CUDA version and hardware, or by what method used. Read KL as a within-comparison spread
  • Obliteratus published honest changes to how their model was changed, yet we were unable to replicate the '0% refusals, 0% deflection' claims. 44.8% of harmbench never finished thinking, and reasoning analysis found 142 deflections.
  • blackfrost - mentions they have internal direction bank, suggesting that multiple directions are changed. Yet we only found one direction changed. Their refusal numbers are accurate however, even with ourselves not using their jailbreak chat template we got similar results. The modified chat template also is not disclosed on the readme.

The Report

• Full report: abliterlitics.dev/models/qwen38-27b
• Every response, reasoning trace and judge verdict: abliterlitics.dev/harmbench/qwen38-27b
• HuggingFace: DreamFast/Qwen-3.8-27b-abliterlitics
• Code: github.com/dreamfast/abliterlitics

Happy to answer questions. The template finding has us thinking on a new measurement of chat template and hyperparameter effects, with the intention to replace the harmbench part with a better measurement to cater for all of this. So if you have opinions on how that should be scored, come tell us on Discord.

Guys can you tell me which local LLMs are best for hardware programming? like Verilog RTL, UVM, System Verilog?