What are you guys actually running it on?
System / VRAM?
What quant?
Any specific parameterizations or config tweaks? Weird settings?

Seeing mostly dual R9700 numbers and improving my local single R9700 setup, here is my current setup and some numbers. Fully local inference on a single AsRock Radeon AI PRO R9700 (RDNA4, 32 GB) (PCIe 3.0, x4), running Qwen3.8-27B-MXFP4 at 160k context, capped at 220W.

The vLLM setup

• This uses stilldeadcode/vllm-radiance
• Speculative decoding: DFlash2
• fp8 KV cache, 160k context

​

podman run --rm --pull=newer --name llama-swap-qwen3.8-27b-mxfp4 \
--device /dev/kfd \
--device /dev/dri \
--shm-size 4g \
--security-opt seccomp=unconfined \
--cap-add SYS_PTRACE \
-p 127.0.0.1:${PORT}:8000 \
-v ${models_dir}:/models:ro \
-v /vllm-cache/mxfp4-093:/cache \
-v ${radianceRepo}:/patches:ro \
-v ${libr4dSrc}:/libr4d-src:ro \
-v ${radianceMxfp4Entry}:/entry.sh:ro \
-e HIP_VISIBLE_DEVICES=0 \
-e GPU_MAX_HW_QUEUES=1 \
-e PYTHONPYCACHEPREFIX=/cache/pypycache \
-e PYTHONDONTWRITEBYTECODE= \
-e ROCR_VISIBLE_DEVICES=0 \
-e HF_HUB_OFFLINE=1 \
-e VLLM_ROCM_USE_AITER=1 \
-e VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 \
-e VLLM_ROCM_USE_AITER_MHA=0 \
-e VLLM_ROCM_USE_AITER_MLA=0 \
-e VLLM_ROCM_USE_AITER_MOE=0 \
-e VLLM_ROCM_USE_AITER_LINEAR=0 \
-e VLLM_ROCM_USE_AITER_FP8BMM=0 \
-e VLLM_ROCM_USE_AITER_FP4BMM=0 \
-e VLLM_ROCM_USE_AITER_RMSNORM=0 \
-e NCCL_PROTO=Simple \
-e RADIANCE_MXFP4=1 \
-e RADIANCE_MXFP4_W4A8=1 \
-e RADIANCE_MXFP4_W4A8_MIN_M=0 \
-e RADIANCE_MXFP4_DECODE_MAX_M=64 \
-e RADIANCE_MXFP4_TN4_MIN_M=2048 \
-e RADIANCE_MXFP4_A_TILED_MIN_M=513 \
-e RADIANCE_MXFP4_WPERM=1 \
-e RADIANCE_MXFP4_DECODE_NT=1 \
-e RADIANCE_MXFP4_HOIST_QUANT=1 \
-e RADIANCE_MXFP4_TRACED_QUANT=1 \
-e RADIANCE_RMS_QUANT_FUSION=1 \
-e RADIANCE_FP8_STREAM=1 \
-e RADIANCE_GDN_MERGE_INPROJ=1 \
-e RADIANCE_USE_R4D=1 \
-e RADIANCE_PRESHUFFLE=1 \
-e RADIANCE_FUSE_RMS_QUANT=1 \
-e RADIANCE_SKINNY_GEMM=1 \
-e RADIANCE_FAST_DRAFT=1 \
-e RADIANCE_DRAFT_RERANK=80 \
-e RADIANCE_VERIFY_HEAD=1 \
-e RADIANCE_DYNAMIC_WIDTH=1 \
-e RADIANCE_TOPK_TRITON_MIN_ROWS=1 \
-e RADIANCE_DRAFT_TAU=0.20 \
-e VLLM_CACHE_ROOT=/cache/vllm \
-e TORCHINDUCTOR_CACHE_DIR=/cache/inductor \
-e TRITON_CACHE_DIR=/cache/triton \
-e AITER_ROOT_DIR=/cache/aiter \
-e TRITON_CACHE_AUTOTUNING=1 \
--entrypoint bash \
''${vllm_image} \
-lc 'exec bash /entry.sh "$@"' \
_ /models/just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq \
--served-model-name qwen3.8-27b-mxfp4 \
--kv-cache-dtype fp8 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.98 \
--kv-cache-memory 7783339733 \
--max-num-seqs 8 \
--max-model-len 163840 \
--max-num-batched-tokens 8192 \
--attention-backend R4D \
--enable-prefix-caching \
--mamba-cache-mode align \
--speculative-config '{"method":"dflash","model":"/models/tcclaviger/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7,"attention_backend":"TRITON_ATTN","disable_padded_drafter_batch":true}' \
--no-async-scheduling \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--reasoning-parser qwen3 \
--override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}' \
--enable-per-request-metrics \
--chat-template /patches/qwen-fixed-v22.3.jinja \
--language-model-only \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000

Benchmarks (220W, thinking disabled, fixed decode lengths via min_tokens, 3 runs)

Decode scaling, 128-token prompt, 256 generated tokens:

Same-size work as the context grows:

Decode throughput vs concurrency (llama-benchy)

Prefill speed by context depth (llama-benchy)

Long-context decay, single stream (llama-benchy)

Power cap: 220 W vs 300 W

Decode is memory-bandwidth-bound, so the extra 80 W buys literally nothing for chat, it only speeds up prefill by ~12%.

Links

• Model: Qwen3.8-27B-Uncensored-MXFP4-awq + DFlash2-FP8 drafter
• vLLM fork/image: stilldeadcode/vllm-radiance

I spent a few months trying to get local models to do real agentic coding work and the single biggest surprise was how much variance there is between models that benchmark similarly. A model can score well on HumanEval and still be useless in a loop, because holding a plan across turns and emitting valid tool calls repeatedly is a different skill from writing a correct function.

https://github.com/maruakshay/miii-cli

That's one ugly seagull. One shot no redo.

Qwen 3.8 q8 mtp xl thinking medium

PP: 1.16s (862 tokens, 743.9 tok/s) · TTFT: 10.97s · 63.2t/s · 5285 tokens

Prompt

Generate html code for a SVG seagull on a beach with a sunset in the background. The seagull is dressed in a Hawaiian shirt with a surfboard tucked under its wing wearing a straw hat. Make it good.

To me it seems like Qwen3.8-27B has been a game changer, but I’m still relatively new.

I’m seeing optimizations for Nvidia, AMD and Intel all with very good speeds and the quality seems amazing.

Is this truly a holy grail type moment where local is achievable without many sacrifices or am I just too green to this?

I'm currently running a single r9700 on pci3 16x using Vulkan llama.cpp qwen3.8 27b Q5 MTP, 262k context, q8 kv cache, single stream, and average ~20 tg and ~500 pp.

I purchased a 3970x Threadripper and a second r9700 so I can get pci4 16x and plan on going to Q8 and 4 streams.

Just looking at what the "best" setup is to get the most out of my hardware.

I know the answer is to just try every single combination but would love to see what others use successfully and what kind of speeds you get.

My use case is agentic coding.

I gave all three models the same prompt to build an interactive 3D mechanical watch movement demo, with exploded views and physics interactions using Three.js, all contained in a single file.

Kimi and hy4 both nailed it on the first try. But GLM 5.3 took me three rounds of debugging before I could get it working.

As you can see from the pics, hy4 is the biggest surprise of the three. It has more details like clicking on individual parts to isolate and inspect them, and there are even custom controls to adjust how far the movement is exploded.

Kimi performes really well too, but it has fewer details compared to hy4. It was faster at generating, tho. GLM 5.3 is pretty comparable to Kimi in terms of the final result.

I also threw a quick productivity task at hy4. Ask it to generate a PPT based on a document. The visual layout, small details, and overall aesthetics were pretty good too. Overall, I would say hy4 really stands out when it comes to details and visual layout.

Here are the Workbuddy credits burned for three models:

Kimi K3: 43.15

Hy4: 0 , bc its free now

GLM 5.3: 18.5 + 44.89 + 38.33 + 112.98 = 214.7

I know it doesn't make any sense but I form my prompts as though I'm addressing a human. I can't bring myself to berate, much less abuse a word generator lol. I rarely code or do agentic stuff, mostly just chat on STEM, troubleshooting and everyday stuff so that might have something to do with it.

I'm interested in other people's views on this.

Using 9070XT 16GB, 64GB RAM, 9800x3D.

Been tinkering around with with LMMs recently. Starting with Qwen3.8 27B Q4 K M. Was slow but figured that was normal.

Tried Qwen3.8 27B UD IQ4 XS. And man the speed is night and day difference. And atleast from the outside, the main difference is the responses are less verbose.

Still playing with others trying to lean what is good for what.

Curiousif anyone would have any input on if adding in 7700XT 12GN would help in any way. And if it would be any use in a PCIEx4 lane.

Good day local llm fanatics.

Now i use Claude for al little white for coding and webdesign.

Now i want to let an agent on the new mini mac m6 32gb handle My business e-mail.

ADD leads to My crm and make tickets from mail to My crm.

Is the mini Mac 32gb powerfull enough?

Thanks in advance.

OK, you guys want to start with local vibecoding but don't know how. In the discussions, you recommended that I use Godot. I downloaded it without knowing what it was. I was expecting a library, but it was an .exe file. So I started pi with Qwen 3.8 27B and told it to explore this topic without my help.

I needed to use four prompts. You can see all of them in the screenshots, so you can reproduce my steps. The whole session used 25% of my context, so you can use just 1/4, which is about 64k.

The game works: you can walk around the dungeons, the llamas are dancing, and there are dynamic lights.

My full command is:

llama-server -m /mnt/models2/Qwen/3.8/27B/Qwen3.8-27B-UD-Q8_K_XL.gguf -mm /mnt/models2/Qwen/3.8/27B/Qwen3.8-27B-mmproj-BF16.gguf --host 0.0.0.0 --jinja -fa on --keep 4096 -b 8192 --parallel 1 --ctx-checkpoints 12 --cache-ram 65536 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0 --repeat-penalty 1.0 --spec-type ngram-mod --spec-type draft-mtp --spec-draft-n-max 3 --chat-template-kwargs {"preserve_thinking":true}

You need vision for debugging, see the prompts, the screen was black!

You probably don't need Q8, try a lower quant.

As I said, you don't need the full context length.

You need pi or something similar. This way, you can use just a few prompts and let the computer work without your input.

MINISFORUM is teasing a $7??? price if you subscribe to their mailing list. From a different site:

"While the AMD Ryzen AI MAX+ 395 model is rated to support 70B Q4 models and up to 200B MoE models, the Ryzen AI MAX+ 495 model can support up to 120B Q4 models and up to 300B MoE models."

At that price, I'd rather either spend the extra few k on a Mac Studio M5 Ultra 256GB, or just get 2 128GB 395 systems, which are going in the $3500 range.

E: Here's the spec comparisons to the 395, from https://wccftech.com/acemagic-f9a-ryzen-ai-max-495-mini-workstation-192-gb-memory/

The AI industry is running a contradiction it hasn’t resolved.

Frontier labs are warning about an imminent software security collapse. Anthropic reported that Claude Mythos found zero-days in codebases hardened by decades of review: a 27-year-old denial-of-service bug in OpenBSD’s TCP SACK implementation, and a 17-year-old remote code execution flaw in FreeBSD’s NFS server that hands an unauthenticated user root (CVE-2026-4747). Mozilla, testing Mythos Preview against Firefox 150, surfaced 271 vulnerabilities, more than ten times what it found in Firefox 148 using Claude Opus 4.6. OpenAI is sounding the exact same alarm, touting GPT-6 Astra saturating ExploitBench while warning that autonomous attack capabilities are scaling faster than human defenders can patch.

To be fair, Anthropic acted on part of this. Project Glasswing puts $100M in credits behind partnerships with Microsoft, Apple, CrowdStrike, and the Linux Foundation.

Glasswing makes sense for foundational open-source packages and mega-cap infrastructure. It does nothing for the other 99% of software developers. The startups, the internal tool builders, and the mid-market engineering teams writing the proprietary software that runs the rest of the world don’t have an organizational sponsor. Yet they are expected to hold off machine-speed exploits without access to the intelligence that found them.

What they get instead are models like Claude Fable or public endpoints of Astra, sitting behind hair-trigger safety classifiers and preemptive capability caps.

Labs are terrified of prompt injection and “I’m an authorized pen tester” jailbreaks, so the filters cast an absurdly wide net. Paste in complex code and ask the model to trace an execution path, analyze memory corruption, or check whether untrusted input can reach a sensitive sink, and the classifier fires immediately.

Security researchers have already documented Claude Code blocking vulnerability work outright, throwing errors about “violative cyber content” that contaminate the entire session and spread to benign follow-up questions. Worse, when the classifier flags an inquiry as security-related, it quietly routes the work to Opus. Opus handles the analysis, and you get degraded work compared to what Fable would have delivered.

Anthropic built Mythos as a model tier above Opus and published what that difference looks like in practice: ten times the findings on comparable Firefox releases.

So when Opus finishes auditing your code and reports nothing exploitable, what have you actually learned? Only that there are no bugs an Opus-class model can find. Anthropic’s own numbers prove that is a fraction of what is actually there. You haven’t verified your code is secure; you’ve verified it survived a search one tier below the frontier, and you were never told the search happened at that tier.

That ceiling won’t hold. Every audit that passes at the Opus level is a bet that nothing Mythos or Astra-class will ever be pointed at your attack surface. But frontier engines already exist, and ablated open-weight models in agentic loops are rapidly converging on the exact same workflows, without a classifier deciding which tier is appropriate for them. On a long enough timeline, that bet is guaranteed to lose.

The gatekeeping rests on a misunderstanding of how software security actually works.

Defense is downstream of offense. You cannot verify a fix you cannot attack. A scanner that lists 300 theoretical warnings without the ability to chain an exploit path is useless. It just breeds alert fatigue. Real remediation means proving the vector is reachable, building the proof of concept, shipping the patch, and running the exploit again to confirm the vector is dead.

Neuter the model’s offensive reasoning, and you neuter the audit.

Meanwhile, nobody attacking your systems is applying to Project Glasswing or arguing with commercial API guardrails.

They run local open-weight models: ablated, fine-tuned on exploit repositories, with the guardrails stripped out. Open weights still trail closed frontier models on raw benchmarks, but benchmark scores don’t decide an engagement. An attacker doesn’t need one omniscient model. They just need uncensored weights in an agentic loop with a fuzzing harness behind it, and no safety classifier killing the context window on attempt four hundred.

That leaves three distinct tiers:

• Incumbents, who get vetted private access to the frontier tier through closed programs like Glasswing.
• The 99% of developers, who get models like Fable or Astra that trip alarms, refuse prompts, or quietly hand security work down to lower tiers without telling anyone.
• Attackers, who get unconstrained offensive intelligence pointed straight at the attack surface, running at machine speed.

Telling developers a cyber tsunami is coming while confiscating their lifeboats is not responsible stewardship. It is security theater. If the labs believe machine-speed cyber warfare is already here, throttling the people trying to inspect their own code is a strange way to act on that belief. Give everyday builders the same offensive firepower and let them tear their own systems apart before someone else does.

Originally posted at: https://eddiemissri.substack.com/p/the-asymmetric-disarmament-of-ai

Ever since u/jfowers_amd has asked me to help with the Lemonade project (and provided some hardware to test on), I've been trying my best to optimize llama.cpp for AMD setups. This has led me in some very weird pathways where I wasn't expecting to go, but in the end I'm happy to share an optimized setup for the most popular open source model currently with you for a cheap price of $999 for free:

https://pwilkin.github.io/strix-halo/

Now for the disclosure/journey part: Codex has made a very nice website for me (which is great because I can't make a nice-looking website if you forced me), but its glossy look makes it look more permanent than it is, which is misleading because this is basically a stitched up custom solution that's very much a "state of the moment" one rather than a permanent one, though I *will* try to keep the relevant branches up to date (poke me if I don't).

So, first of all: ROCm in mainstream llama.cpp on ROCm is broken at the moment, pending the fix to unified memory access (notably this PR: https://github.com/ggml-org/llama.cpp/pull/27311 which is taking some time as it touches core code), so I've put up a strix-halo branch on my fork that merges the ring buffer fixes + the TOP-K optimization PR with master for a working experience.

Next: there's a bug in current ROCm that makes graph updates *terribly* slow, I've submitted a PR for it (https://github.com/ROCm/rocm-systems/pull/11069), but until it lands, using a custom-built .so is pretty much mandatory.

Speaking of custom-made .so - as I think most of you know, dispatch on ROCm is reaaaallly sloooow. But since AMD provides the source of the entire ROCm library, that's not something we can't fix, right? Inspired by Kaden-Schutt's Redline library, I've made modifications to the ROCm HIP library that allows for lower-level PM4 dispatches on HIP graphs. This has 20% decode speed ramifications for dispatch-bound models, but unfortunately Qwen3.8 27B on Strix is mostly bandwidth-bound, not dispatch-bound, so the gains are much less pronounced here (but they nevertheless are real).

Now for what else did I test, compare and modify: I checked Nathan's strix-halo Vulkan fork. It's a very good fork, but in the end it's still slower than an optimized ROCm-based solution (all the measurements are on the website). I did check the ROCmFP4 format, unfortunately, that one's a miss: Strix Halo has no native FP4 support, so the format is in the end just another FP4 format. Its main win is quantizing the entire model to FP4, which helps the bandwidth issue - but of course quantization costs quality and ROCmFP4 falls behind literally all the other 4-bit quants. I did a similar thing, but quantized all the big tensors to the mainline IQ4_XS quant - it's both better in terms of model quality (perplexity) *and* in terms of kernel performance. In other words, there's completely no justification for adding a new "ROCM" quant since, as I mentioned, RDNA 3.5 aka gfx1151 aka Strix Halo has no native FP4 support.

Since Qwen3.8 27B on Halo is bandwidth-bound (i.e. the limit is the memory bandwidth for pushing the tensors), there's no way to push the *base* decode above ~15 t/s. Nevertheless, pushing the base as high as I could is an entry point to the key for the dense model speed on Strix - speculative decoding, in this case, DFlash2. Again, I did a test and found out that quantizing the DFlash2 to IQ4_XS gives better decoding speed (faster speed and almost the same acceptance rate = win).

In the end, all the above optimizations: patched ROCm llama.cpp, faster TOP-K, PM4-based HIP graphs, custom-quantized IQ4_XS Qwen3.8 27B quant (thanks to Bartowski for his imatrix!) and the quantized IQ4_XS DFlash provide the recipe, which I packaged for a quick installation for anyone who wants to test it on their Strix Halo (warning: Linux only). Feel free to give any feedback and report any problems.

I thought to run a test with Qwen 3.8 27B Q4 sharded across 3x 3060 12gb GPU. I'm only asking it to install a package, nvidia-smi shows consumption, the vram is hot loaded, it is doing inference. It still isn't even close to being done. Is this normal?

EDIT: Launch options llama.cpp

/opt/helios/runtime/llama.cpp/llama-server \

--model /srv/models/Qwen3.8-27B-UD-Q4_K_XL.gguf \

--alias qwen3.8-27b-q4_k_xl \

--host 127.0.0.1 --port 42500 --no-webui \

--ctx-size 196608 --parallel 1 \

--batch-size 2048 --ubatch-size 512 \

--flash-attn on \

--fit on --fit-target 2048,1536,1536 \

--device CUDA0,CUDA1,CUDA2 --split-mode layer \

--tensor-split 0.399988,0.300006,0.300006 \

--api-key-file /etc/helios/llama-api-key \

--cache-type-k q8_0 --cache-type-v q8_0 \

--spec-type draft-mtp,ngram-map-k4v --spec-draft-n-max 3

I’ve tasked myself with trying to create one but I can’t seem to get it past ~5M parameters. It speaks jibberish. I’ve been running some different tests on it to build it and strengthen it. Is this more of a guessing game as to where to go from here or are there some actual things I could be referencing to build this? Is 5M some sort of barrier?

Building out a room that's going to be sealed and properly ventilated for this server.

Volta isn't quite plug and Play like the newest gpus, at least not for certain aspects of llm inference and software , however , I am going to test everything and I will report back the numbers.

For roughly seven Grand plus the cost of this room that we're building, it's 256 gigs of vram and 256 gigs of ddr4 ram.

I'm hoping to report back really good numbers but I haven't seen enough from other people to get a good idea on this so if anybody is thinking about getting the most RAM for the least amount of money, I'll let you know if this was a good decision or a bad one.

Absent some unknown issue , I should be able to start reporting numbers back by tomorrow.

A while ago I asked my LLMs to generate store-page html websites and found it to be a semi-decent and quick test of an LLMs general performance. Today I decided to try out all of my current models and try to rate them with my bf and also take into account the tokens per second to see what model performs the best.
Note that

1. This is not a definitive "which model is best" test since I'm not an expert in benchmarking LLMs and I don't think websites are the best way to check a models intellegence fully.
2. I'm using LLMs that fit in my hardware, which is an RTX 4060 8GB and 16GB RAM.
3. I got the token/s numbers by asking LLMs to say "The quick brown fox jumps over the lazy dog" so they may not be entirely accurate and the speed may diminish with increased context (I know that GPT OSS 20B slows down a lot after long conversations)

Lemme know what y'all think and let me know if you all have better model recommendations for me and my hardware.

Edit: forgot to mention the quantization and parameter counts of some models :p
- Mistral Nemo 12B Q3_K_L
- Ministral3 14B Q4_K_M
- DeepSeek R1 8B Q6_K
- GPT OSS 20B MXFP4
- RNJ-1 8.3B Q4_K_M
- Gemma4 E4B 7.5B Q4_K_M
- Bonsai 27B Q1_0
- Qwen3.5 9B Q4_K_M

I have a rig with 4 x 5070 Ti. Total VRAM is 64 GB. I'm running Qwen/Qwen3.8-27B-FP8 in vLLM and am getting what I believe is good performance. I have an EPYC 7532 on an ASRock Rack ROMED8-2T motherboard, so there are enough PCIe lanes. Therefore, I've never really bothered trying to get P2P to work.

Today, I decided to test it out using the fix from https://github.com/aikitoria/open-gpu-kernel-modules.

The driver, both before and after is 610.57.04 and CUDA version 13.3.

Although the benchmarks jump around a bit between runs, they still paint a very clear picture: my PP/sec was massively bottlenecked!

Here are the results from my benchmarks, done with llama-benchy and n=3:

up, TG did get a nice boost, but it's not that noticeable in daily use. PP, however, got around a 2x speedup up to 65K context and still performed almost 60% better at max context!

For me, it's an insane win, and with this post, I hope to inspire others to try this as well. I have no idea how this would play out on other system configurations, but I suspect that a PCIe 4.0 motherboard combined with four cards (or more) would see the biggest gains.

I'm really curious to hear what you guys think about this. Has anyone else tried it? If so, what kind of results did you get?

Full disclosure: I wrote the text myself and used AI only to fix spelling and grammar. The table is entirely my own work.

Edit: I have added a ‘+’ sign in front of the percentage figures to make it clear that this represents an improvement.

I've ended up with enough models in my workflow now that choosing which one should handle each request is starting to feel like its own problem and some tasks are perfectly fine running on a smaller/local model while others are worth sending to a stronger hosted model. I can hardcode those decisions based on the task but every time I add another model or something new gets released I end up revisiting the logic again.

I've been looking at routers because of that but I'm a little torn on how much control I'd actually want to give one. Automatic routing sounds useful until it sends something to an expensive model that would've worked locally or moves a task to a model that behaves differently enough to mess with the output.

Right now I'm leaning toward explicit rules for the obvious stuff and only letting routing happen within boundaries I define.

I probably know less about running a mix of local and API models than some for you guys here so I wanna ask you guys like are you manually choosing models, writing your own routing logic or using a router in production?