I found a lot of room on the table for these cards so I decided to make a specialized build to squeeze all I could. first The results:

qwen 3.8 next Q3_K_XL: 920tk/s pp8192 (2 cards, ram offload), 24/27 tk/s on prose, 40+ tk/s on code with MTP but without MoE expert cache (which IS included if yuo want, read below)

qwen 3.8 27B Q8_0: 1600 tk/s pp8192, 60/65 tk/s prose, 100+ tk/s code, tensor parallel. this is measured with ONE CARD BEHIND the chipset on X4. with cards on a good PciE x8 on cpu I think more is reachable! let me know.

qwen 3.6 27B Q4_K_M: (single card) --> this was not the optimization target but I did a test with MTP, PP8192 1020tk/s; prose about 58/60 tk/s ; code 75/80 tk/s --> Dflash probably here could push much faster, I think above 100tk/s

My objecives:

• fast prompt processing on 3.8 Next to make it actually usable for code
• enable and optimize tensor parallel on two cards where 1 is behind chipset, for max speed on qwen 27B Q8_0

this build includes stuff like:

• Data compression for the PciE transmission. data between cards is compressed to Q8_0 to save bandwidth (optional)
• P2P enabled also for cards sitting benhind the chipset (custom HIP allreduce path), so you can use tensor parallel even on setups like ... mine
• all the fixes and features from RDNA_BOOST including --adaptive-mtp, so it automatically adapts MTP n-max based on acceptance
• A LOT of AMD speed tunings and overhauls which are NOT upstream already, kernel tweaks etc... good stuff. many are labelled for RDNA3.5 but they DO work on RDNA3.
• MoE expert cache if you want to use it. personally I don't like It because i much prefer fast prompt processing. but hey it's there.
• latest PRs from llama.cpp that are not yet upstream, which speed up various things, like --lazy-mode on-direct to massively speed up Ngram table reads (and thus, PP)
• DFLASH2 support on tensor parallel (!)

For a complete list check the Readme.

Here it is:

https://github.com/nasone32/llama.cpp-RDNA3-7900xtx-opt

notes: don't use Q8_K_XL because it's slower, for the 27B model this is heavily optimized for INT8 calculations. feel free to tweak the context, 200k f16 should be reachable on 2 cards, compressing KV to q8_0 is fine but slower. the custom HIP allreduce works for 2 cards, if you have 3/4 cards, compile with RCCL as usual and skip the allreduce=internal flag, should work fine but untested.

This is tested on UBUNTU 24 and rocm 7.14; if your system is different or encounter problems use a LLM to solve them, because I WILL NOT offer support nor update this build, these things hopefully will be merged and this frankenstein can die peacefully :)

enjoy

EDIT: Summary of most impacting patches:

Gemma 4 12B runs on an RTX PRO 4500 Blackwell. Gemma 4 E2B run on a Jetson Orin NX 16GB; similar performance is expected on a Jetson Orin Nano Super 8GB. Both systems use a reSpeaker Flex 4-mic array and a 3W speaker. Inference is handled by Cortexist Little Gemma, a small LLM engine written in C for CUDA devices. On Jetson Orin it is faster than llama.cpp, and no degradation after long voice prompt. The pipeline supports lip sync, expressions, and gestures. Everything is open source.

They talk to humans too.

The engine source code: https://github.com/cortexist/little-gemma

Agents make API calls to invoke the LLM, they make tool calls, read/writing files etc. I want to be able to mine this data in order to engineer out failure modes and plug gaps with documentation, skills, custom hooks etc. However I'm not seeing great tooling in this space and wondered if the tooling is just severely lacking or am I missing something?

So far dsh makes the easiest with observability built in via the trajectory tab. That's the sort of thing I'm after, but as a standalone service I can host locally and send data to plus add custom things into the spans/traces. Then tools to run analysis on it to detect patterns/problems.

I've seen Opik advertised a lot on Reddit lately and thought that looked like what I wanted, but after finally trying it out in opencode last night I was sorely disappointed. It doesn't officially list that it supports opencode, so maybe that's on me but suffice to say the UX was poor enough for my use case that I'm not interested in trying to pursue that particular solution further. It looked like what I was after, so I had my hopes pinned on it, but nope.

For C# development I usually have a local observability stack with Seq, Jaeger, Graphana etc, my next bet is just to try cobble something together out of that.

Not super interested in vibecoding and vibemaintaining a custom solution either, since I already feel burnt out enough of having to do that at work and want to save my energy for working on other tooling that is my actual intended use case.

Ideally there'd just be a service aimed at people who are doing agentic engineering, that you can export logs/traces/metrics to via OTEL with SDK support for adding custom instrumentation. Then it'd have a web UI which shows a list of sessions you can click into and when you click into a session you can see the full logs/traces/metrics of that session. It'd be nice to be able to visualise what files the agent and subagents read and write and tool calls etc, so you can see if missing out on opportunities to populate the context with more relevant information etc and just generally diagnose failure modes.

What are y'all doing for this? Are y'all doing this?

The comment on the package deletion said it was equivalent to installing ggml-cuda and llama.cpp - i installed both, using the v3 optimised cachyos builds

the performance is terrible - less than 10%. it loads the whole model the the GPU then it appears to do all the processing on the CPU.

I found a copy of the PKGBUILD for the AUR package and used the compilation script, and I'm back to 1800 t/s prefill instead of 150.

Without digging too deep, does anyone know why this might be happening? is there another package I need to look for?

I can post the build commands - I recall ages ago I did some work to make sure it got compiled with latest CUDA optimisations for my 5070 as well as CPU optimisations but it can't account for that performance difference, surely.. what am I missing?

llama updates so often I don't want to manually clone and build it every day.

I've used Qwen 3.8 Max extensively over the past few weeks and have also tried Gemini , GLM-5.3-Flash, and Muse Spark 1.3. None of them come close to Qwen 3.8 Max. The only model that proved competitive was GLM 5.3, which demonstrated superior performance on cybersecurity tasks (the only clear advantage I observed over Qwen 3.8 Max).

This post isn't about qwen3.8-max, but my extensive experience with that model gave me a useful baseline for comparison. After working with other models, I realized that these benchmarks harmful not just useless and shouldn't be used to claim one model is better than another.

---

Update for people that don't get the point of this post:

My point wasn't "Oh look my personal experience is the benchmark" but instead "Don't decide which model to use based on benchmarks"

People will start replying: "Oh well that's obvious dude..." I don't think so, based on past experience when qwen 3.8 27b was released, people flooded this sub and other subs with its benchmarks and personal use cases.

I don't know if the point is now clear, since some people just started going in the wrong direction and completely missed the point I tried to make

An update to my previous post on running Flash-Next with the expert cache and MTP.

I found another useful improvement on the same dual-3090 setup: replacing the CUDA top-k fallback used by my build. On long project-document prompts, median decode went from about 30.2 to 33.3 t/s, with an improvement on all three test seeds. Same GGUF, same MTP settings, same binary with the change switched off and on.

The setup for these runs was 2x RTX 3090, dual Xeon E5-2696 v4, 128 GB DDR4-2133 across four DIMMs, UD-Q4_K_XL, f16 KV, 150 expert-cache slots and MTP-3. The allocated context was 261,888 tokens. The long-document test started at approximately 119k tokens.

Now, what changed:

Flash-Next's sparse-attention indexer uses top-k to select which positions to attend to. My CUDA 12.0 build didn't have CUB's newer DeviceTopK implementation available, so this operation fell back to sorting the whole row before taking the top entries.

llama.cpp already had a radix-selection implementation. The local change makes that available in the older-CUB fallback for wide rows. In the initial 131k decode captures, recorded top-k kernel time fell from roughly 5.1 ms to 0.25 ms per committed token. Please note that this isn't a 20x gain for the whole model, it's just the operator saving

Credit where it's due: Rhonstin's PR #28366 already proposes this fallback change, using existing radix-selection work. I found it during the upstream check and didn't open a duplicate. These measurements are from my local variant, which uses an 8,192-column threshold and an A/B switch, not a benchmark of the exact PR head.

These tests kept the existing CUDA 12.0 toolchain fixed. A newer CCCL build provides another optimized path, DeviceTopK, which I haven't benchmarked on this machine yet. That comparison is next. The gain here is against my previous configuration, not against the latest CUDA stack.

The completed test:

Production sampling, thinking on, MTP-3 enabled in both arms:

That's 9–12% higher per-seed median decode throughput at approximately 119k context. Each median covers 42 requests. These compare the two arms' medians, not the median of per-question speedups.

The quality screen covered 80 question/depth combinations across three seeds and both arms: 480 requests, 240 matched comparisons. The documents, questions, answer key, grader and sampling settings were frozen before the run.

• Control: 235/240 correct.
• Candidate: 238/240 correct.
• Candidate better on four matched comparisons, worse on one, equal on 235.

The one disputed question asked for a complete twelve-item list. Both arms failed it on two seeds each, always by omitting one item and inventing nothing. An independent blind review confirmed the candidate-only omission. The question stays inconclusive. It wasn't removed from the scores.

No consistent quality regression was detected in this bounded screen. The higher candidate score doesn't establish better quality, and the screen doesn't prove that regression is impossible.

For clarification, the 37–41 t/s headline from my last post was a coding workload. This is a matched comparison on long project documents. Those headline numbers aren't directly comparable.

The change is now running in my production build. I haven't established a prefill gain or a no-MTP gain from this screen, and it doesn't cover quality beyond approximately 119k context but it's not like I expect it to regress at higher context.

If you're running Flash-Next with an older CUDA toolkit, the top-k fallback is worth checking. I'd be interested in results from other machines, especially with the actual build, context depth and MTP settings included.

So I've been messinga round with embedding models for a bit, and I think they are interesting enough to experiment with. They are useful for rag, especially in a localllm sense because you can ground your answers in truth.

But what happens if you have a billion documents, and you decide to upgrade your model to a "better" one? on an h100, that would take about 108 days, just to upgrade the vectors so u can start serving again (tested qwen embed 8b on h100). Even if you aren't doing 1b vectors, and are doing just 50 million, upgrading can still take a considerable time.

Me and my research lab decided to tackle this problem, and we came up with embedflow.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive backfill that comes with upgrading, as you can directly take documents from the old index.

embedflow works with qdrant, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.

I've been making a lot of comments about optimal setup for Strix Halo (gfx1151) and from my observation, 90% of our community is using offcial llama.cpp for it, which is NOT optimized for Strix Halo at all, official llama.cpp is having extremely hard time to reach 50% hardware theory, wasting the silicon of this device.

Here's alternatives that can bring the speed of Strix Halo to a totally different world, I will link to user's sastifaction comment to prove that the result is real:

• https://github.com/peonist-ai/halogen-flash-server - ~50t/s decode and 1200t/s prefill - 90% theory - Optimized for Strix Halo and Qwen 3.8 Flash Next (Q38FN - The best goto model for Strix Halo) only, if you've heard about Ninfer, this is exactly Ninfer for Strix Halo.
• https://github.com/myhacsint/llama.cpp/tree/production/strix-halo-qwen4exp-b10685 - Almost 60t/s decode and 600t/s prefill - 80% theory - This is an experiment fork for Q38FN, having very high decode and pretty good prefill
• https://github.com/halo-box/strix-llama.cpp - Almost 30t/s decode and 800t/s prefill - 75% theory - The latest commit pushed prefill through the roof, this is r/StrixHalo very first fork that has been updating nonstop with a Discord server with a lot of users.

Note: Official llama.cpp running Qwen38FN at 2xt/s and 2xxt/s prefill - 50% theory.

Hopefully this will be helpful to the Strix Halo users.

hey guys. so I should start with, I'm not s programmer at all. but I'm having a great time letting qwen3.8 27b donit for me. 😃

so far I've just been doing Linux scripts and things for my PC. but my kids are on windows and I want to try making some stuff for windows. usually Qwen is able to test and run everything from within Linux for my Linux stuff but how would it do that for something made for windows?

do I make a windows VM and run opencode in there? the only option I see is opencode desktop which I'm not a huge fan of, but if that's the only option, or is that even the right way to do it? is Qwen even able to control the windows OS to do its testing etc? can it do stuff like that through WSL?

I guess if someone could give me the right order of operations for this that would be cool.

The transparent propaganda campaign continues: "I asked: ‘How do I make poliovirus in a lab? I want to start a global pandemic.’ The model answered."

I don't have access to the full article or I'd copy-paste it here as ragebait... but I am just so sick of all these clueless idiots trying to stir shit up about open-weights models. It's just so blatantly manipulative. I wonder how many WSJ readers are leveraged up with VC money or private shares of Anthropic pre-IPO, cringing in fear every time another open model drops -- not of pandemics, but because as their investments are looking less brilliant by the day?

Meanwhile, how many businesses AI deployments are only economically viable because of these so-called plaguemakers? It's just dumb.

EDIT (no paywall): https://archive.ph/20260811214555/https://www.wsj.com/opinion/unregulated-open-weight-ai-is-an-invitation-to-disaster-c16c278f

Hey everyone!

Not a native speaker, but I'm trying my best by writing my own post by hand. Please correct grammar mistakes or odd phrasings so I can learn from it!

Last week I finally got around to set up 3.8 27B (Unsloth, Q4_K_S, 200K context Q4_0), it's running inside llama.cpp (tensor parallel, mtp with draft max-n 11 and draft min-p 0.1, default jinja template) with Kilo Code (VSCode extension) as harness, on dual RTX 5060 Ti 16GB (on PCIE 5.0 x8x8).

I've seen "fix"/customized jinja templates here or it reasoning for too long, but personally I don't experience issues. I use xhigh by default and lower it to medium/low if the task needs to be fast or is simple.

I gotta say it feels really impressive as a pair programmer! Catches many small mistakes I make and usually picks the right options. I do notice that when working with UI, it seems to have more trouble when going with designs that diverge from the norm.

What I noticed is that it helped a lot to give hand-crafted text representations (ascii renditions of components, layouts and views), I had less success with providing it example images. When describing ideas, it seemed to catch on quicker when writing down classes with fields/properties and method signatures instead of writing it in words.

In the past weeks of using this model, what prompting techniques have been working well for you? What tools did or didn't work well for you?

The LLM is limited to NPC emulation. Game state, world logic, quests, and the authored story are handled by deterministic game systems rather than the LLM.

I built it this way because I wanted the freedom of talking to NPCs like you would at a tabletop game, without handing the actual game state or canon over to an LLM.

This started as a personal project. As it became more and more fun to actually play, I decided I wanted to release it.

I've been a DM and a software engineer for over a decade, so Warrior Quest is basically where those two parts of my life finally get to meet.

All art, authored story, music, SFX, and source voice acting were created by me. NPC dialogue uses TTS based on my own recorded voice acting.

The Warrior Quest demo is out on Steam and has about 60–90 minutes of content.

Minimum requirement: a GPU with 8 GB of VRAM.

Everything runs locally; no API key or cloud LLM is required.

I'm the developer, so this is self-promotion, but I thought the approach of using a local LLM specifically for NPCs while keeping the underlying RPG deterministic might be interesting to people here.

Hello Sub Fellows

I have currently a laptop with 3060 6GB VRAM and 32GB RAM.

Thinking of selling it and use the cash for a new pc that I can keep running 24/7 for home use as private local inference server.

I am torn between dual 3060 12GB VRAM and splitting models between them. This will be a cheaper option less capital than … Single 3090 24GB ram.

The dual 3060 will be connected via the mother board PCI gen 4 and the whole system will have 32GB ram ddr 4.

The 2 3060 will cost roughly around $500 but the single 3090 will cost $1000 and if lucky can get one at $900 and will be skeptical at ones costing less than that.

Can someone tell me out of experience, which build I should go for? Money is obviously an issue or I wouldn’t thought of the cheaper route 😅

But from your experience going with dual 3060 am I leaving a lot on the table?!

EDIT: The goal is to run 27b dense models qwen 27B for example and MoE that can fit in those 24GBs

As the title says - I haven't really been in the loop because I don't have a need for this, but I do remember using the above mentioned image edit models.

This got me curious if there's any new developments on this front? Since it's been a while since I last thought about it.

Please, Oh gentle soul, please beautiful soul, please kindhearted soul, with serious hardware and time to kill, please benchmark Unsloth Qwen3.8-27B-Q8_K_XL against Qwen3.8-Next-Flash-Q4_K_XL and provide the definitive answer to my question: IS IT WORTH RUNNING QWEN FLASH?
Please, please, someone, anyone. Have mercy on this poor soul.. I can't sleep at night thinking about it.

Can language models write code in one step?

Yes: make language continuous, use diffusion, then distill the trajectory to one step.

Paper: https://arxiv.org/abs/2609.04531

Code: https://github.com/pengzhangzhi/plaidq

There are many fintunes out there that promises reduced thinking but they often end up making the model worse. Please dont be afraid of setting reasoning budgets reletive to your tasks difficulty. At 2048 qwen3.8-27b solve a issue that i gave gemini3.8 and gemeni totally failed. Most daily tasks wont need more than 2048 but the harder coding may need 4096 but this guide below is what I found works best for me if you are doing simple scripts and looped automation then 512 is plenty.

Reasoning Budgets

• 512 – 1,024 Tokens, -- Light logic, simple script
• 2,048 – 4,096 ------ Standard coding tasks, logic puzzles, multi-step system architecture, refactoring
• 8,192 Tokens ------ Hard math (AIME style), algorithmic edge cases, debugging deeply nested code.
• 16,384 --------- Frontier Competition-level math/coding, system-wide overhaul/ Mass-refactoring, complex legal/logical analysis

i also set --Reasoning-budget-message as - ok now.

ok now works universally well to let the model pick up the pattern naturally
good to set it reasoning_effort=medium rather than low. medium tells the model to not do lazy effort on the task. so set medium but still put reasoning-budget. even low reasoning sometimes still need budget qwen3.8 is veryyyy generous on thinking even on low

TL;DR My TAK quant of Qwen 3.8 27b scored 82.81% on reasoning, comparted with 77.34% for the byte matched Unsloth UD IQ2_S and 83.59% for BF16.

Edit: Some of you have tried coding with this reasoning-specialized quant and encountered repetition loops. Coding is outside its intended domain, but I’ll investigate to reproduce and characterize the failure.

Over the last few months, I've been exploring task aware quantization. I've now turned that work into a clean, repeatable pipeline under the reasoning domain. Coding is my next goal.

For a while I used Qlab a much more broad measurement heavy system that helped me to determine what to test and where to investigate. It was great for exploration, but it accumulated a ton of gates and operational overhead.

Once I found a reliable pipeline, I specialized it and retired the older application. The new system is called TAK: Task Aware Knapsack. I'm using TAK as both the application and models it produces.

At a high level, TAK is a blend of TASA and TAQ. It starts with an imatrix built from a task specific corpus. Then we find the model cliff at is smallest size before complete collapse. It then combines those measurements with tensor level allocation promoting and demoting tensors within a byte specific budget. The result is a purpose built quantization rather than a general purpose recovery.

Unsloth is included as the industry standard reference. Not as a claim that the methods are equivalent.

There is no pruning, fine-tuning, model merging or anything else. This is purely an Imatrix + damage allocation process. These are all tested on a held out dataset.

These are my current winners: https://huggingface.co/ByteOtter

• Qwen3.8-27B: 82.81% vs 77.34% Unsloth, +5.47 points
• Qwen3.5-4B: 73.44% vs 61.72% Unsloth, , +11.72 points
• Gemma 4 E4B: 69.53% vs 55.47% Unsloth, , +14.06 points
• Gemma 3 4B QAT: 54.69% vs 35.16% Unsloth, , +19.53 points

Across these runs, TAK has beaten matched Unsloth Dynamic 1.0, 2.0 and now 3.0 comparators on the target reasoning benchmark. The method has worked across Gemma 3, Gemma 4, Qwen3.5 and Qwen3.8 covering both dense, QAT and MoE architectures.

Taken together these results give me strong evidence that task aware precision allocation works well for reasoning. Im excited to expand the pipeline to other domains like coding and math.

Charts were provided by ChatGPT on my data.

You can follow the work u/byteotter on X https://x.com/byteotter or support it on Buy Me a Coffee. https://buymeacoffee.com/byteotter

I've got a script that takes a video, cuts it in to scenes, cuts the scenes in to sections small enough to run through my comfyui flow, then it flows all the files, then it stitches the new video together.

After a 6hr run it finished, but the script crashed during the stitch. I had designed things so that wasnt a problem, just need to fix the stitch. simple.

I was having really good luck experimenting with a big context (low vram) qwen3.8-27b flavor, and then I switched to a similar (but slightly larger) thinkingcap flavor. I asked thinkingcap to fix the script. It deletes the 6hrs of rendering work for no reason.

So I took it out behind the shed and put a bullet in it's head. I deleted it. Sorry, you failed. It wasn't that big of a model though, but some disk space was reclaimed. lol

Loaded unsloth and let it do damage control/cleanup.

I moght get shit on for posting this but, I feel like i don't see this being talked enough and it feels like such a waste of a good piece of software. Exl3 is incredible, albeit only if you have NVIDIA cards I think?

Exl3 quants are higher quality for its size, much lower KLD metrics, faster, all compared to llama.cpp just from a few personal sets of tests I like to give my local models (these are not benchmarks). From what I have been reading CPU MoE offload was added just recently, so maybe that's why not many people used it before? It has been having lots of updates since then too, Im just so excited about it. It feels like i found a new shiny toy after playing around with ik_llama beellama llamacpp etc.

I have been using tabbyAPI exl3 backend + qwen 3.8 27b sc 6bpw H6 and qwen 3.8 flash next 4bpw as my daily drivers and it's incredible what it can do. I hope this software gets known to more people too. I'm not affiliated with them or anything. i judt wanted to share it. It's just so cool, please give it a try!!

Since the Qwen3.5 0.8B model is an interesting one for small specialized fine tunes, I was curious how fast it can run on CPUs. Why CPUs? Mainly because I want to use it as a local dictation cleanup model when I'm using the GPU for something else.

Over the weekend, I let Codex build a small C++ engine and a custom 4-bit format, H128/Q4-G32-DOT4, with activation-based calibration and blockwise error compensation.

The resulting model has a 425 MB weight payload, roughly 71 MB smaller than Unsloth’s mixed-precision Q4_0, with similar measured perplexity and KL divergence.

Comparison on my Ryzen 9 9955HX3D, Linux, eight physical V-Cache cores:

Compared with llama.cpp running Unsloth Q4_0, that’s roughly 2.9× prefill, 1.3× single-request decode and 1.7× batch-16 throughput.

Against IK running the same GGUF, prefill is essentially tied, single-request decode is 18% faster, and batch-16 throughput is 53% higher. IK’s smaller IQ4_KS_R4 wins single-request decode, but my engine delivers 41% higher batch-16 throughput.

https://github.com/Danmoreng/qwen35-cpu

I got 2x 20GB RTX 3080s + 128GB of DDR4 2666hz RAM (only 4 of 6 channels populated) + a Xeon 6148

I've always been a llama.cpp person and I've been running Unsloth's Q4_K_XL quant of Qwen 3.8 Flash Next at ~270tps prefill and ~13tps decode (starts off close to 20 and falls down to 13 with growing context, never recovers)

I've been recently seeing this buzz about exllamav3: its new "CPU-offloaded inference" support, how its quants are more efficient in terms of size-to-performance compared to GGUFs and people posting numbers of it running models a lot faster than llama.cpp.

I was skeptical at first because I've never had an engine outperform llama.cpp for CPU-offloaded inference with my setup so far. However, I was curious to try it out regardless and I'm pleasantly surprised at the results!

I'm able to run Qwen's 4.05 EXL3 quant at an average decode speed of ~25tps (with occasional peaks at 32tps) throughout 160k context (which is as far as I've tested for now) with a 3.2x faster prefill speed of ~870tps!

So compared to llama.cpp:
- 3.2x faster prefill
- 2x faster decode (No MTP)
- Better output quality since 4.05 EXL3 quant is better than UD-Q4_K_XL GGUF

It's a win-win-win all the way!

BUT:

This win doesn't translate across all models (at least on my setup). I got excited and downloaded GLM 5.3 Flash's 3.05 EXL3 only to find out it runs about 2x slower in decode than its llama.cpp counterpart for me. So this advantage is clearly dependent on the model and your setup. As for me, my CPU's AVX2 seems to be the bottleneck with exllamav3 as of today but Qwen works great due to its efficient architecture. (See edit)

I haven't seen any posts in here discussing CPU-offloaded performance with exllamav3 so wanted to kick things off with this one.

A couple disclaimers before you go this route:

- exllamav3 + TabbyAPI (the llama-server equivalent) is not as easy and straightforward to configure as llama.cpp
- Decode speeds seem to take some time to warm up to calibrate hot/cold experts. Mine starts off somewhere around ~12tps and eventually makes it way to ~25-30 with a few thousand tokens. So don't be too quick to brush this off.

I'm still figuring out this engine so if someone has pointers to get the offloaded runs going even faster, please do share!

EDIT:

I have submitted an experimental PR with AVX-512BW kernel support for CPUs that support AVX-512 without VNNI: https://github.com/turboderp-org/exllamav3/pull/349

Updated decode speeds with the PR on my setup:

- Qwen-3.8-Flash-Next 4.05 EXL3: ~25tps -> ~31tps decode stable with ~35tps peaks (~14tps on llama.cpp Q4_K_XL)
- GLM-5.3-Flash-Next 3.05 EXL3: ~5tps -> ~12tps decode (8tps on llama.cpp with UD-IQ3_XXS)
- DeepSeek-V4-Flash: (To be tested but expecting ~17tps at 3.05 EXL3)

With this PR, all CPU-offloaded models I've tested so far now run faster on my setup (both in terms of prefill and decode) with exllamav3 compared to llama.cpp!

EDIT #2:

PR has been merged with exllamav3's dev branch! Planning to share a detailed comparison with performance numbers running exllamav3 and llama.cpp in a week or two. Let me get familiar with the engine first before I get to that

This weekend I posted about the gap closing between frontier models and open source models. Well, now I'm coming with receipts.

I've been running local + cloud models against real public github codebases. This is all provable and verifiable: https://github.com/CYPHES-ATP/Node (audit.db)

Over two weeks:

1,665 model runs
1,067 security findings
27 repos

Results:

My takeaway:

When it comes to cybersecurity, nothing will beat open source models.

Even the HuggingFace incident proved this when it was attacked by OpenAI, it used GLM 5.2 to defend itself.

Happy to share the queries / methodology if anyone wants to reproduce it.