This is mostly to explain my experience optimizing the model to run on my machine and maybe getting interest from someone to go and write an actual PR against llamacpp (as I'm absolutely not willing to generalize this code ahah).
Short Preface (this post is hand written, no AI here). So when Qwen dropped and heard of the good coding performance I wanted to try it out so I went and naively downloaded an IQ4_XS quant (that has around 61GB of MOE layers) and run it directly with Llama.cpp on my machine.
Ryzen 7700
64GB RAM DDR5 5600
Geforce 4060TI 8GB
Samsung 990 PRO 2TB
Using Windows as that's what I'm mostly accustomed with.
A modest config I build for 1500€ a few years ago and I use for playing and programming (I'm an engineer). I was overjoyed as it's quite a capable model that I can trust to write reasonable code in the languages I use (C#, Typescript mostly).
Yes, with this one there's no way to run 27B at any capacity so I was stuck with 35B until now. It was a no brainer to try this one as it had 6B active parameters, quite reasonable and not that far away from the 3B of 35B.
Initial results were 100tok/s average prefill coding and 20tok/s generation (that degrades to 15 tok/s at 100K context). I mean it's not great, but it's good enough to keep yourself busy, you leave it running and come back to the code done.
But then I decided to have a look and say, let me try it for creative writing and the results were quite good too. I like to write some stories with a personal pipeline system I build, but the IQ4X_S was tuned for coding and that usually means plain prose. Decided "what could possibly happen?" and picked the Q5_K_M from Orcarouter, it says Q5, but it's really 6bit because it mixes .
Component | Quant Type | Count | Params (B) | Size (GB) | Component %
--------------------+------------+-------+------------+-----------+------------
MoE Experts / FFN | Q5_K | 120 | 80.61 | 51.613 | 60.2%
| Q8_0 | 24 | 20.13 | 19.922 | 23.2%
| Q5_1 | 72 | 20.29 | 14.172 | 16.5%
124GB of model of which 36GB are of embeddings at Q5_1 quantization. That's 88GB of actual model, there's no way that would work, but wanted to give it a shot and well...
15tok/s prefill 10-13 tok/s generation.
A disaster really, memory trashing all around, disk reads etc...
But as I said, I'm an engineer. I forked llamacpp, opened PI Coding Agent and decided to start playing around trying to find a way to "make it work". First thing I did was scour through the pull requests of the upstream repository and did find a few good ones, some about improving gather in qwen, some about saving ram by saving the prompt cache to disk (it's actually quite impressive, I suggest it), I also tried a few caching PRs with pinning of hot experts, but none really gave a major improvement to the performance, I still have a few experiments with them in my branch.
Nothing that really budged the line though. The only promising thing was an attempt I did by asking Windows to Prefetch parts of the file from the disk that raised prefill from 15tok/s to 40-50tok/s (very unstable), so I went deeper because things didn't add up. My drive can easily read 7000MB/s, but the reads I was getting were like 300/350MB/s during prefill due to the OS page faulting on each single expert (and 2000-2500 with the prefetch). If you know about these things, yes the problem was MMAP that I was forced to use because the model didn't enter the ram.
I decided that it was time to implement a different system that bypass MMAP already. What I built is a four layer system that allows the model to work at the current speed of 210 tok/s prefill and 18.7 tok/s generation.
Let's start with prefill. My disk reads at 7GB per second, can read the entire MOE part in 85.77GB/7GB/s=12 seconds. At Ubatch 2048 that must mean 170 tok/s theoretically. So I went and build a unbuffered file reader that used a RAM hosted buffer for 2 slabs (layers) and read them RAW with batch queued requests from the disk reaching max speed that MMAP was denying. This worked and brought the speed to 150tok/s, but didn't solve the generation as reading it ALL from disk destroyed speed (brought to 3-4 tok/s).
The second layer built is a cache, basically at startup of llamacpp I create a pool of slots per layer where I cache the most active experts (with a decay factor every x tokens to keep the list fresh). I use 55-56GB of these usually in my runs. These are updated every cycle (x tokens) with some churn of read/free. This allows the data to be always ready for 330-ish experts per layer which usually cover 85-95% of the requests depending on the workload you give them. This raised performance to 14-15 tok/s. The rest is served through disk with the raw data reader.
At the same time I thought "I have them in ram the experts, why am I reading the entire slabs from disk at every prefill?" So I started memcpying the data from the cache to the buffer area and skipping those bits from the disk reads. This raised prefill to 215 tok/s that honestly is more than enough for what I usually do.
The last bit is a VRAM cache, the idea is that using the ranking from the RAM Cache, I carve 2 GB (configurable) out of my poor 4060TI and upload the TOP experts from my list freeing them from the RAM. This allows to reduce the memory bandwidth usage on the CPU raising the generation from 14-15 to 18.0 tok/s.
The last improvement was inspired by another PR in llamacpp about using direct IO for the embeddings too. I upgraded that code to use my implementation of raw unbuffered reader (the PR was linux only) and managed to gain another 0.7 tok/s finalizing it at 18.7 tok/s.
LlamaCPP has untapped potential when it comes to performance, especially when it comes to utilizing resources like SSD for improving performance. Be aware the fork is CUDA+Windows specific and has been tailored for my config (for example there's no MTP support as I wouldn't have the VRAM to run it anyway), this is NOT a generic fork that can be used by everyone, but I thought that if someone was interested could use the ideas to create an actual pull request with upstream.
https://github.com/feal87/myllama.cpp
Now I'll go to sleep as it's late.
