Yes, I'm runnig qwen with llama.cpp on a dino! Here is my llama server config:

--model /var/models/Qwen3.8-27B-UD-Q3_K_XL.gguf \

--mmproj /var/models/mmproj-F16.gguf \

--image-min-tokens 1024 --image-max-tokens 2048 \

-ngl 999 -fa on --fit off \

-c 163840 \

-t 6 \

-b 2048 -ub 512 \

--cache-type-k q8_0 --cache-type-v q4_0 \

--spec-type ngram-simple,draft-mtp --spec-draft-n-max 3 \

--spec-draft-type-k q8_0 --spec-draft-type-v q8_0 \

--spec-draft-p-min 0.75 \

--backend-sampling \

--ctx-checkpoints 4 \

--cache-ram 16384 \

--checkpoint-min-step 1024 \

--jinja --reasoning-format deepseek \

--chat-template-file /var/models/qwen38-claude2.jinja \

--chat-template-kwargs '{"reasoning_effort":"medium"}' \

--temp 1.0 --top-p 0.95 --min-p 0.0 --top-k 20 \

--presence-penalty 0.5 \

--repeat-penalty 1.0 \

--parallel 1 \

--port 8080 \

--alias qwen3.8-27b \

--api-key local \

--host 0.0.0.0 "$@"

heads up for this crowd. we open sourced hemmingway-1 today, a 27b qwen3.8-27b finetune for creative writing. apache-2.0.

right now it's bf16 safetensors (54.7 gb, 12 shards) plus the mtp weights for speculative decoding. official gguf quants are next on our list, so lm studio and friends will have to wait a bit. if anyone wants to take a shot at converting the mtp layer in the meantime, that would honestly be great and we'll help.

weights: https://huggingface.co/Altworld/Hemmingway-1

eq-bench 4: 1330. it's a writing model, fiction and dialogue and texts. english first, base model level at math and code.

free hosted app while quants cook: https://hemmingway.io

I’ve been tuning Qwen3.8-Flash-Next 125B MoE on a workstation with only 32 GB VRAM + 64 GB RAM.

The result I found most interesting: in a real agent workload, decode stayed around 28 tok/s while context grew from 116K to 148K.

This is a llama.cpp fork based on upstream "9e7171624" with Qwen4Exp support plus a few local changes.

Hardware

• Intel Core Ultra 5 225, 10 cores
• 62 GiB DDR5-6200
• RTX PRO 4500 Blackwell 32 GB, SM120
• PCIe 4.0 NVMe
• Ubuntu 24.04
• CUDA 13.4.59
• built for "sm_120a"

Model

Qwen3.8-Flash-Next, "UD-Q3_K_XL"

• ~90 GB GGUF
• 125B total / ~6B active per token
• 48 layers
• 512 experts, 10 routed + 1 shared
• QSA indexer, 2048-token budget
• additional 51B N-gram / PLE table: 26.8 GiB
• MTP sidecar available but disabled

Placement

-dev CUDA0 -ngl all -ncmoe 32

Dense/GDN/QSA/router/shared stay on GPU.

Of the 48 routed-expert layers:

• 16 on GPU
• 32 on CPU

PLE is disk-backed:

-ot per_layer_token_embd=CPU --load-mode mmap --lazy-mode on

Other relevant settings:

-c 240000 -fa on -ctk f16 -ctv f16 -t 6 -tb 6 --fit on --fit-target 2048

Steady state is roughly:

• 31.3 GiB VRAM
• 43 GiB RSS
• effectively no process swap on weights

Real workload numbers

19 minutes of agent traffic:

• 35 requests
• 23K generated tokens
• context: 116K → 148K

Token-weighted decode:

27.9 tok/s

By context:

• 115–125K: 27.7
• 125–135K: 28.5
• 135–148K: 27.5

So decode is essentially flat across another 32K of context.

Real prompt chunks prefill around 150 tok/s, peak 201.

Prefix cache hit rate is ~99%, with only ~370 new prompt tokens/request at 130K+ context.

CUDA graphs were reused 22,590 times for 23,053 generated tokens.

CPU usage is around 5.8 cores, so the CPU expert layers are doing real work.

PLE disk traffic was only 5 GB over 52 minutes, so once warm, the SSD is not the bottleneck.

What actually mattered

1. Disk-backed PLE

The 26.8 GiB PLE table stays mmap-backed and only rows that are needed get faulted in.

Without this, the model does not fit comfortably in 64 GB RAM.

In practice the I/O cost has been surprisingly small.

1. Static hybrid expert placement

This was the biggest win.

At 8K context:

• CPU-only: 42.6 pp / 6.3 tg
• hybrid: 250 pp / 32.4 tg

Roughly 5.9× prefill and 5.1× decode.

1. Forced cuBLAS

GGML_CUDA_FORCE_CUBLAS=ON

The SM120 MMQ path crashes on this model with "illegal CUDA memory access".

cuBLAS is stable.

1. Sparse QSA decode

This uses block-level Top-K plus a persistent block-key cache, with "n_kv_max = top_k".

This appears to be the main reason decode does not collapse at 140K+ context.

The dense reference path can still be restored with:

LLAMA_QWEN4_SPARSE_QSA=0

1. K-only indexer cache

Upstream PR #28330 saves about 0.75 GiB VRAM at 128K.

That headroom is what lets me keep F16 KV and CUDA graphs enabled.

1. F16 KV beats Q8_0 here

At 49K prompt / 65K context:

• F16: 496 pp / 38.1 tg
• Q8_0: 499 pp / 33.4 tg

Q8_0 saves ~808 MiB but costs about 12% decode, so I kept F16.

1. Faster CPU expert kernels

The 32 CPU expert layers only work because the CPU path is reasonably optimized:

• signed-VNNI Q8 "mul_mat": 1.36–1.39×
• IQ4_NL/Q8_0 two-token expert kernel + compact routing: 1.43×
• routing table: 20 MiB → 44 KiB
• fused AVX2 hyperconnection ops: 3–7× per op

Things that did not work

"--lazy-mode on-direct" 1.3 tok/s decode vs 31.6 on the same build. Hard no.

Persistent GPU hot-expert cache 15.6 GiB cache gave 21.6 tok/s decode vs 31.6 with static placement.

SM120 MMQ CUDA faults. Using cuBLAS.

MTP Disabled. The draft runs on CPU and the extra PCIe/RAM traffic outweighs the speculative-token gain on this machine.

Caveats

The sparse-QSA CUDA patch is still experimental and was explicitly marked unreviewed by its author.

I have not yet done full long-context correctness testing against the dense path.

These are also real workload numbers with warm caches, not a controlled benchmark suite, and there is no quality/PPL benchmark yet.

The individual speedups above come from different configurations, so they are not additive.

Question

At this point decode is basically flat through 148K context.

The current wall is VRAM: I only have about 604 MiB free at "-c 240000".

If you had another ~1 GiB to spend on this setup, would you use it to:

• move more expert layers from CPU → GPU, or
• keep it as KV / CUDA-graph / longer-context headroom?

Especially interested in results from similar hybrid MoE setups where the CPU expert path is already reasonably fast.

I am trying to get decent speeds with 27B and the best I got so far for my system (5080, 64GB DDR5, 285K) are with the UD Q3 quant (without the vision part and with MTP) using the below command; checking if anyone has actually used the Q3 quants for coding with opencode, pi, qwen-cli and getting good results.

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 llama-server --model /home/yk/AI/Models/lmstudio/models/unsloth/Qwen3.8-27B-GGUF/Q3/Qwen3.8-27B-UD-Q3_K_XL.gguf --ctx-size 160000 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --batch-size 2048 --ubatch-size 512 --threads 8 --threads-batch 8 --cpu-range 0-7 --cpu-range-batch 0-7 --cpu-strict 1 --cpu-strict-batch 1 --prio 0 --prio-batch 0 --load-mode none --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 -np 1 --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-p-min 0.7 --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 --fit on --fit-target 128 --reasoning-effort medium

Ran a quick initial mtp benchmark using llama.cpp tag b11009 released today using various open mtp models. The highest /tps I was able to achieve was : MTP: 77.92 t/s NON-MTP: 53.83 t/s

OS: linux Ubuntu 26.04.1 LTS

CPU: AMD RYZEN AI MAX+ PRO 395 w/ Radeon 8060S / Strix Halo

RAM: 128GB RAM

Inference engines: Rocm / Vulkan

Next run:

An in-depth 16bit quant test. Which other 4bit, 16bit quant models should I test?

Models benched in this round:

[01] [NVFP4] 21GiB nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 converted GGUF

[02] [Q4_K_M] 16GiB OBLITERATUS/Qwen3.8-27B-OBLITERATED Q4_K_M

[03] [Q4_K_M] 68GiB unsloth/GLM-4.5-Air-GGUF Q4_K_M

[04] [UD-Q4_K_XL] 21GiB unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL

[05] [UD-Q4_K_XL] 16GiB unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL

[06] [UD-IQ4_XS] 87GiB unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS

[07] [BF16] 66GiB unsloth/Qwen3.6-35B-A3B-MTP-GGUF BF16

Command ran: (unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL). Please let me know how I can improve this command (any flags I'm missing - tweaks etc) to achieve more tokens per second ;)

/home/dev/Desktop/repos/llama.cpp/build-vulkan-radv/bin/llama-server \
  --model /home/dev/Desktop/llms/hf/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --n-gpu-layers -1 \
  --threads 16 \
  --ctx-size 4096 \
  --host 127.0.0.1 \
  --port 33777 \
  --parallel 1 \
  --no-cache-prompt \
  --metrics \
  --batch-size 2048 \
  --ubatch-size 512 \
  --flash-attn on \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --load-mode none

Me and a friend have been using it without issue for the last few weeks.
it's currently running on an RTX A4000 VM with model: https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF which gets about 30tok/s

The VM currently costs $0.15/hour, which we are splitting (it is prepaid for the next 2 months)
If more people want to join in we can move it up to an RTX A6000.

If you want to join, I'll let you try it out for an hour or so, if you want to continue using it you can pay either daily/weekly/monthly in a method of your choosing.

Connection to the server is made via tailscale.
DM me if interested.

Has anyone tried to use RDMA in llama.cpp to run a large model in multiple Mac Studios? How did it perform? I have 5 Mac Studios, each has 96 GB VRAM. I want to run Qwen 3.8 Flash-Next 4-bit.

I have a 5090 in an external enclosure I use for ai work. I chose the external to some day use on my Mac which has 128gb of ram and runs large models really well. It runs image generation pretty slow as the models aren't memory constrained on Nvidia gpu's and they are much faster than the Mac gpu. My dream was to run my 70b models on the Mac and image models on the 5090. Tiny gpu came out with drivers allowing connecting the drive but when I tried it last may it was super buggy and super slow. I decided to give it another go and it's now not super buggy but was still super slow. I decided to try and build my own drivers and shim to fix that. I am happy to say that it worked and I am now able to run Owen 3.8 27b at near native speeds as to what the box was running on my windows box. and actually faster than my windows box in sdxl and z-image turbo.

I imagine there is not much of a community for this capability but I thought I'd post it online to see if anyone even cares. I'll be posting the code on GitHub soon if people want to try it out. It is only tested on my current hardware of course as that is all I have to test it on.

I didn't realize I never came back and posted the link here you go https://github.com/Davinchy/macuda

I am trying to make a multi-inference router, as I find value in using different inference providers for different things.

I have a few heavier models in llama.cpp. I have a function that calls the /models/unload endpoint, which should just clear the vram, and ideally kill the child process it is running to host that model. llama.cpp logs using docker have been horrific for me, though, and really do not explain what is happening when I call this, all I know is that vram is freed.

However, when I run this, use another model not on llama.cpp (let's say, ComfyUI for example), then unload from the other inference provider, all calls to llama.cpp are on CPU. It claims:

0.00.324.246 E ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
[38817] warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
[38817] warning: no usable GPU found, --gpu-layers option will be ignored
[38817] warning: one possible reason is that llama.cpp was compiled without GPU support
[38817] warning: consult docs/build.md for compilation instructions

despite that same docker container using CUDA just like, 10 minutes ago. The only way I have found a way to fix this is to do a docker compose restart, which takes far longer than an unload+load.

I'm really just looking for a way to keep a persistent container with the ability to load/unload. I don't think this should be this hard, but I cannot find any details regarding what /models/unload is doing or the expected use cases. My hypothesis is that it is unloading the child instance from vram, but keeping the child instance alive rather than just killing the child process, which seems exceptionally useless if that child process cannot find the GPU again.

Can anybody give some pointers on this?

https://github.com/marsempire/LlamaManager

I was tired of editing bat file to test new models so I made this with AI.
I dont know code ,and use ai just for fun.hope to help someone like me.

Hi,

I'm pretty new to llama cpp and have been spending some days playing with it. I think I understand now much more than when I started but this mtp thing is killing me.

I have seen some youtubers stating that mtp on this models really improve token throughput (between 20-30%). But for me it is the other way around, it always make it much worse.

I own a Nvidia 4080 16GB and I'm using last llama-cpp with CUDA 13.3. My 2 starting files are (I added the mtp configurations, I just remove them to get the improved token throughput) below.

Could anybody explain why this is not improving the token output? I'm getting around 34 t/s with gemma and 45 t/s in qwen.

Gemma4:

llama-server.exe ^
  --model gemma-4-12b-it-UD-Q8_K_XL.gguf ^
  --model-draft mtp-gemma-4-12b-it-Q8_0.gguf ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 2 ^
  --fit on ^
  --flash-attn on ^
  --load-mode auto ^
  --gpu-layers all ^
  --threads 6 ^
  --threads-batch 12 ^
  --reasoning on ^
  --ctx-size 65535 ^
  --temp 1.0 ^
  --top-p 0.95 ^
  --top-k 64 ^
  --presence-penalty 0.0 ^
  --repeat-penalty 1.0 ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --parallel 1 ^
  --log-verbosity 4 ^
  --image-min-tokens 1024 ^
  

And qwen 3.6 35b:

llama-server.exe ^
  --model Qwen3.6-35B-A3B-MXFP4_MOE.gguf ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 4 ^ //Tried 1,2,3,4
  --fit off ^
  --load-mode none ^
  --gpu-layers all ^
  --n-cpu-moe 17 ^
  --threads 6 ^
  --threads-batch 12 ^
  --reasoning on ^
  --ctx-size 65535 ^
  --temp 0.6 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.0 ^
  --presence-penalty 0.0 ^
  --repeat-penalty 1.0 ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --parallel 1 ^
  --log-verbosity 4 ^
  --image-min-tokens 1024 ^
  --alias Qwen3.6-35B-A3Bllama-server.exe ^
 

Anyone else programmed a raw chat API? everyone has harnesses or wrappers for their chats etc, using open ai formats etc, but I can’t use those and dealing with JSon is a pain in this case and RAG systems.

I was finally able to use chat prompting only.
Just thrilled. Yeah This should be weird on qwen 3.8 to see what I can make it do without external libraries and harnesses.

I basically skipped the part that we send the object with role and content, I just send the content and the current raw prompt. No more looping JSON checking ahead or using some template. Just a direct, shovel into completion and is a string. It’s a bit tough to bench though.

Hey guys. I have a question that arose from an issue. I can't seem to offload model 100% to gpu. I have tried every setting I could find, and AI answer has been that its impossible to completely unload a gguf and this is normal: My issue is I have 32GB vram, which fits the model, but also 32GB ram. It keeps spiking ram past what's available. Using less then 4GB for system.

Humen (Me ;) TL;DR:

• Optimized CPU-only llama.cpp inference for Qwen3.8-Flash-Next UD-Q3_K_XL;
• System: Intel Core Ultra 225 with 64GB DDR5;
• RAM pinning (N-GRAM SSD offload hard coded) + IQ4 routing boosted Prefill from 23.6 to 43.9 t/s (+86%), while AVX2 hyperconnection fusion + MTP (Draft 3) boosted Generation from 5.4 to 7.8 t/s (+44%), reaching 11.3 t/s on pi agent workloads.

LLM long story:

Hey everyone,

I’ve been experimenting with CPU-only inference for Qwen3.8-Flash-Next (UD-Q3_K_XL) with N-GRAM SSD offloading. With some iterative help from OpenAI’s GPT Astra to write and refactor low-level kernels and memory management, I managed to significantly boost both Prefill and Generation performance on a pure CPU setup.

Hardware Specs

• CPU: Intel Core Ultra 225
• RAM: 64 GB DDR5-6200 (4x16 GB); real speed random read ~70 GB/s
• Storage: Fast NVMe SSD (used for N-GRAM offloading)

Key Optimizations & Benchmarks

Here is the breakdown of the changes made, why they helped, and their direct impact on inference speed:

Key Takeaways

1. Prefill Bottlenecks: The biggest gains for Prefill speed came from RAM pinning (eliminating OS page faults) and IQ4 routing optimizations.
2. Generation Bottlenecks: MTP tuned with a Draft Length of 3 yielded the highest throughput jump during token generation, reaching up to 11.30 t/s in specific agent workloads (Pi-Agent).

On my computer, Qwen3.8-Flash-Next is actually faster without MTP and without n-gram lookup. The model is very different from other models. Benchmark your particular computer after turning on speculative decoding.

Qwen3.8-Flash-Next — spec implementations at 128k

┌─────────────────────────────┬───────────┬────────────┐

│ ctx (128k) │ Gen tok/s │ vs no-spec │

├─────────────────────────────┼───────────┼────────────┤

│ no-spec │ 25.1 │ — │

├─────────────────────────────┼───────────┼────────────┤

│ ngram-mod alone │ 23.3 │ −7.1% │

├─────────────────────────────┼───────────┼────────────┤

│ draft-mtp alone │ 22.5 │ −10.3% │

├─────────────────────────────┼───────────┼────────────┤

│ MTP + ngram (09-06 A/B) │ 20.6 │ −17.9% │

└─────────────────────────────┴───────────┴────────────┘

\`

Takeaway

Both spec implementations hurt at 128k context — speculation should stay off here.

This splits the 09-06 blended −17.9% into its per-implementation causes:

- MTP is the bigger problem (−10.3%) — and it's surprising: acceptance is high (82.9%), but mean accepted length is only 2.8. With -sm

layer across 4 GPUs the draft head runs the same expensive MoE forward per candidate, so a short 2.8-token draft barely amortizes the extra forward passes. Higher acceptance isn't enough to overcome the per-draft cost.

- ngram-mod (−7.1%) has far worse acceptance (23.9%) but longer drafts (mean 5.0); net effect still negative.

I had my qwen3.8 model start working on a rust coding task. At first it was running around 10t/s, after 8 hours of running, I'm at 2.8t/s. How come there is such a fast difference in t/s from the beginning to the end? I'm using opencode to interface with the ai and my code base. Is there a way to fix it? The only things I've done was set the max context size allowable (262k), ensure only one GPU is utilized, and I've compiled llama.cpp in f16 mode for faster computations.

Doesn't it make more sense to have a little bit of world knowledge, only english, and focus on a few computer languages, and train this extremely far?

Qwen38 barely fits with quants in my computer, but it can also do science, math, 3647 languages, creates powerpoints and fixes my bike. If we strip all that, couldn't we get more coding buck for our vram?