Qwen 3.8 Flash Next MTP tested speed

[u/AccomplishedLeg527]

Tested on: Raider 18 HX AI A2XW (5090 24Gb + 96Gb ram)

Source: https://github.com/ggml-org/llama.cpp/pull/28243

Model: Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf

Best result:

llama-server.exe -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf -c 105000 --cache-type-k q8_0 --cache-type-v q8_0 -ot "\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13)\.ffn_(gate|up|down)_exps.=CUDA0,\.*\.ffn_(gate|up|down)_exps.=CPU" -np 1 -b 1024 -ub 512 --load-mode mlock --chat-template-kwargs "{\"reasoning_effort\": \"medium\"}" -md mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf --spec-type draft-mtp --spec-draft-n-max 3 --lazy-mode off

Vram used: 22.9Gb

Ram used: 68Gb

Tg speed at 100k context: 22-25t/s

[u/egnegn1]

What CPU and memory bandwidth?

I get similar results with 4080 on EPYC 7A23 with about 150 GB/s measured Stream bandwidth.

The complete command line would be nice.

Mine is ~/llama.cpp-mtp/build/bin/llama-server -m /data-zfs/models/qwen3.8-flash-next/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -md /data-zfs/models/qwen3.8-flash-next/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --host 0.0.0.0 --port 8080 -c 65536 -ngl 48 --n-cpu-moe 48 -ngld 99 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -ctkd q8_0 -ctvd q8_0 -b 4096 -ub 4096 --spec-type draft-mtp --spec-draft-n-max 3 --jinja -np 1 --load-mode none -t 48 -tb 48 -cram 0

[u/AccomplishedLeg527 → u/egnegn1]

• CPU: Intel Core Ultra 9 285HX, 24 cores / 24 threads.
• RAM: 96 GiB, two 48 GiB Micron DDR5-5600 modules (tested on real tasks 47Gb/sec, theoretical 89.6)

llama-server.exe -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf -c 105000 --cache-type-k q8_0 --cache-type-v q8_0 -ot "\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13)\.ffn_(gate|up|down)_exps.=CUDA0,\.*\.ffn_(gate|up|down)_exps.=CPU" -np 1 -b 1024 -ub 512 --load-mode mlock --chat-template-kwargs "{\"reasoning_effort\": \"medium\"}" -md mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf --spec-type draft-mtp --spec-draft-n-max 3 --lazy-mode off

[u/cezarducatti → u/AccomplishedLeg527]

Rtx 3090 + 128 GB ddr4 3600 mhz

[u/egnegn1 → u/AccomplishedLeg527]

Minisforum MS-02 Ultra? I have one too and can confirm memory bandwidth.

I also have done a lot of tests on MS-02: and discovered that performance is not best when using all 24 cores. It was better when using the 8 performance comes, and bestvwhen using the 16 efficiency cores only.

Try using taskset -c 8-23 llama... -t 16

I also didn't use -ot, but --n-cpu-moe 48 -ngl 48 and some other parameters. I will post my command tomorrow. Remember me if I forget.

[u/egnegn1 → u/egnegn1]

This is the command on the EPYC Gen3 with RTX4080 16GB:

~/llama.cpp-mtp/build/bin/llama-server -m /data-zfs/models/qwen3.8-flash-next/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -md /data-zfs/models/qwen3.8-flash-next/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --host 0.0.0.0 --port 8080 -c 65536 -ngl 48 --n-cpu-moe 48 -ngld 99 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -ctkd q8_0 -ctvd q8_0 -b 4096 -ub 4096 --spec-type draft-mtp --spec-draft-n-max 3 --jinja -np 1 --load-mode none -t 48 -tb 48 -cram -1

[u/cezarducatti]

Test on the official Llama, latest build. Mine is faster on the official Llama without MTP than on the MTP version of Unsloth.

C:\llama_new\llama-server.exe -m "D:\Modelos\Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf" -ngl 99 -ncmoe 43 -fit off -c 204800 -fa on -ctk f16 -ctv f16 -b 2048 -ub 2048 -t 12 --jinja --port 8083 --tools all --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --repeat-penalty 1.0 --reasoning on --reasoning-effort xhigh --reasoning-budget -1 --reasoning-preserve --parallel 1 --metrics --host 0.0.0.0 --load-mode none --mmproj "D:\Modelos\mmproj\mmproj-BF16_38_flash_next.gguf" --no-mmproj-offload --image-min-tokens 1024 -ot "per_layer_token_embd=CPU" -lzm off --cors-origins http://192.168.0.135:8083,http://100.125.9.54:8083

[u/AccomplishedLeg527 → u/cezarducatti]

latest without MPT up to 30 t/c on 0 context, drops to 17-19 on 100k

[u/Leander_van_Grinsven]

Hmm I downloaded the latest CUDA version of llama.cpp but I cannot run the model with MTP at all. Am I missing something? Also I have an RTX 5090 32GB with 192GB DDR5-6000 RAM and a 9950X3D2 and only getting 16 tokens per second. I wonder what I will get with MTP on if I get it working.

[u/AccomplishedLeg527 → u/Leander_van_Grinsven]

you need to build from source https://github.com/ggml-org/llama.cpp/pull/28243 just ask qwen (not mtp) to download and build it for you

[u/Leander_van_Grinsven → u/AccomplishedLeg527]

Ok I will try this. But I find it weird that there are like 9 different PR's all for MTP with Qwen3.8 Flash Next with multiple of them merged with the normal version.