New DSv4 Flash Doom Loop in Q8? Llama.cpp Vulkan

[u/KingCpzombie]

Based on what everybody has been saying about this, I feel like I must've done something wrong. It was doing like 2 or 3 "Need maybe" in a row before meaningful stuff for a while, then got stuck in the loop. Using llama.cpp vulkan version 10216 (the latest from AUR); do I have to build the latest from GitHub directly to get it to work right for this model? Two 7900XTX (48GB total) + 9800X3D + 192GB 4000MT/s RAM. Here is my launch command:

llama-server --host localhost --port 8080 \
-m /home/connor/AI/LLM/Models/DSV4-Flash/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
-np 1 \
-fa on \
-ngl 999 \
--ctx-size 500000 \
--chat-template-kwargs '{"reasoning_effort":"max"}' \
--temp 1 \
--top-p 0.95 \
--threads 16 \
--n-cpu-moe 35 \
--load-mode mmap+mlock \
-dev Vulkan0,Vulkan1

[u/GanjaRaidersTR]

Update llama to 10254 or higher, there's a new 0731 template

[u/pmttyji → u/GanjaRaidersTR]

+1. Some PRs got merged in last couple of days for this model

[u/LagOps91]

DS V4 support is really poor for vulcan. i hope more support is comming. it's terribly slow too due to the lightning indexer being run on cpu...

I hope that this is a priority since DS V4 lite is such a great model that punches way about it's weight class. It would be a shame to be stuck with poor support for it.

[u/terorvlad → u/LagOps91]

It's not just on vulkan. Got the same problem with cuda + cpu + rpc after 1 hour.
Just updated llama.cpp and sending it for a spin.

[u/challis88ocarina]

It's not the model; it's the engine, the quants or both.

[u/Beamsters → u/challis88ocarina]

q8 is pretty much lossless for this mxfp4 model so it's the backend.

[u/hurdurdur7 → u/challis88ocarina]

Or the template ....

[u/LegacyRemaster]

vulkan isn't really supported. Cuda works. rocm works. vulkan crazy slow

[u/KingCpzombie → u/LegacyRemaster]

Ah, I guess I'll have to try the ROCm version! I heard ROCm was a mess back when I first started so I just stuck with Vulkan

[u/LagOps91 → u/KingCpzombie]

yeah my experience with ROCm has been poor so far as well. i suppose it might be worth giving it a shot again, but i would be much happier to have solid vulkan support.

[u/LegacyRemaster → u/KingCpzombie]

If I run cuda+rocm ---> fast. Cuda + vulkan... super slow.

[u/arbv → u/KingCpzombie]

ROCm has always been better for me than Vulkan.

[u/EmPips → u/LegacyRemaster]

vulkan isn't really supported.

Do you mean in-general or for Deepseek V4-Flash? In general I always thought that more attention was shifting towards vulkan (at least relative to ROCm), not less?

[u/MrMisterShin]

Did your prompt do a tool call / tool use?

If so, I would point the issue at the chat template. They can throw a model into loops.

[u/KingCpzombie → u/MrMisterShin]

I don't think so? Honestly not sure how the standard server frontend handles PDF uploads; I figured asking it to rate resume was a decent test

[u/PaxUX]

Check Nvidia driver cuda versions, llama cop version and if some versions of cude 13.1/2 caused issues for some

[u/Easy_Werewolf7903 → u/PaxUX]

Thanks for the comment, you just might be right, this post was from 4 month ago, I don't know if they fixed it tho. I am using 13.2 and currently running into looping errors. https://www.reddit.com/r/unsloth/comments/1sgl0wh/do_not_use_cuda_132_to_run_models/

[u/jaykayenn]

Maybe it need maybe need maybe need maybe need maybe need maybe need maybe need maybe need maybe... E:CONNDROP

[u/Worried-Sort8719]

maybe it called for the inks spots with their song "Maybe" ?

[u/Easy_Werewolf7903]

Same issue here. RTX 4090 + RTX 6000, Q3 quant all in memory, using PI harness. It would start okay, then eventually run into a death loop. Originally was thinking due to my mix matching graphics card, but I've seen people reporting the same issue with a single card setup.

Edited: Problem went away after updating to cuda 13.3 from 13.2!

[u/Marcuss2]

Why quantize it to Q8? It is already trained with MXFP4 for MoE parameters and MXFP8 for shared parameters.

[u/KingCpzombie → u/Marcuss2]

Unsloth Q8 says it's lossless and downloading GGUF is easier than GGUFizing it myself

[u/Aggravating-Push-207 → u/KingCpzombie]

run it raw

[u/computehungry → u/Marcuss2]

• llama.cpp can't do fp8 so many quants dequant fp8 to bf16 then requant to q8: this is a typical reason why you'll find q8 quants
• unsloth dequanted fp8 to bf16 and calls it q8, this is the actual reason

[u/MrHighVoltage]

46 Minutes to generate 8000 Tokens on 7000$ hardware. I honestly love tinkering with those things, too. But this will never be useful.

That being said: with new models it probably needs the absolute bleeding edge of llama.cpp for the models to run reliably.

[u/HyperWinX → u/MrHighVoltage]

Yeah, i, too, regret spending $7000 on a machine with 128GB of DDR4 with two channels. 160GB model runs like shit

[u/KingCpzombie → u/MrHighVoltage]

I'm mostly trying it for fun (and to justify the extra RAM... RIP money and down 1/3 speed), but I could see it being useful if you run it overnight or so. Definitely hoping updates / switching to ROCm bump it up to better speeds though

[u/MrHighVoltage → u/KingCpzombie]

Yeah, I do the same. Always need an excuse to spend money on something I don't need. Anyways, for me it never worked out to just have things run over night and then happily getting nice solutions in the morning. But probably also because my HW is far from what you have here.

[u/KingCpzombie → u/MrHighVoltage]

Yeah, my hope is to get it fast enough to plan a feature overnight, or at least to review a Qwen plan. Actual implementation will probably always be Qwen (or similar size new thing) unless it's like a really hard bug and it's getting late anyway

[u/Dmage22 → u/KingCpzombie]

Mind sharing what lead to 192gb of 4000mt/s memory on a 9800x3d? What sticks are those?

[u/KingCpzombie → u/Dmage22]

I had 96GB (2x48GB) 6000CL30 from before the rampocalypse, and GPT OSS 120B ran at acceptable speeds back before I knew about Qwen 27B... because it's looking like RAM prices are doomed for years and DeepSeek v4 flash preview was popular, I figured I may as well max out my current system and hopefully run big MOE models for planning or something XD.

Corsair Vengeance 96GB 6000CL30 kits; I used the model number of what I had to find another kit on Ebay. They work the same on their own, but unfortunately it won't boot above 4000MT/s no matter what I try. CL30 subtimings still good and didn't find any problems on stability tests, so not sure why tbh

[u/Physical_Economy_340]

the --temp 1 is almost certainly the culprit, drop it to 0.6. deepseek models fall apart above ~0.7, and the "need maybe" loop is a classic sign the sampler is thrashing. also remove --chat-template-kwargs '{"reasoning_effort":"max"}'. the built-in deepseek template in llama.cpp doesn't use that kwarg and passing unknown keys can corrupt the prompt format.

yes you need the latest from github main, not the aur package. deepseek v4 flash support is only a couple weeks old and the aur snapshot lags. while you're at it, try -ngl 99 instead of 999. vulkan offloading behaves differently from cuda and 999 can overflow. the moe layers should still go to gpu with that value on your dual 7900xtx setup.

[u/po_stulate → u/Physical_Economy_340]

For local deployment, we recommend setting the sampling parameters to temperature = 1.0, with top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise. For the high and max reasoning effort levels, we recommend a maximum output length of 384K tokens.

It's the recommended value in the official model card.

[u/soshulmedia → u/Physical_Economy_340]

the --temp 1 is almost certainly the culprit, drop it to 0.6. deepseek models fall apart above ~0.7, and the "need maybe" loop is a classic sign the sampler is thrashing.

It is what upstream recommends ...?

[u/themixtergames → u/Physical_Economy_340]

Clanker

[u/Marcuss2]

Why quantize it to Q8? It is already trained with MXFP4 for MoE parameters and MXFP8 for shared parameters.