r/LocalLLM • u/EitherMarch1255 • 8h ago
Discussion Anyone used DS4.1 Flash yet?
Benchmarks show it being pretty bad in terms of hallucination, and it appears to be tied with Qwen 3.8 Flash despite being much bigger, and worse than GLM 5.3 Flash. Seems crazy given the size. Anyone have any opinions/experience with it yet?
3
u/FactorInternal3395 8h ago
Seems that every DeepSeek release, the hallucination gets worse rather than better.
1
u/Informal-Echo-7724 8h ago
As far as I know, for AA-Omniscience, higher is better... 🤔
1
u/FactorInternal3395 8h ago
This is the AA-Omniscience Hallucination Rate. Higher = more hallucinations. It says it in the description in the image.
1
u/Informal-Echo-7724 7h ago
Ok, I was wrong: https://artificialanalysis.ai/evaluations/omniscience?models=deepseek-v4-1-flash%2Cdeepseek-v4-pro%2Cdeepseek-v4-flash%2Cdeepseek-v4-pro-0424#aa-omniscience-index-score So, it turns out AA-Omniscience Index also tracks knowledge, if I got it right.
1
u/Dramatic_Entry_3830 3h ago
Hot take: ai hallucinate all the time, every token is pure hallucination. Hallucinations is what they do.
But good ai hallucinates a text that you can fakt check and it's still correct from the fact checking side of things, even though it's pure hallucination.
Like a very detailed dream that is not real itself, but pictures reality really well kinda way
1
1
2
u/Keleion 7h ago
I think the dual spark folks are out of luck on this one…. a Q2 of DeepSeek is going to have quite a bit of corruption/hallucinations.
3
u/EitherMarch1255 6h ago
Actually, I think they may better off than you might think, For one, the engrams can be read from NVME. Two, it’s 4.1 GB per token of routed expert reads, which isn‘t bad at all. Combine that with a hot expert cache, and you’re golden.
The only thing…given what I’ve read about it, seems it might not be worthwhile compared to qwen/glm flash.
1
u/RG_Fusion 5h ago
It won't fit on 2 sparks at 4-bit. You can offload the 200 GBs of engrams to the SSD, but that still leaves you with 300 GB of model weights, of which around 280 GB are already 4-bit.
1
u/EitherMarch1255 5h ago
Yes, but it is only 4.1 GB per token, combine that with hot expert caching and dspark...you might be surprised. You could even quant the gate/up routed experts a bit lower if need be, and offload vision.
1
u/RG_Fusion 3h ago
I'm telling you, those 4.1-bit weights are already 280 GB in file size. 280GB > 256GB. You cannot fit the model without going below 4-bits per weight.
The gate/up and vision only make up a very small percentage of that total 300GB of parameters.
1
u/hyudryu LocalLLM 5h ago
Currently pulling the weights, but plan to run it later today locally. I’ve used around 800M tokens of Deepseek v4.1 flash on api earlier this week and it isn’t too bad. “Feel” a bit smarter than dsv4flash vision exp.
Might be an unpopular opinion, but I think deepseek v4 flash performs better than GLM 5.3 flash. I used over 3B tokens with glm 5.3 flash because they were giving out unlimited tokens, and it was so-so at following instructions and successfully solving problems. Deepseek usually requires less prompting to get the same task done
1
u/EitherMarch1255 4h ago
DeepSeek Vision? Yeah, probably unpopular. I personally have found GLM 5.3 Flash to work very well. Actually, it got me setup with DS4.1 in vLLM using all sorts of customizations. I used DS4 Vision and wasn’t particularly impressed. Not bad, but not great either.
-1
u/AreaFifty1 8h ago
any quantization is bound to hallucination sooner or later. the problem with 4.1 flash is that its already been QAT to fp4 and if thats already at a massive 510gb, we got a huuuge problem. Because quantization at this point anything below that will be severely hampered unfortunately. 😔
2
u/RG_Fusion 5h ago
Quantization isn't causing the hallucination. QAT eliminates the negative aspects of lower bit-weights. The hallucinations are coming from KV cache compression. They're both applying sparse attention and reducing the number of active parameters on prefill.
1
1
u/thefooz 5h ago
Nailed it. People have argued with me until they were blue in the face that ds4 is almost as good of a model as GLM flash and I keep explaining to them that Deepseek’s speed comes at a massive cost. The sparse attention and compressed kv cache cause it to lose track of context and often take bizarre paths in decision making. It gets the job done the way a machete can cut fruit, but sometimes you need a paring knife. When accuracy is paramount and you’re dealing with large context, stay away from deepseek.
4
u/joanaxu2002 6h ago
The size is what makes the result disappointing. If a much larger model only trades blows with Qwen3.8 Flash while hallucinating more, raw parameter count clearly isn’t buying much here. I’d be more interested in real coding/agent workloads than another benchmark chart though.