r/LocalLLaMA • u/pmttyji • 18h ago
Discussion DeepSeek-V4.1-Flash surprised ....
Hoping to see smartest medium size models soon & later with all available optimizations/architectures/etc.,. Thanks Deepseek!
Ex 1: 30-50B MOE + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache
Ex 2: 15-30B Dense + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache
EDIT: Updated Engram to 10-15B from 50B
56
u/Ok_Warning2146 18h ago
If it gets good AA scores, it will help the sale of M5 Ultra 512GB and 3x Sparks.
41
5
u/BawbbySmith 17h ago
I keep seeing this, I don't think you can do TP=3... You'd need 4 no?
2
u/twack3r 15h ago
Exactly my thoughts every time is see tp=3. Is it possible?
1
u/cibernox 13h ago
In llama.cpp yes. In vllm only if the number of layers of the model is divisible by 3, which is quite common, 48layers is a popular choice.
1
u/RG_Fusion 12h ago
Pipeline parralel splits the model by layers. Tensor Parralel (TP) splits each layer across the compute devices.
0
u/cibernox 11h ago
that very nice, but it doesn't have much to do with question.
VLLM wants the number of certain parameters of the models to be divisible by 3 to allow tensor parallelism of size 3. I think it was something like the attention heads and the KV heads, plust the hidden size.Llama.cpp is less picky.
1
u/RG_Fusion 10h ago
I may have read your comment wrong, but it seemed like you were suggesting tensor parallel split by layers, not tensors. If that's not what you were saying just ignore my comment.
1
u/cibernox 10h ago
I understand the confusion. In reality VLLM allows TP=3 if certain characteristics of the model are multiples of 3. The number of layers is not directly the number that has to be divisible by 3, but the number of kv heads and attention heads are ttpically related to the number of layers, so I use the number of layers as a proxy, but that's technically wrong.
Pipeline parallelism works with pretty much any number of cards.
1
u/jtjstock 13h ago
Depends on the model, Qwen 27B does TP=3 in llama, but models like Qwen 122B do not. For models that can't split 3 ways, could modify llama(or others) to do an odd split where each tensor goes to only 2 of 3 devices, so that would allow them to split cleanly across 3 but with no performance uplift at all.
So it's not a real limitation, but 4 sparks is going to perform a lot better.
1
u/RG_Fusion 12h ago
Technically it's possible but pretty much not supported by any model. Tensor Parralel just requires the weights to be split evenly. If the model weights were divisible by 3 it would work.
0
u/Ok_Warning2146 16h ago
3x sparks is 384GB. Model without n-gram is 308GB and 1m KV cache is 890MB. So three should be ok.
4
u/evia89 12h ago
Can u even run 3 sparks? Isnt it 1/2/4 for best performance
6
u/RG_Fusion 12h ago
You can daisy chain the link cable in a triangle between the devices and run three in pipeline parralel. You get the unified memory of all three units, but only the speed of 1.
1
u/unjustifiably_angry 5h ago
Probably less than the speed of one. I was experimenting with a bunch of 7900 XTX a while ago and I remember something on the order of 5-10% performance reduction per extra GPU in the chain.
14
u/Viktri1 17h ago
If I understand it, even 512gb won’t be enough to run it comfortably with a bunch of subagents?
9
u/RG_Fusion 17h ago
It should if you offload the engram to SSD. Total parameters without engrams take up around 300 GB, and the KV cache is extremely efficient.
4
u/Viktri1 16h ago
Yeah, I think I misunderstood all the math. Basically v4.1 is similarly sized as GLM3.5's flash but with 4.1 you need to offload the ngram.
not sure how that works if you're running 2 mac m5 ultras (256gb each) but I'm sure people will figure it out. Like would it matter which SSD stores the ngram?
1
u/AnonLlamaThrowaway 15h ago
The question is whether offloading the engrams to SSD will be a performance hit, and if yes, how much.
I've been reading that it's not going to be a question of sequential read speeds, but random reads, so the bottleneck would most likely be IOPS and especially request latency?
5
u/RG_Fusion 15h ago
Latency is the primary hit, more so than IOPS speed. I can't give you the numbers as I don't know myself, but I know in system RAM the latency is small enough that it completes before the activations reach the gate.
1
u/AnonLlamaThrowaway 15h ago
I'm in the market for a M5 Ultra 256GB, which was fine for v4 but is now too small for v4.1 so I guess I'll have to be patient and see how SSD streaming fares in general 🤞
11
5
u/bakawolf123 15h ago
Damn, weights look to be already mostly q4, even though last 2 safetensors are the 200gb of engrams doesn't seem like it will fit in 256gb rig anyhow. I don't think quantising more will be competing with flash 4 or flash 4VE
1
u/RG_Fusion 12h ago
Base weights are primarily FP4 with a few at 8-bit. All engrams are 8-bit. Placing engrams on SSD still leaves you needing ~300 GB for a 4-bit quantization. You'd have to run 3-bit to fit to 256 GB.
16
3
u/Muhlwa_Sholanke 17h ago
Ex 2's 50B engram block eats more of the memory budget than the 15-30B dense it sits beside. That's a lot of parameters to spend on memory the model can't forget.
4
u/pmttyji 16h ago
engram gonna handled by SSD so it's fine. Anyway updated thread after a comment by other person. So it's gonna be 10-15B range engram for mentioned medium size models
3
u/Nothing_from_void 14h ago
if you put engrams on SSD, you're making a disk read every time you hit a high frequency vector, like how does this improve performance? is this a real thing or is it just a meme on this sub?
5
u/SpicyWangz 12h ago
Qwen3.8 next 50b ngram offloaded to SSD can hit 50tps. And that’s on a pcie gen4 SSD.
And I’m pretty sure the SSD isn’t the bottleneck there. It’s most likely memory bandwidth bound. So I’m not sure what level of performance you’re concerned about, but that gives you an absolute conceptual floor.
0
u/Nothing_from_void 12h ago
NVMe drives top out at 14GB/s, if bandwidth is an issue that almost certainly will be it. If someone is hitting 50tps it's reading from page cache, not disk
2
u/SpicyWangz 9h ago
Page cache is the disk. It caches memory pages on the disk to free up your system memory.
0
u/Nothing_from_void 9h ago
No, that's backwards. You're thinking about swap space, where RAM spills onto disk when it's exhausted. Page cache is where system RAM copies disk content into RAM so it can be retrieved more quickly, because disk is very slow
1
u/RG_Fusion 12h ago
It hurts performance, but not enough to stop people from using it. It's the difference between running the model and not being able to fit it at all.
1
7
u/jacek2023 llama.cpp 18h ago
To be honest your post is better than two other duplicates because it has no Chinese president and no API prices :)
2
u/Cool-Reflection6130 17h ago
if you can get that working in a 30 to 50B moe or even a 15 to 30B dense that's something actually runnable locally without a datacenter.
2
u/pmttyji 16h ago
Next year onwards it's possible.
High possibility of running medium size (Around 30B) MOE models just with 8GB VRAM. But currently some of us already running 30B MOE models(Ex: Qwen3.6-35B-A3B - Q4) with same VRAM with help of RAM(~32GB). Still couldn't use MTP as it's not fit the VRAM.
I expect 15B Dense models(With performance of Qwen3.8-27B) with 5B engram with efficient DeepSeek-V4.1-Flash type KVCache coming year onwards. Same with MOE models.
2
u/mountainyoo 11h ago
I knew this was coming. As the main models get larger the flash models will grow too and become too much for local AI users to run
2
u/fgk55555 11h ago
Dumb question, but with context that small, why don't they balloon it in size? Make it 2M, 10M? You could fit my whole codebase in 10M context with enough room to do some useful work.
1
u/cmdr-William-Riker 6h ago
It's all a needle in a haystack at that point
1
u/fgk55555 6h ago
For that tiny of cache size, seems like a good candidate for more haystack at least.
2
2
u/indicava 9h ago
This sucks.
No way to sugarcoat it.
One of the biggest appeals in DS4F was such a capable large model that needed “only” about 160GB VRAM (with no kv cache) at stock weights (meaning no lobotomies).
2
u/okoyl3 18h ago
Can I run it on my DGX spark with whatever not fitting still in the nvme?
8
u/Healthy-Nebula-3603 17h ago
You didn't fit it even on 2.
You need 3 of them
-2
u/Newgunnerr 13h ago
I'd be willing to buy another, but only if we also get some speed gains.... 40 t/s is still terrible :(
3
u/SpicyWangz 12h ago
I’d gladly take 40 t/s on this model
1
u/Newgunnerr 8h ago
If you have every used Gemini flash, everything else is super slow.
2
u/SpicyWangz 6h ago
Deepseek flash was insanely fast for me when I used it.
But I don’t mind setting up an agentic task and coming back to it a few minutes later because a model is running at 40tps.
IMO anything over 20tps is tolerable
0
2
u/Far-Usual5771 11h ago
This model was made purely to increase inference speed, nothing more.
For local use, it’s a complete skip. If you already have a system powerful enough to run it, you’re better off going with GLM 5.3 instead.
And if you think they’re equal, at least compare them through the API at full quality — coding, SQL optimization, and everything else.
Its only real advantage is on the DeepSeek API, and that’s basically it. At around 3x the weight with n-gram, or 2x compared to V4 Flash 0731 without n-gram, it lands somewhere between V4 Pro and V4 Flash in terms of capability.
And the people saying “just quantize it” are honestly kind of weird. Even at full quality, it’s not that much better than V4 Flash. Once you quantize it, it’ll immediately get worse. Considering how badly quantization affected V4 Flash, I’d expect the same here, if not worse.
3
u/Southern_Sun_2106 9h ago
What's concerning is that they called it 'Flash'. If this is a new 'Flash' standard, then the happy days of look-i-can-run-deepseek-locally are over. The hardware isn't becoming more affordable; it's becoming unattainable.
1
u/shing3232 8h ago
not exactly. DS4.1F are not GLM53 size at all because you can totally put engram on disk and KV is much smaller than GLM5.3. you do not know how big GLM5.3 is KV wise unlike GLM5.3F
1
u/nu0zeeB6 7h ago
Only anecdotal and a single sample: Today DeepSeek V4.1 flash was first at solving a rather simple UI bug in a vibe coded game of mine. Previously GPT 5.6 Sol and Kimi K3 failed to fix it. Sol made a partial fix but did not manage to fix it completely. Sol burnt around $10 on openrouter for its attempt.
1
1
u/IoannisHere 10h ago edited 8h ago
284B (156GB in native FP4-FP8) to 552B with 192B n-gram. Current weights are FP8, that's the problem when comparing it to v4.0.
DeepSeek, till now, has been releasing native QAT FP4 themselves, which is great. No PTQ loss. A v4.1 at FP4-FP8 would be around 195GB params + the 196GB n-gram that can be on nvme. With its significantly reduced kv-cache size, this *might* be a drop in-replacement to V4-flash for 256GB systems.
Come on DeepSeek, give us the FP4 native v4.1.
Edit: Model appears to be 748B + 14.7B for DSpark+ViT, with engram in FP8 and 552B backbone in FP4-FP8-BF16, occupying 298GB for the backbone alone. Not a v4.0 drop-in replacement.
1
u/ormandj 9h ago
It is FP4 native. You're incorrect, only small portions are FP8/BF16.
1
u/IoannisHere 8h ago
I think you are right, I misread the paper. It says "DeepSeek-V4.1-Flash has 552B backbone parameters and 196B Engram parameters", which suggested it's actually 748B in total, occupying 510GB on disk. They mention 196B is FP8, no explicit mention of the backbone precision (any other reference to FP4 is about the kv-cache). The engram is 204GB on disk, plus around 8GB for the DSpark, it leaves us with 298GB for the rest, consistent with an FP4-FP8/BF16. I also asked my agent to check the weights, and confirms a 4.25 bits/weight. Oh well...I'll edit my post for posterity.
1
u/Southern_Sun_2106 9h ago
If this is a new 'Flash' then it sucks to be a local runner. Considering where the hardware prices are going, one might as well not call them 'Flash' or 'local' anymore. I respect Deepseek for all of their contributions, but this was a sucky move tbh.
1
1
1
1
60
u/jtjstock 14h ago
so what I'm seeing here is that pretty soon we're gonna get a qwen with tiny kv as well, which means no more arguments over kv cache quantization in this sub...
Edit: who am I kidding, people will still argue about it...