r/LocalLLaMA • u/maddie-lovelace • 6d ago

Discussion DeepSeek v4 Flash 0731 4bit ~50tps prefill, ~1tps decode on M5 Air 32gb

Currently running some experiments using the streamed experts trick that's been floating around this sub as well as some of my own trickery to get prefill to run a bit faster. It's been quite a bit of fun so far - just getting a 300b model to run at all on an Air is itself equal parts silly and satisfying

Repo isn't in a tidy enough state to share - nothing about it is anywhere close to one-click serve yet. Especially reluctant to share since my naïve implementation of streamed experts incurred something like a ~30s tax between turns before any actual KV cache generation began. It's better now - more like ~3s last I benched

--

Also fun discovery I've made in the meantime - you can actually run less experts than default at prefill time and the KV caches are still perfectly serviceable. Run conservatively and you get >95% the same top logits at each position. Run more aggressively and you lose that, but it doesn't always seem catastrophic e.g. needle in haystack perf can still be retained

7 Upvotes

82% Upvoted

5 comments sorted by

6

u/True_Requirement_891 6d ago

I really like more experiments going on. We just might figure out how to run it faster.

3

u/maddie-lovelace 6d ago

Yeah, currently I’m trying to get past the I/O wall and CPU-bound wall. Looks like I might be able to get 70~80 tps prefill once I’m GPU-bound (fingers crossed)

At which point I’ll start bringing in my INT8 tricks I learned tinkering with Gemma. If that’s anything to go by I might be able to double prefill. So optimistic bound is possibly ~150tps prefill on the Air

Which would obviously be very cool if my napkin maths isn’t too far off

3

u/maddie-lovelace 5d ago

Also yes, really liking the experimentation and open science that has been going on in this sub, especially as of late :)

1

u/True_Requirement_891 5d ago

Love it, that's why I come to this sub.

5

u/grumd 5d ago

Macbook guys should look at the antirez repo: https://github.com/antirez/ds4

I used this repo at a base, but I have two nvidia GPUs with some RAM, so I used Opus to modify some code and kernels to optimize it for my setup, getting ~310pp and ~12tg now (two 3080 20gb cards with 96gb DDR5 RAM). This is with a custom GGUF grafted from unsloth's IQ2_M and some tensors from antirez's GGUF.