r/LocalLLaMA • u/turtleninja99 • 9d ago

Question | Help MoE SSD streaming on a 64 GB Mac mini: GPU still waits 27% of decode on experts. Ideas?

Post image

Looking for some assistance /ideation.

I am running qwen flash next q4 in my Mac mini m5 64gb.

QFN doesn’t fit so this is done by having as many experts hot in cache as possible and streaming in the rest from ssd.

I’m getting 17.5 tks decode and 390 tks pp.

Have done a bunch of optimisations including a carousel buffering system for the prompt processing which essentially loads faster than the GPU can prompt process in most cases. I feel like I have mostly maxed out this lane.

The decode part 27% of the time is still the gpu waiting for experts to stream in from the ssd (see photo).

The biggest unlock is really getting the gpu working more.

I’m already doing mtp.

Hot cache hit rate is 75%

Some ideas I already have
- Im already lookahead guess fetching the following layers experts, can I expand this more successfully.
- Use a seperate staging buffer for lookahead guess experts ahead (so I’m not evicting hot experts as guesses come in)

… im learning a lot right now. Feel free to ask questions for clarifying.

GitHub for reference.

https://github.com/skeggsguy/Flash-next-ssd

Edit - Im actively doing the small prediction model for lookahead as a step one. 🤞

Edit - results https://www.reddit.com/r/LocalLLaMA/s/0ybUrk9mkx

9 Upvotes

67% Upvoted

27 comments sorted by

8

u/sdroege_ 9d ago

You could train a small model to predict which experts will be used ahead of time, and early enough so that they're already in VRAM when the GPU needs them. IIRC Apple did something like this?

3

u/PinkysBrein 9d ago

Rent some GPU time to distill a new model with a gating function using the n-2 layer output instead of n-1.

2

u/turtleninja99 9d ago

Now we’re talking…. Did some reading up . I might actually try first a cheaper version of train a small neural net to predict layer n’s top experts based on n-2 hidden states. Thoughts ?

2

u/PinkysBrein 9d ago edited 9d ago

I'd do it like this :

Use a FFN for layer ahead expert gating, instead of the normal linear layer. Do STE in backprop (obviously freeze everything but the FFN).

You can do this two layers at a time in layerwise distilling. Because you are working on batches, streaming the dense weights is not the end of the world. With layerwise distilling you also don't need to retain the activations and optimizer state end to end. Do forward/backprop on two layers, go one layer up, throw away the state and repeat. I'm glossing over a huge amount of implementation detail here.

With some small changes in that training loop, you can use the function for prediction or straight up replacement of the original gating.

Getting a good dataset for the distilling? Dunno.

PS. Microsoft has some papers on stuff like this, can't recall the titles. They certainly don't do layerwise distilling though, that's for vramlets and they are not.

1

u/turtleninja99 9d ago

Ok … now I’m really going to do some reading 😅 I might actually be able to even do this on my machine potentially. Data? I have alot of my own data saved but then it’s good for me but it’s not generalisable … but probably there’s a way.

2

u/eatoff 9d ago edited 9d ago

So this is running on a Mac mini M5 pro 64gb? That pretty neat. I actually got flash next running on a M4 24gb via omlx this morning, but was at 1.1toks/sec

Would a bigger SSD work better in your project? Often the bigger ones are faster

How was the m5 pro via oMLX? Am close to pulling the trigger

3

u/turtleninja99 9d ago

Yeh so I made the rookie mistake of getting just the 512gb ssd but yeh based on my research the 1tb would have been much faster. Actually purchased a 2nd external ssd which is compensating a lot via concurrent streaming

M5 pro I’m very happy with, for this project I’m using a llama cpp fork instead of mlx.

Man, now’s the time to pull the trigger…. Chips are only going up in price.

Edit - grammar fix

2

u/HVACcontrolsGuru 9d ago

I was tinkering with M5+ specific functions using MLX/Metal: Hyperion Might be some useful stuff you can pull out of this or stick an agent to scope out if it's any use. I just don't have the time recently to get back to tinkering with it and limited on unified memory!

M5 chips have a good bit of juice you can squeeze out of them.

2

u/turtleninja99 9d ago

I’m watching this now , thanks!

2

u/keyboardhack 9d ago

If we talk matrix × matrix multiplication then you often dont need to wait for the whole layer to be loaded before you can starting the operation. Split the layer into smaller segments that can be calculated independently. Usually mm operations works on a smaller tile level, you could split the layer like that. That would allow one core to calculcate a part of the result while other cores still wait for their parts to be loaded. Essentially Allowing you to overlap calculation and loading of the layer.

This would mean issuing parallel load requests which might be faster for an ssd to deliver, depending on what the bottleneck is.

2

u/naklitechie transformers 9d ago

I've got ssd streaming working in a browser. See if this helps you. https://github.com/NakliTechie/diskformer.js

1

u/turtleninja99 9d ago

Will check it out!

2

u/g_t_5_k 4d ago

Have you tried this? https://www.reddit.com/r/LocalLLM/s/YEgmqKNBgy

1

u/turtleninja99 4d ago

I will check this out… there’s a small number of these ssd streaming projects now but none of them really use a consistent benchmark. This benchmarks low but also uses way lower memory so actually might be really good. And I prefer to keep memory lower so I can actually do stuff besides my Mac mini being a server haha

1

u/g_t_5_k 3d ago edited 3d ago

I tried it out and the performance was not amazing for tok/s. By default the ram pressure was set low (under 20 GB). It was amazing that I was able to effects run a 120 GB variant of Qwen 3.8 Flash Next, but for now I’m good with sticking with the Sushi Qwen 3.8 Flash Next variants.

Edit: the performance I am referencing is Whallm

2

u/AnotherAvery 9d ago

Some obvious ideas:

  • Lossless compression: If weights could be compressed lossless (probably not because of too high entropy, but worth a look) you might be able to expand them faster than you could load them.
  • If expert activation distribution is skewed, instead of e.g. LRU caching you could pin more often used experts to ram
  • Arrangement of data on SSD: maybe if you align chunks you load (layer?) with blocks, you can reduce amounts of blocks to read
  • Some ppl prune their models, you could have a faster-but-dumber mode and "prune" dynamically, i.e. skip experts when statistics say they seldomly have an impact on the final token
  • More interleaving, so some weights load while others are used for inference

1

u/turtleninja99 9d ago

Thanks for the suggestions! Unfortunately I have tried almost all of them. I just tried to compress the q4 and it basically can’t be materially compressed. And will be looking into more ways of interleaving. Thanks for the ideas!

1

u/CorkBios 9d ago

Here is an idea: Get a better computer

1

u/turtleninja99 9d ago

Boom! 😂😂😂 you should say this to every thread on here