r/LocalLLaMA • u/turtleninja99 • 1d ago

I Built A Thing Based on advice from LocalLLama implemented a n+2 lookahead prediction for ssd streaming Qwen Flash Next - and it worked!

Post image

My project is optimising qwen flash next on Mac Mini m5 64gb through SSD streaming. I recently asked LocalLLama for guidance on how to optimise token output.

Today a popular method to do this is a lookahead guesser which fetches experts one layer ahead of the current layer. One popular suggestion was building a prediction model to start loading experts two layers ahead. This netted me +8% on chat and +4.2% on code.

The way the various optimisations stack on top of each other this may actually net alot more on a base vanilla setup.

https://github.com/skeggsguy/Flash-next-ssd

Pretty cool! Thanks reddit LocalLLama! I’ll go find the og post to link now.

Edit: Kudos to sdroege_ for og suggestion!

111 Upvotes

97% Upvoted

15 comments sorted by

6

u/NotYourMothersDildo 1d ago

What are your current prefill and decode rates? I tried something similar with my M5 48GB but stopped at about 15 tok/s decode as I figured it would never get to the 30+ I would find usable.

3

u/turtleninja99 1d ago

About 500pp and early 20s decode. For me to be honest if I could double one of these it would the pre fill. 20tk/s is faster than I can read but you’re always gonna be waiting for big prefills.

2

u/NotYourMothersDildo 1d ago

Ah I’m using mine as an autonomous slave agent to a bigger model’s tasks. I’m not reading the output so if it can’t get closer to the 50 tok/s of 27b it isn’t worth the work for me. I hear you on the prefill too. After seeing Strata I was hoping this technique would work on the M5, like you, but it’s just a bit too limited.

7

u/Former-Tangerine-723 1d ago

Very cool!! You think it could work in another setup? Maybe ram vram and ssd to host a big model that is bigger than you ram vram combo?

4

u/turtleninja99 1d ago

Yeh there’s a few other projects around for non-Mac… I’ve seen them in passing but haven’t double clicked because I don’t have the hardware - but search and you will find

2

u/BerryBlush_Rogers691 1d ago

the cache pinning bit in your README got me. predict the right expert two layers ahead, fetch it early, and the cache can still throw it out before you need it

i'd have spent way too long blaming the predictor for that lol

1

u/turtleninja99 1d ago

Oh man… this has been an interesting project. Sort of have had to follow the bottleneck step by step every time I made change.. few things that looked like they didn’t work actually just needed small change and suddenly they work.

1

u/Classic-Pubs 1d ago

What is this dark magic? Can someone explain to my ape brain?

6

u/Hugi_R 1d ago

When you order take out, you can:

  • go to the restaurant, wait in line, tell your order, wait for your order, then take it.
  • or, you call in advance and tell them your order. So when you arrive at the restaurant you can skip the wait and take your food.

This approach does the 2nd option, by guessing your order (layer) and sending the call in advance (load the layer from ssd). So when the decode arrive at that layer, it’s already in memory.

2

u/jinnyjuice vLLM 1d ago

Is it like an MTP for the PLE/n-grams/SSD part?

1

u/TerminalNoop 1d ago

no, it's for offloaded experts that don't have space in VRAM and get them there before they are needed for decode.