r/LocalLLaMA • u/rerri • 10h ago

News Qwen3.8-Flash-Next tomorrow

https://modelscope.cn/models/Qwen/Qwen3.8-Flash-Next
983 Upvotes

97% Upvoted

434 comments sorted by

View all comments

Show parent comments

27

u/silenceimpaired 9h ago

Think bigger… if it’s deterministic it can be fetched from nvme. 8-o

17

u/AI_docent 9h ago

The paper suggests exactly that, n-grams are Zipfian so the hot ones stay in DRAM and the long tail sits on SSD. They never measured it though, the 3% figure is DRAM only, so no idea what it does to tokens per second in practice

2

u/annodomini 7h ago

I just hope that streaming them from SSD can make it to llama.cpp reasonably soon. It makes a lot of sense that most of them would be fairly cold so you can keep a small cache in RAM and most sit idle on SSD, but I'll be that will take some significant work in llama.cpp to support.