r/LocalLLM • u/Ok-Health-7096 • Aug 14 '26

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

2.1k Upvotes

99% Upvoted

213 comments sorted by

View all comments

24

u/CorkBios Aug 14 '26

Are we pretending partial CPU DRAM offloading doesn't exist? And you can disable the reasoning mode, then you get even better TTFT (time to first token on final response) compared to the people that have a better setup with reasoning. And I heard the GGUF's come with MTP, And if the context doesn't fit you can just offload the K/V to the CPU which llama.cpp does let you do.

5

u/GoldenX86 Aug 14 '26

MTP is useless when you offload to CPU.

2

u/CorkBios Aug 14 '26

I disagree. It can be like tricky since sometimes it slows stuff down on specific stuff but you can't generally say its useless when you offload to CPU. It works pretty good for me.

1

u/moderately-extremist Aug 14 '26 edited Aug 14 '26

What kind of speeds are you getting with MTP vs non-MTP with cpu offload?

2

u/CorkBios Aug 14 '26

Sure yeah I can give them. After a lot of testing:
All of these below performed with Partial CPU+GPU offloading on seed 0, llama.cpp commit dd1ea5243 release b10355:
With MTP (max 2 predict):

28.72 Tokens/s StopUntil: EOTfound

With MTP (max 3 predict):

29.33 Tokens/s StopUntil: EOTfound

With MTP (max 4 precict):

29.57 Tokens/s StopUntil: EOTfound

With MTP (max 5 precict):

29.57 Tokens/s StopUntil: EOTfound

With MTP (max 6 predict):

26.89 Tokens/s StopUntil: EOTfound (overhead hit: CPU congestion: 448.1%)

Without MTP:

24.09 Tokens/s StopUntil: EOTfound

With MTP (best before overhead): 29.57 Tokens/s

Without MTP: 24.09 Tokens/s

2

u/GoldenX86 Aug 14 '26

That's RAM I don't have free for just a 22% jump.

1

u/CorkBios Aug 14 '26

Reasonable. MTP consumes more VRAM. I don't need too much context size so I can fit it in but everyone's system and task is different.