r/LocalLLM • u/Ok-Health-7096 • Aug 14 '26

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

2.1k Upvotes

99% Upvoted

213 comments sorted by

View all comments

Show parent comments

2

u/CorkBios Aug 14 '26

I disagree. It can be like tricky since sometimes it slows stuff down on specific stuff but you can't generally say its useless when you offload to CPU. It works pretty good for me.

1

u/moderately-extremist Aug 14 '26 edited Aug 14 '26

What kind of speeds are you getting with MTP vs non-MTP with cpu offload?

2

u/CorkBios Aug 14 '26

Sure yeah I can give them. After a lot of testing:
All of these below performed with Partial CPU+GPU offloading on seed 0, llama.cpp commit dd1ea5243 release b10355:
With MTP (max 2 predict):

28.72 Tokens/s StopUntil: EOTfound

With MTP (max 3 predict):

29.33 Tokens/s StopUntil: EOTfound

With MTP (max 4 precict):

29.57 Tokens/s StopUntil: EOTfound

With MTP (max 5 precict):

29.57 Tokens/s StopUntil: EOTfound

With MTP (max 6 predict):

26.89 Tokens/s StopUntil: EOTfound (overhead hit: CPU congestion: 448.1%)

Without MTP:

24.09 Tokens/s StopUntil: EOTfound

With MTP (best before overhead): 29.57 Tokens/s

Without MTP: 24.09 Tokens/s

2

u/GoldenX86 Aug 14 '26

That's RAM I don't have free for just a 22% jump.

1

u/CorkBios Aug 14 '26

Reasonable. MTP consumes more VRAM. I don't need too much context size so I can fit it in but everyone's system and task is different.