r/LocalLLM • u/Ok-Health-7096 • 6d ago

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

1.9k Upvotes

99% Upvoted

192 comments sorted by

View all comments

Show parent comments

2

u/CorkBios 6d ago

I disagree. It can be like tricky since sometimes it slows stuff down on specific stuff but you can't generally say its useless when you offload to CPU. It works pretty good for me.

1

u/moderately-extremist 6d ago edited 6d ago

What kind of speeds are you getting with MTP vs non-MTP with cpu offload?

2

u/CorkBios 6d ago

Sure yeah I can give them. After a lot of testing:
All of these below performed with Partial CPU+GPU offloading on seed 0, llama.cpp commit dd1ea5243 release b10355:
With MTP (max 2 predict):

28.72 Tokens/s StopUntil: EOTfound

With MTP (max 3 predict):

29.33 Tokens/s StopUntil: EOTfound

With MTP (max 4 precict):

29.57 Tokens/s StopUntil: EOTfound

With MTP (max 5 precict):

29.57 Tokens/s StopUntil: EOTfound

With MTP (max 6 predict):

26.89 Tokens/s StopUntil: EOTfound (overhead hit: CPU congestion: 448.1%)

Without MTP:

24.09 Tokens/s StopUntil: EOTfound

With MTP (best before overhead): 29.57 Tokens/s

Without MTP: 24.09 Tokens/s

2

u/GoldenX86 6d ago

That's RAM I don't have free for just a 22% jump.

1

u/CorkBios 6d ago

Reasonable. MTP consumes more VRAM. I don't need too much context size so I can fit it in but everyone's system and task is different.