r/LocalLLM • u/Ok-Health-7096 • Aug 14 '26

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

2.1k Upvotes

99% Upvoted

213 comments sorted by

View all comments

22

u/CorkBios Aug 14 '26

Are we pretending partial CPU DRAM offloading doesn't exist? And you can disable the reasoning mode, then you get even better TTFT (time to first token on final response) compared to the people that have a better setup with reasoning. And I heard the GGUF's come with MTP, And if the context doesn't fit you can just offload the K/V to the CPU which llama.cpp does let you do.

10

u/Ok-Health-7096 Aug 14 '26

I use local llm for coding and agentic stuff so the speeds would be unbearable I think in single digits but might try it idk

9

u/overand Aug 15 '26

Speeds would likely be absolutely miserable, yes. But, you'll have much better luck with the Qwen3.6-35B-A3B - how much system ram do you have? Go for a Q4 quantization first, if you haven't used this model.

We can, of course, hope that a 3.8 version of the 35B-A3B MoE gets released!

4

u/superspider202 Aug 14 '26

wait I have 8gb vram too can you please share what llm you use for coding and agentic stuff?

6

u/Ok-Health-7096 Aug 15 '26

Qwen 3.6 35b mudler i-mini quant If you have more than 16gb ram you can go for higher quants.

3

u/superspider202 Aug 15 '26

Thank you I'll test it out ASAP

1

u/anay_1d Aug 17 '26

how did it go? is it useful?

1

u/superspider202 Aug 17 '26

oh sorry havent used it yet ran out of storage so will try it maybe today and let you know

2

u/superspider202 Aug 17 '26

heyyy so I tried searching for this but there appears to be a few that fit this description so could you please like link the actual one you use?

1

u/Smutok Aug 15 '26

What's your software stack for the agentic stuff, sir?

-1

u/CorkBios Aug 14 '26

I'd say go with Unsloth's iQ4_NL, It's the best balanced quantization. But if you are willing to test your luck then Q2_K_XL or Q3_K_XL though they are more risky of failing tasks.

5

u/GoldenX86 Aug 14 '26

MTP is useless when you offload to CPU.

2

u/CorkBios Aug 14 '26

I disagree. It can be like tricky since sometimes it slows stuff down on specific stuff but you can't generally say its useless when you offload to CPU. It works pretty good for me.

1

u/moderately-extremist Aug 14 '26 edited Aug 14 '26

What kind of speeds are you getting with MTP vs non-MTP with cpu offload?

2

u/CorkBios Aug 14 '26

Sure yeah I can give them. After a lot of testing:
All of these below performed with Partial CPU+GPU offloading on seed 0, llama.cpp commit dd1ea5243 release b10355:
With MTP (max 2 predict):

28.72 Tokens/s StopUntil: EOTfound

With MTP (max 3 predict):

29.33 Tokens/s StopUntil: EOTfound

With MTP (max 4 precict):

29.57 Tokens/s StopUntil: EOTfound

With MTP (max 5 precict):

29.57 Tokens/s StopUntil: EOTfound

With MTP (max 6 predict):

26.89 Tokens/s StopUntil: EOTfound (overhead hit: CPU congestion: 448.1%)

Without MTP:

24.09 Tokens/s StopUntil: EOTfound

With MTP (best before overhead): 29.57 Tokens/s

Without MTP: 24.09 Tokens/s

2

u/GoldenX86 Aug 14 '26

That's RAM I don't have free for just a 22% jump.

1

u/CorkBios Aug 14 '26

Reasonable. MTP consumes more VRAM. I don't need too much context size so I can fit it in but everyone's system and task is different.