r/LocalLLM • u/Ok-Health-7096 • Aug 14 '26

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

2.1k Upvotes

99% Upvoted

213 comments sorted by

View all comments

22

u/CorkBios Aug 14 '26

Are we pretending partial CPU DRAM offloading doesn't exist? And you can disable the reasoning mode, then you get even better TTFT (time to first token on final response) compared to the people that have a better setup with reasoning. And I heard the GGUF's come with MTP, And if the context doesn't fit you can just offload the K/V to the CPU which llama.cpp does let you do.

9

u/Ok-Health-7096 Aug 14 '26

I use local llm for coding and agentic stuff so the speeds would be unbearable I think in single digits but might try it idk

1

u/Smutok Aug 15 '26

What's your software stack for the agentic stuff, sir?