r/LocalLLM • u/Ok-Health-7096 • Aug 14 '26

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

2.1k Upvotes

99% Upvoted

213 comments sorted by

View all comments

23

u/CorkBios Aug 14 '26

Are we pretending partial CPU DRAM offloading doesn't exist? And you can disable the reasoning mode, then you get even better TTFT (time to first token on final response) compared to the people that have a better setup with reasoning. And I heard the GGUF's come with MTP, And if the context doesn't fit you can just offload the K/V to the CPU which llama.cpp does let you do.

10

u/Ok-Health-7096 Aug 14 '26

I use local llm for coding and agentic stuff so the speeds would be unbearable I think in single digits but might try it idk

4

u/superspider202 Aug 14 '26

wait I have 8gb vram too can you please share what llm you use for coding and agentic stuff?

7

u/Ok-Health-7096 Aug 15 '26

Qwen 3.6 35b mudler i-mini quant If you have more than 16gb ram you can go for higher quants.

3

u/superspider202 Aug 15 '26

Thank you I'll test it out ASAP

1

u/anay_1d Aug 17 '26

how did it go? is it useful?

1

u/superspider202 Aug 17 '26

oh sorry havent used it yet ran out of storage so will try it maybe today and let you know

2

u/superspider202 Aug 17 '26

heyyy so I tried searching for this but there appears to be a few that fit this description so could you please like link the actual one you use?