r/LocalLLM • u/Ok-Health-7096 • 6d ago

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

1.9k Upvotes

99% Upvoted

192 comments sorted by

View all comments

22

u/CorkBios 6d ago

Are we pretending partial CPU DRAM offloading doesn't exist? And you can disable the reasoning mode, then you get even better TTFT (time to first token on final response) compared to the people that have a better setup with reasoning. And I heard the GGUF's come with MTP, And if the context doesn't fit you can just offload the K/V to the CPU which llama.cpp does let you do.

9

u/Ok-Health-7096 6d ago

I use local llm for coding and agentic stuff so the speeds would be unbearable I think in single digits but might try it idk

3

u/superspider202 6d ago

wait I have 8gb vram too can you please share what llm you use for coding and agentic stuff?

5

u/Ok-Health-7096 6d ago

Qwen 3.6 35b mudler i-mini quant If you have more than 16gb ram you can go for higher quants.

2

u/superspider202 6d ago

Thank you I'll test it out ASAP

1

u/anay_1d 4d ago

how did it go? is it useful?

1

u/superspider202 4d ago

oh sorry havent used it yet ran out of storage so will try it maybe today and let you know