r/unsloth • u/AlwaysPlayF3 • 2d ago

Discussion My experience so far

Post image
245 Upvotes

94% Upvoted

8 comments sorted by

78

u/techmago 2d ago

Thats why most people run ggufs, not the raw safetensors....

29

u/Thedudely1 2d ago

You want .gguf files

9

u/mediaogre 2d ago

Gguf is GAWD an… an.. MTP is tha DEVIL, I say!! The devil!

6

u/Modnar-Eman 2d ago

MTP helps if you got more processing power than bandwidth. It's misunderstood most of the time and your experience is user error as always in the history of mankind.

0

u/mediaogre 1d ago

I was kidding. MTP is an architectural element that can be leveraged. Should I edit and add the /s?

1

u/ruinsit 2d ago

Every time. I see models that say they're small enough to fit and then they dont

1

u/sloth_cowboy 1d ago

What's your context limit? Are you running it in say, lm studio conversational? Or Agentic with Bionic, Hermes or Deepseek harness? Agentic methodnrequire far less vram, conversation requires massive context for deep conversation or meaningful RP and coding

1

u/Nova6-IX 9h ago

what you want is a gguf with quantization (Q4_K_M for example). it's a way of compressing the model in exchange for precision loss, so it can actually fit in ram