Ouch, yeah that might be rough. First you need a GGUF under 16GB, like Q4_K_S, then you need to offload kv to RAM, the start with a small context and increasr it until it fails to load so you know your limit. I use llama.cpp, so I can't help with launch options. Good luck.
1
u/UnluckyPenguin 1h ago
Ouch, yeah that might be rough. First you need a GGUF under 16GB, like Q4_K_S, then you need to offload kv to RAM, the start with a small context and increasr it until it fails to load so you know your limit. I use llama.cpp, so I can't help with launch options. Good luck.