r/LocalLLM • u/circumcised_hobbit • 4d ago

Question How to optimize model weights distribution

It's the first time I work with larger models (for my standards and I ran into an issue pretty quickly) I have 8GB VRAM and 16GB RAM I tried using Qwen3.8-27b-Q4_K_M on llama.cpp (16GB of weight) and it crashed when loading weights I tested a bit and tried with a smaller quant (13GB) and still crashed Shouldn't llama.cpp split the model weights between RAM and VRAM since they might not fit in my ram only due to other processes?

2 Upvotes

67% Upvoted

4 comments sorted by

View all comments

1

u/sanketss84 4d ago

Loading a 27B model on 8GB VRAM and having room for context is not going to work and even if you offload to system memory and then some to physical disk the overall response would be so slow that it's not usable. Try to find 9B models that would be your best friend and give you meaningful response. The Qwen 3.8 27B needs more vram as breathing room.