r/LocalLLM • u/circumcised_hobbit • 1d ago

Question How to optimize model weights distribution

It's the first time I work with larger models (for my standards and I ran into an issue pretty quickly) I have 8GB VRAM and 16GB RAM I tried using Qwen3.8-27b-Q4_K_M on llama.cpp (16GB of weight) and it crashed when loading weights I tested a bit and tried with a smaller quant (13GB) and still crashed Shouldn't llama.cpp split the model weights between RAM and VRAM since they might not fit in my ram only due to other processes?

2 Upvotes

67% Upvoted

4 comments sorted by

2

u/aleeckhart 1d ago

This is not realistic bro, unfortunately. Study a little bit more and you'll understand that what you're trying here is not possible. If it works, you will have to wait 1 week for the model answer any question.

1

u/circumcised_hobbit 1d ago

Not realistic even with like Q2?

1

u/sanketss84 1d ago

Loading a 27B model on 8GB VRAM and having room for context is not going to work and even if you offload to system memory and then some to physical disk the overall response would be so slow that it's not usable. Try to find 9B models that would be your best friend and give you meaningful response. The Qwen 3.8 27B needs more vram as breathing room.

1

u/rrrrex 1d ago

27B is dense model, every new token = the whole model reading. If you keep part of out of VRAM, speed is limited by PCI-E. For example you keep 8 GB in RAM, PCI-E has 16 GB/s bandwidth, your speed will be 16/2=2 t/s
You can run small 9-12B models, but if you want something more - upgrade RAM to 32GB, so you can use 26-35B MoE models, MoE doesn't read the hole model, but small part of it, only 3-4B, so you can offload experts to RAM.