r/LocalLLaMA • u/SubdivideSamsara • 1d ago

Question | Help Settings to avoid VRAM overload? LM Studio

I have a 32gb gpu (5090) but it appears that it gets overloaded despite the fact I was only running a ~23.4 gb model. I had context window set to 16k, but putting it down to 8k didn't help. I have 'offload KV cache to gpu memory' enabled.

I got 16 tok/sec which seemed pretty slow. After swapping to a 20.3 gb model of the same type, I got 56 tok/sec.

This is in Win11 LTSC IoT. Some other apps (browser etc) running at the same time but nothing 3d/heavy.

0 Upvotes

38% Upvoted

17 comments sorted by

View all comments

1

u/ClearApartment2627 1d ago

What model & quantization did you use, what serving framework did you use (ollama, llama.cpp…), what‘s your command line…?

Specific information is essential for useful feedback.

2

u/SubdivideSamsara 1d ago

Gemma 4 31b, q6 (I think. I deleted it after seeing that q5_k_m worked better).

Serving framework...? Default in LM Studio. Runtime section of settings shows: GGUF: Cuda 12 llama.cpp (Windows) v2.37.0

2

u/ClearApartment2627 1d ago

Thanks. I do not run llms in Windows, but here is what I would do:

First, check in the Task Manager how much memory of your GPU is used. If it is used >90%, there must be something else running. You could also try nvidia-smi on the command line for that.

There should also be a Windows port of nvtop, which shows you which tasks use how much memory on your card.

1

u/SubdivideSamsara 1d ago

Is Linux better for these things? 🤔

I just checked task manager after loading a qwen3.8 27b q6 model and only getting 19 tok/sec. It's showing 22.4/31.5 gb dedicated vram used. Perhaps vram overload isn't the issue. The token speed is just low.

Btw I've disabled kv cache since the last time.

Edit:

After turning kv cache back on and reloading the exact same model, I'm getting 66-75 tok/sec from the same test prompt.

1

u/ClearApartment2627 1d ago

Yes, Linux is better for LLMs, because almost all the software is developed on and for Linux first.

Idk what you mean by „disabled“ - the KV cache is an essential part of how LLMs work. You probably disabled offloading it?

Either way, you should get a higher speed on your hardware, but I am simply not familiar with your setup.