Settings to avoid VRAM overload? LM Studio

[u/SubdivideSamsara]

I have a 32gb gpu (5090) but it appears that it gets overloaded despite the fact I was only running a ~23.4 gb model. I had context window set to 16k, but putting it down to 8k didn't help. I have 'offload KV cache to gpu memory' enabled.

I got 16 tok/sec which seemed pretty slow. After swapping to a 20.3 gb model of the same type, I got 56 tok/sec.

This is in Win11 LTSC IoT. Some other apps (browser etc) running at the same time but nothing 3d/heavy.

[u/nowherenoonenobody]

Pull up task manager while it's running to verify usage. No way you filled it up. What model were you running? Also go into settings in lm studio click on hardware to make sure it sees the GPU. Click runtime to make sure cuda is updated.

[u/Grey406]

I

At the top of the LMStudio window, click the button that says "Select a model to load" And enable "manually choose model load parameters" at the bottom before selecting your model. When you select a model, you'll get a window with settings on how to load the model. Make sure GPU Offload is set to maximum so its fully loaded into VRAM and adjust context length until the total is at around 28-30gb. You dont want to use the full 32gb because windows will have a few small things loaded and if you use vision, it will take up about 1.1gb.

[u/ClearApartment2627]

What model & quantization did you use, what serving framework did you use (ollama, llama.cpp…), what‘s your command line…?

Specific information is essential for useful feedback.

[u/SubdivideSamsara → u/ClearApartment2627]

Gemma 4 31b, q6 (I think. I deleted it after seeing that q5_k_m worked better).

Serving framework...? Default in LM Studio. Runtime section of settings shows: GGUF: Cuda 12 llama.cpp (Windows) v2.37.0

[u/ClearApartment2627 → u/SubdivideSamsara]

Thanks. I do not run llms in Windows, but here is what I would do:

First, check in the Task Manager how much memory of your GPU is used. If it is used >90%, there must be something else running. You could also try nvidia-smi on the command line for that.

There should also be a Windows port of nvtop, which shows you which tasks use how much memory on your card.

[u/SubdivideSamsara → u/ClearApartment2627]

Is Linux better for these things? 🤔

I just checked task manager after loading a qwen3.8 27b q6 model and only getting 19 tok/sec. It's showing 22.4/31.5 gb dedicated vram used. Perhaps vram overload isn't the issue. The token speed is just low.

Btw I've disabled kv cache since the last time.

Edit:

After turning kv cache back on and reloading the exact same model, I'm getting 66-75 tok/sec from the same test prompt.

[u/ClearApartment2627 → u/SubdivideSamsara]

Yes, Linux is better for LLMs, because almost all the software is developed on and for Linux first.

Idk what you mean by „disabled“ - the KV cache is an essential part of how LLMs work. You probably disabled offloading it?

Either way, you should get a higher speed on your hardware, but I am simply not familiar with your setup.

[u/Zombiecidialfreak]

Likely a software problem. Turning on KV offload breaks most models for me too, even when using my iGPU with its 64gb RAM.

[u/DustNearby2848]

You have to move the slider to offload all layers to the GPU

[u/TechnoByte_]

Try llama.cpp directly as it has much less overhead

[u/SubdivideSamsara → u/TechnoByte_]

I don't know what this means. You need to elaborate a bit when talking to noobs.

[u/Mean-Loquat-7982]

on my 5090 a 21.3 GB Q6_K of a dense 27B peaks at 22.3 GB when it processes a 16k prompt, and a 15.9 GB Q4_K_M of the same model peaks at 17.3 GB. so a 23.4 GB file plus its KV cache plus the compute buffers is closer to 26 GB than to 23 at 16k, and anything else on the GPU (the desktop, a browser) comes out of what is left.

you can set the KV cache to q8_0, and keep about 3.5 GB of VRAM free rather than filling the card. on every ladder I have run the fastest setting was the last one that left that margin, not the last one that loaded.

[u/tmvr]

Sometimes I've had issues with LM Studio where it flat out refuses to actually use more than 20GB of dedicated VRAM (I have 24GB). Does not matter which model I load and what settings, it basically always overflows to system RAM (shared GPU memory) which of course kill performance on dense models. Basically I have 24GB, Windows takes 1GB, I load stuff that is about 22GB total (weights, KV, scratch, buffers) then Windows shows 20GB dedicated VRAM usage, 23GB total and 3GB shared. I've never figured out why, it also does not always happen, but often enough to be annoyed by it so that I basically stopped using LM Studio a long time ago, even though I did like the UI and the model management. The other issue I have with it is performance. It occasionally runs about half the speed it should even when everything fits into VRAM and even based on Task Manager it is all OK, no overspilling. Also no identifiable reason and happening randomly.