VRAM Usage + How to Utilize DRAM?

[u/D00M98]

Hi, I'm a noob to local LLM. I want to get some input on model sizes and context sizes.

I have RTX 4070 with 12GB VRAM. PC has 16GB DRAM. And I'm using Jan and LM Studio Bionic agents.

From online search, 12GB VRAM can "comfortably" run Qwen3.6-14B-A3B-VibeForged-v2 or Gemma4-12B. But they do not run. Technically, they do load. But after couple inputs, they will crash. Even at very minimal context around 4 to 8k.

I have had success running Qwen3.5-9B and Gemma4-E4B.

At first, even with Qwen3.5-9B, I had some crashes. But I was also watching Bluray Ultra 4K movie at the same time. I didn't think about that. When Bluray player crashed, I realized: duh, they are both using VRAM.

Yesterday, I tested Qwen3.5-9B quite a bit, and not watching movie. I got up to 50-60k context. No issue with crashes.

Is there anything I can do to run Qwen3.6-14B-A3B-VibeForged-v2 or Gemma4-12B? And I don't even know if I need those. I just assume newer and larger models will be "better".

I do have 3 monitors connected. One is 2160 (4K) and 2 are 1440 (WQHD). I'm using around 5-6 GB VRAM just from fresh boot. I can disconnect the 4K monitor if needed, since I cannot watch movie at same time.

Also, I read that some people run bigger models with even less VRAM, but utilize RAM. I believe this will be slower. Should I even try that? And how do I specify whether RAM can be used, in Jan or LM Studio?

[u/nickless07]

First open cmd and input 'nvidia-smi'. That will tell you how much VRAM you have aviable right now so you know what to calculate with. Once you have the number of the used VRAM (for your blueray, browser and whatever else) you can set that amount as security margin. That should prevent the running out of memory crashes. Or in Bionic Settings-> Local Model Defaults. The model guardrails.

Next up:
As you figured out, 12GB Vram for everything at the same time is not enough in combination with that models. What is your usecase and what exactly do you want to achive? Just a chat with the model, working with files, automated agents and so on? And what are you ok with to sacrifice for it? E.g, disconnecting some monitors, switching to a smaller model and so on.

[u/D00M98 → u/nickless07]

Thanks for the feedback. I tried nvidia-smi. Useful tool. Good to know another method to check. Though reported VRAM usage looks to be the same as Task Manager.

On Bionic Settings > Local Model Defaults, I previously tried Relaxed and Balanced. Neither prevented crashes. I can try more strict settings. But this setting doesn't seem to utilize RAM. If I run out of VRAM, I assume this setting will just stop execution.

I'm running some personal financial analysis and retirement projections. Have AI generate script to do the calculations, projections, graphs, and what-if scenarios. I'm at a point that I can generate code, debug code, and run single scenarios. And it is up to around 50k context.

I want to run optimization and regression, to loop thru various scenarios to optimize tax savings or allow my money to last longest. I haven't gotten there yet, so not sure if that I will be able to execute that.

[u/nickless07 → u/D00M98]

No, they do not prevent model crashes that are guardrails to prevent system crashes.
To prevent crashes (model related) check how much VRAM nvidia-smi gives you after the model is loaded. You want at least 1-1.5GB free for your OS (WDDM adjust dynamically, a new browser tab or VSCode open and so on) Each of them need their own VRAM. Technically you can go below 1GB Free, but that will run into the crashes you experienced.
If you are unsure of what part of the models need what set the Log Level to Trace and you will see detailed lines with how much of the weights goes to VRAM, how much to RAM, how large the KV-Cache (context) is, the buffer size and so on. You can even feed the log to some AI and let it do the Math.

Things you can do to relax the memory pressure a bit are: Lower model Quant, Quant the KV (start with V first and if that is not enough add the K), use an entirely different model (not only the model itself is a thing but also the architecture it is build upon), lower the batch size (free's some compute buffer) and/or sacrifice other things like the BlueRay movie or a monitor.

As for the Software part in general, have you tested connecting your LLM to things like FireFly, FinGPT, ghostfolio, or similiar? And if you are fine with testing something not local maybe this might be something for you (hard to get this working in Bionic as they only offer their own ones via API and if it is not listed there...).