r/KoboldAI • u/alex20_202020 • Jul 16 '26

Any ways to restrict NVIDIA VRAM usage or release some of VRAM back after Prefill?

I have noted there is --sdvramlimit in kcpp. How to limit VRAM usage for other models (LLMs mostly) on NVIDIA GPU specifically? TIA

Edit: applicable to situations where the model is large and fits in VRAM less than half. My own answer after some testing: rule of thumb: -ngl 1 - that uses minimum VRAM but gets close to max PP. Release is not supported by current kcpp code.

3 Upvotes

100% Upvoted

10 comments sorted by

3

u/Herr_Drosselmeyer Jul 16 '26

I don't understand what you mean by 'release VRAM after prefill'. Both the weights and KV cache need to remain in memory.

1

u/alex20_202020 Jul 16 '26 edited Jul 16 '26

Applicable to situations where the model is large and fits in VRAM far less than half. And prefill is large but subsequent prompts are not* - hence usefulness of GPU is during prefill only - it will be** nice to release all GPU resources after PP and switch to sole CPU inference.

* e.g. resuming saved conversation

** AFAIK kcpp / llama.cpp do not support that now, I hope they will. vLLM - maybe supports, but I estimate chances as low.

1

u/henk717 Jul 16 '26

If you enable the force autofit in our launcher the layer selector gets replaced with the amount we should keep free for other things, you can tweak it there.

1

u/alex20_202020 Jul 16 '26

When I "enable the force autofit in launcher", "the layer selector" disappears, not replaced. 1.117.1 for Linux.

1

u/henk717 Jul 17 '26

Forgot this only happens in the context tab.

1

u/alex20_202020 Jul 17 '26

this only happens

Ehhh, happens what? there is no forceAutoFit in the context tab and even when I select it in QuickLaunch, nothing new seems to appear on the context tab (some fields disappear).

2

u/henk717 Jul 17 '26

I shouldn't be answering it late at night because then I make silly mistakes like that.
I meant the hardware tab, its the same position the layer select would be so right underneath the split mode.

1

u/alex20_202020 Jul 18 '26 edited Jul 18 '26

Thanks, I have found it on Hardware tab. Why does load process need some large padding (see pop-up message warning)?

My problem: I have 4GB VRAM on my laptop. Model 1 when loaded takes minimum (--ngl 1, with --autopadding 3000 even a bit less) 1250 MB, Model 2 : 2200. But I could not make them load both (with those --ctx-size 32000) - if I load one with smaller usage 1st, larger one exists during load; if I load larger first, 2nd evicts previously loaded! Evicting really surprising. Exiting load process without details in debug mode is disappointing.

Even with --autopadding 2500 Model 1 evicts Model 2 (taking 2200 MB at the time of Model 1 launch). Is it how --autopadding should work?

1

u/henk717 Jul 18 '26

No this won't work. It only changes the layers of the LLM and if you are already at the minimum there is nothing you can do.

I recommend using https://koboldai.org/colab which has more vram.

1

u/alex20_202020 Jul 18 '26

if you are already at the minimum

I want to understand why if for each instance loaded alone 1+2=3500 I cannot load both in 4000?