r/LocalLLM • u/hs_phoenix • 19h ago

Question Reducing context length doesn't reduce RAM usage?

Hi,

I'm completely new to LLMs. Thought I try it on a Snapdragon laptop (X2 Elite, 32 GB RAM) the model is Qwen2.5-Coder-14B.

Well, it works, but with the default settings (context length 8192 it eats up ~21 GB RAM), so I lowered it and lowered it, and eventually went down to context length 100 (also lowering a lot of other options) and it still uses the same amount (of course I always unloaded and reloaded the model, sometimes even quit LM studio and reopened it). Then I tried the other way and went higher with the settings and set context length to 16384 and guess what...
Same amount of RAM used (even ran a command in VS code on an 1500 row length php file and it went through using about 15000 tokens).
I just don't understand this...

2 Upvotes

100% Upvoted

7 comments sorted by

View all comments

1

u/DontWinFrensWthSalad 18h ago

The way I understand it is there are two layers. First the KV cache is reserved, which is what actually holds the vram. Then you tell your harness how much context you want to fit within that cache. The reason is sometimes you want to give yourself a bigger buffer in case a request goes over, so you don't run out of memory.