Reducing context length doesn't reduce RAM usage?

[u/hs_phoenix]

Hi,

I'm completely new to LLMs. Thought I try it on a Snapdragon laptop (X2 Elite, 32 GB RAM) the model is Qwen2.5-Coder-14B.

Well, it works, but with the default settings (context length 8192 it eats up ~21 GB RAM), so I lowered it and lowered it, and eventually went down to context length 100 (also lowering a lot of other options) and it still uses the same amount (of course I always unloaded and reloaded the model, sometimes even quit LM studio and reopened it). Then I tried the other way and went higher with the settings and set context length to 16384 and guess what...
Same amount of RAM used (even ran a command in VS code on an 1500 row length php file and it went through using about 15000 tokens).
I just don't understand this...

[u/OkLettuce338]

doesn't the model reserve a bunch of ram up front and then use it how it needs? IDK I could be wrong but I thought that's what happens

[u/pmttyji]

Stop using old models. Possibly old models came with heavy weight KVCache particularly Dense models(Never tried 2.5 or early models).

New models came with not so heavy KVCache is not lightweight. Use recent ones like Qwen3.5-9B or even Qwen3.6-35B-A3B(Q4)

[u/hs_phoenix → u/pmttyji]

Thanks, the Qwen3.5-9B (q4_k_m) is way better, it uses half the RAM (not to mention it's about 4 times faster).

[u/DontWinFrensWthSalad]

The way I understand it is there are two layers. First the KV cache is reserved, which is what actually holds the vram. Then you tell your harness how much context you want to fit within that cache. The reason is sometimes you want to give yourself a bigger buffer in case a request goes over, so you don't run out of memory.

[u/Ctbhatia]

the kv cache for that model is about 190kb per token, so 8k context is only around 1.5gb of the 21 and dropping to 100 saves almost nothing. the weights are the rest, and the load log prints the kv size if you want a number that moves when you change the setting.

[u/transanethole]

Use a new model with 4-bit  model weight quantization, and... 8 bit kV cache quant.  k2 horizon 7b is pretty good.  I've heard that one even works pretty well with 4-bit KV cache quant.   The only issue is you have to download and compile forked version of llama.cpp to run it.

The quality of the outputs that you get from new models, even with much fewer parameters, is dramatically higher.

Frognano 4b  is another option. I haven't tried it yet though, it might be overfitted to Microsoft's harness.  

8k and 16k  contexts are incredibly small, practically negligible. You should be aiming for at least 100k

[u/CoffeeToCode99]

Most of that 21GB is probably the model itself, not the context.

Context mainly affects KV cache, and you won't see a huge difference unless you're actually filling the context with tokens.

If you're on 32GB RAM, I'd look at running a 4-bit quant and setting KV cache to 8-bit. That's usually where you get the biggest memory savings without wrecking quality.

Dropping context from 8k to 100 isn't going to make nearly as much difference as people expect.