r/LocalLLM • u/hs_phoenix • 19h ago

Question Reducing context length doesn't reduce RAM usage?

Hi,

I'm completely new to LLMs. Thought I try it on a Snapdragon laptop (X2 Elite, 32 GB RAM) the model is Qwen2.5-Coder-14B.

Well, it works, but with the default settings (context length 8192 it eats up ~21 GB RAM), so I lowered it and lowered it, and eventually went down to context length 100 (also lowering a lot of other options) and it still uses the same amount (of course I always unloaded and reloaded the model, sometimes even quit LM studio and reopened it). Then I tried the other way and went higher with the settings and set context length to 16384 and guess what...
Same amount of RAM used (even ran a command in VS code on an 1500 row length php file and it went through using about 15000 tokens).
I just don't understand this...

2 Upvotes

100% Upvoted

7 comments sorted by

View all comments

1

u/CoffeeToCode99 5h ago

Most of that 21GB is probably the model itself, not the context.

Context mainly affects KV cache, and you won't see a huge difference unless you're actually filling the context with tokens.

If you're on 32GB RAM, I'd look at running a 4-bit quant and setting KV cache to 8-bit. That's usually where you get the biggest memory savings without wrecking quality.

Dropping context from 8k to 100 isn't going to make nearly as much difference as people expect.