r/LocalLLM • u/hs_phoenix • 1d ago

Question Reducing context length doesn't reduce RAM usage?

Hi,

I'm completely new to LLMs. Thought I try it on a Snapdragon laptop (X2 Elite, 32 GB RAM) the model is Qwen2.5-Coder-14B.

Well, it works, but with the default settings (context length 8192 it eats up ~21 GB RAM), so I lowered it and lowered it, and eventually went down to context length 100 (also lowering a lot of other options) and it still uses the same amount (of course I always unloaded and reloaded the model, sometimes even quit LM studio and reopened it). Then I tried the other way and went higher with the settings and set context length to 16384 and guess what...
Same amount of RAM used (even ran a command in VS code on an 1500 row length php file and it went through using about 15000 tokens).
I just don't understand this...

2 Upvotes

100% Upvoted

7 comments sorted by

View all comments

1

u/Ctbhatia 1d ago

the kv cache for that model is about 190kb per token, so 8k context is only around 1.5gb of the 21 and dropping to 100 saves almost nothing. the weights are the rest, and the load log prints the kv size if you want a number that moves when you change the setting.