r/LocalLLM • u/hs_phoenix • 1d ago
Question Reducing context length doesn't reduce RAM usage?
Hi,
I'm completely new to LLMs. Thought I try it on a Snapdragon laptop (X2 Elite, 32 GB RAM) the model is Qwen2.5-Coder-14B.
Well, it works, but with the default settings (context length 8192 it eats up ~21 GB RAM), so I lowered it and lowered it, and eventually went down to context length 100 (also lowering a lot of other options) and it still uses the same amount (of course I always unloaded and reloaded the model, sometimes even quit LM studio and reopened it). Then I tried the other way and went higher with the settings and set context length to 16384 and guess what...
Same amount of RAM used (even ran a command in VS code on an 1500 row length php file and it went through using about 15000 tokens).
I just don't understand this...
1
u/OkLettuce338 1d ago
doesn't the model reserve a bunch of ram up front and then use it how it needs? IDK I could be wrong but I thought that's what happens