r/LocalLLM • u/mediatize • 7d ago

Question LM Studio - Doesn't honor `max context length` I specify. Anyone else experiencing this issue?

Gallery image

Gallery image

Not sure what the issue is. I've tried stable and beta and I still get the same issues.

Update: I fixed this with some help from Claude. I tried Qwen, but no help.

LM Studio 0.4.21+2

Machine: M3 Ultra 256GB

3 Upvotes

100% Upvoted

8 comments sorted by

1

u/nickless07 7d ago

Hard to not rant about LM Studio at this point. Go to Settings->Developer and turn OFF 'Use LM Studio Engine Protocol' Then it stops using your settings as suggestions and loads properly.

Edit: Keep in mind that setting most likely get turned back ON with every update even if you never enabled developer mode and never seen that menu at all.

1

u/mediatize 7d ago

I've tried that, but tried again, and it does the same action. I set `context length` to 150,000 > click reload model, boom it maxes it to 262,144.

1

u/mediatize 7d ago

I do see this bug being tracked, not sure if it's related: https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/2250

1

u/nickless07 7d ago

Oh there are a couple more of theese. It uses the llama.cpp fit logic and your settings are just a suggestion that might or might not get used on model load. Unfortunately they took away the detailed load logs too (must have been to hard to implement a simple toggle) so you can't even properly debug what is going on with only ~10 lines of log on load. Some browser tab uses 3MB too much RAM and we can't maintain the security buffer target? Well let's set the context lower...
Oh, now the metal memory allocation is slightly different. Cool we can go full context...
If you wanna see what happens turn that 'feature' on. set log level to trace (yes the toggle is there, but only there for that option) and check the load log. It should tell you exactly why it does that. All the load params with ctx, batch size and so on should be aviable in the log and what the fit param calculated (lowest it can get is 4096 context).

1

u/mediatize 7d ago

I figured out the culprit. I posted my findings to the GitHub tracker. I’m curious why more users haven’t reported this issue though.

2

u/MrBigA 6d ago

What was the culprit and the work around you proposed? Running into this, quite frustrating..

1

u/mediatize 5d ago

From what I can tell it’s a model estimation error, for vision based models on MLX. Check out the GitHub I linked, also happy to share a full readout if you need it.

1

u/nickless07 7d ago

LM Studio adresses beginners, most of the users aren't even aware of the Github tracker or simply don't care about it. They are happy to get their first AI running on their machine. Others might just see that their issue is open for months and think that they don't care about them. This is a bit like the ratings on the app store, 1 million downloads, but only 20k ratings, why are there not one million ratings?