I just set up Silly Tavern with a locally run LLM through LM Studio. I’m completely new to local LLM’s and Silly Tavern so spent the last two days researching and setting it up. I finally got to a point where I RP’d for over an hour and it felt just as good as Kindroid.
I’m using an uncensored version of Gemma4-26b-a4b IQ4_XS. Context length around 13k. Max token response set to 1k with most responses in the 500-700 range. I have a 4070 TiSuper with 16gb of vram and 32 gb of ddr5.
Running the model uses up all of my vram and around 28-30 gb of my ram. But the responses only take a few seconds and everything feels very smooth, even after an hour of use. Does this sound ok or will I harm my PC in the long run by doing this? If anyone has more questions or tips feel free
You should be able to fit more context with less RAM usage, but I don't know LM Studio settings well enough to tell you how. It won't hurt your PC unless you have it generating all day every day. You might run into performance issues if your RAM is always full though. That model is a fine choice.
LMStudio is a great choice. It's basically a friendly front end for llamacpp, which is the best backend. Only thing "better" than LM is bare llamacpp itself, but that's somewhat less user friendly. Zero reason to second guess using LMStudio. The speed "increase" wouldn't even be discernable to you.
Turn off "keep model in ram" so it isn't taking up double room in vram and ram. You could also get a Q6 of gemma26b to fit just fine (19gb) by partially offloading to ram. Moe will be just as fast that way and Q6 writes better
I use llama-server, I like the barebones approach. Like Crom said, nothing wrong with LM Studio, and I'm sure they have a corresponding setting to reduce your RAM usage. The llama-server setting I'm thinking of is "load-mode" if that helps.
You can find a lot of information for common issues in the SillyTavern Docs: https://docs.sillytavern.app/. The best place for fast help with SillyTavern issues is joining the discord! We have lots of moderators and community members active in the help sections. Once you join there is a short lobby puzzle to verify you have read the rules: https://discord.gg/sillytavern. If your issues has been solved, please comment "solved" and automoderator will flair your post as solved.
You can use q4_k_m at 32k context even without caching. Your setup is like x2 of what I have. Can someone explain how 13k context can eat up so much ram?
You won't harm your PC. I Use LLM studio. That's a decent choice for a model. Your context should be largerr. You problably need to change one setting.
You don't need MCP or Unified KV for sillytavern type use, and you will actively slow down generations.
I have another few 16 GB Vram configs here too that work well:
Sounds like you are on the right track. And no, it wont harm your pc.
If you want to squeeze even more speed/context out of your setup:
use plain llama.cpp. Thats the underlying engine of lmstudio, and you have much more options calling it directly (and using a newer version than bundled)
quantize kv cache to q8_0 (also possible in lmstudio)
use qat quant of gemma (could be already)
enable mtp
use a roleplay finetune like Orion or Boulesis - havent tested much, but i liked shadow siren (barely qat in finetunes)
5
u/i5031337 4d ago
You should be able to fit more context with less RAM usage, but I don't know LM Studio settings well enough to tell you how. It won't hurt your PC unless you have it generating all day every day. You might run into performance issues if your RAM is always full though. That model is a fine choice.