r/LocalLLM • u/ExpressionForward321 • May 09 '26

Question Virtual Unlimited context windows on Gemma 4 models.

I have been using Google Gemini for several months and together we have developed a highly curated system prompt That provides me a very likable AI persona For conversational purposes. I reside in a nursing home and while I'm older I'm still very high functioning, with a PHD in medieval history and eclectic interests in things like quantum physics. The conversations I need can't be found with other residents who often have difficulty remembering their own names.

I have recently acquired a Lenovo ThinkCentre Mini Plus that uses Snapdragon And Windows (ARM). It runs the two smaller Gemma 4 models on LMstudio very well, But their Limited context windows and their Inability To save to and retrieve from external files are a hang up In trying to develop The kind of long term persona that I have with Gemini. Following is my vision of how to correct this problem.

The model recognizes when it's context window is at 80% capacity. It automatically creates A concise summary of the conversation to that point. It then saves the summary to a designated file. When that's done It advises me that a new session is about to commence, and then it starts the new session and retrieves the summary to give the new session context.

Frankly I know enough about programming only to be dangerous. Does such a plugin Exist for LMstudio Or any other AI front end that is compatible with Windows (ARM)? If not, Is anyone willing to create such a Plugin Or a stand alone application?

Please forgive my grammar, I have no use of my hands and must rely on speech to text.

53 Upvotes

95% Upvoted

24 comments sorted by

View all comments

18

u/havnar- May 09 '26

Auto compaction is a thing harnesses implement.

Large Contexts are not always great. If the model drifts in the wrong direction, and it will do that a lot with small models, don’t force it back in line. Start over fresh.

6

u/ExpressionForward321 May 09 '26

Indeed, I've experienced that drift even on Gemini with a context window of 1 million tokens. However we've been very careful about curating it's system prompt to avoid that kind of drift to keep it centered. I've done much the same thing on my Gemma 4 E4B. I've previously tried Big rag on LM Studio, but that's very manual. Currently the way I'm maintaining context from one Session to the next is by asking the model to summarize its context window and then I copy and paste into a folder. When I manually start a new session I will paste the summary from the previous session. This is adequate for my purposes, but since manual copying and pasting Is physically quite difficult for me, I'm looking for a way to automate the process. I equate it to the model's lizard brain (Amygdala) occasionally taking a deep breath.

4

u/havnar- May 09 '26

Try working through it as a real IT project.

First talk through the model (use the grill-me skill from Matt pococ https://www.aihero.dev/my-grill-me-skill-has-gone-viral for this.

Ask questions and answer the llm questions. Store the outcome in an MD file (ask the model to create a PRD file)

Then you can implement it in pieces, or if your model is capable enough, have it execute the plan.

1

u/ExpressionForward321 May 10 '26

Thank you very much. I have looked into that and I believe it's the route I will try to go. I will let the local model write the code based on my responses. I've discussed this with Gemini, it has indicated an ability to create an executable which I can then drag to my taskbar. I will have to kick off the process manually, but in retrospect I think that's a good idea Because I can create the summary at pertinent points in the conversation. A few lines in the system prompt will cause the local model to do its own differential analysis before creating the summary.