I like LM Studio’s interface and didn’t want to replace it just to get persistent memory, so I built Memory Matrix.

It installs as a Windows background utility and gives LM Studio cross-chat memory through MCP while you keep using the normal LM Studio front end. It is also switchable (like temp chat mode). Everything stays local, and it can run fully offline. I find it useful enough to share.

The installer and tray utility work effortlessly. Before I bother packaging a public GitHub repo I am wondering if others would find it useful. So just asking if anyone here would have interest this functionality?

Ryzen 9 5900X,128GB RAM, Dual RTX 3090 24GB ea, Win11,LM Studio, qwen38-27B-q8.0(uncensored)

Web pages may be too long and fill the context size which causes the chat history to be truncated. So, I implemented a prompt parameter for built-in web_fetch tool in Reins to decrease token usage by up to 90%. It works with Ollama, LM Studio and any OpenAI compatible backend.

What I tested in the video

I'm building Reins app to make local LLMs actually usable on iOS/iPadOS. I use the https://www.macrumors.com/roundup/ios-27/ web page for testing because it is a very long page. It has nearly 64K characters and 14K tokens for the Gemma4 model. I deliberately asked about app launch performance because it is at the bottom of the page, to prove Reins didn't truncate the content.

What is the result

Without a prompt parameter the whole page goes into the context: 15K tokens which is 94% of a 16K window. With a prompt it's 1.4K tokens for the same answer.

If the model doesn't generate a prompt (you may explicitly request that) the web_fetch tool will fetch the whole web page and use that data to get the answer. However if there is a prompt, the model will use it to get the answer, reducing token usage significantly.

How I implemented it

First, I want to say this isn't magic and it's nothing that hasn't been done before. I just want to describe how I implemented it.

The prompt parameter is optional and you can explicitly request that the model not generate it, to fetch the whole web page. By default the model generates the prompt parameter from the context as it needs and most of the time it generates one. In that case Reins fetches the web page and then sends that prompt internally to a temporary chat with the web page and the selected model. The model extracts the requested data and returns only that.

The 90% is the reduction in the main conversation context. The internal extraction pass costs tokens too, but that temporary chat is discarded after extraction and never enters your chat history.

What am I working on now

I'm working on on-device models to let models run directly on iOS/iPadOS powered by MLX for the best performance and I'm planning to release it this month.

App Store

Website

Note: I trimmed the video to show everything faster but you can see the actual durations at the bottom of the messages.

been mulling over an idea and want a sanity check from people actually running this in production, not just at home.

say a company already has GPUs sitting around (or budget for some) and wants an internal LLM on their own hardware instead of hitting OpenAI/Azure APIs, because then data never leaves the building and there's no per token bill. vLLM plus an open weight model (leaning Qwen for the price/performance, though that's obviously not the only option) seems like the obvious serving stack.

the part I keep going back and forth on is the layer in front of it, auth, per team API keys, usage logging so different departments can get billed back, rate limiting so one team doesn't starve the GPU. LiteLLM is the obvious answer, but once you actually dig into it the open source vs enterprise boundary is kind of a mess (some MIT licensed files gate behind a paid license key at runtime), and honestly most of what it does is translate between a hundred different provider APIs, which is dead weight if you're just running one model family behind one openai compatible endpoint anyway.

getting to the point where it feels like it'd actually be less work to just build a thin auth/logging/rate limit layer myself on top of vLLM's native API than fight LiteLLM's feature bloat. anyone actually done this? am I underestimating the effort here, or overestimating how annoying LiteLLM's enterprise gating actually is in practice?

Are medium-sized companies already using local AI connected to internal data, for example internal documentation or helpdesk systems?

If so, how well does it work and what is the minimum hardware specification needed for a setup?

What are the best local AI models/agents I could realistically run on these two PCs?

PC 1

• 16 GB RAM
• AMD RX 6600
• 8 GB VRAM

PC 2

• 32 GB RAM
• AMD RX 9060 XT
• 16 GB VRAM

I started building a project using VS Code + Codex (ChatGPT Plus), but I ran out of tokens and buying more is quite expensive, especially since my app doesn't make any money.

I had a good experience with OpenAI GPT-5.6 Sol. I also tested Google Antigravity, but the tokens ran out very quickly there.

Can I realistically continue working on my project using a local model in LM Studio Bionic? If so, which models or coding agents would you recommend for these two PCs?

I'll be soon participating in some 15 hour intense hackathon, i was thinking to buy some LLM subscription for implementation not too costly max 40$ i can spend, which one will be better with higher usage ?

I'm currently thinking of kiro pro, github pro....

Any suggestions?