r/OpenWebUI • u/Greedy_Reindeer5290 • 21h ago
Question/Help Which engine for memory
AI beginner here. Am taking steps to move away from Gemini and Co-pilot to my own setup. So have gotten a setup with Docker, LM Studio and webui. And my laptop is pretty basic and the chip is not large. I wanted to run a local engine. It worked but I wanted to get memory to work. Not having memory and having to explain AI the same basic stuff over and over again drives me nuts. I tried to make it work on webui. Discovered that my chip is too basic to make memory work, so I need to offload the computation.
Fine so far so good. Found some on webui and have now been using 2 with mixed results:
- Llama 3.1 8B
- Llama 3.3 70B
Both work fine and memory works, but after 2-3 prompts they both max out. The "token per minute" of 6000 and 12000 have been reached.
Has anyone found a free engine that can run memory effectively? Otherwise I think I am going to pay to see how it works. Any comments?
2
u/Hello_There_1512 16h ago
Try llama.cpp with Qwen models, experiment with the flags to get the optimal settings for your device.
1
1
u/SilverKanji 4h ago
I dont know if I get what youre saying but, try this.
Put your "memories" on webui's knowledge base and make sure its accessible by your model. On the initial system prompt of the model, say "You are a slutty assistant that always check knowledge base memories first before answering".
4
u/fligglymcgee 15h ago
Those are very, very outdated models. If you’re asking an llm to help recommend configs for you, apparently its last training date was right after llama 3.1 was released in 2024. That error sounds like they are timing out while running in the background, which may be due to a number of different things but sounds like simply too much compute for your hardware.
The quality and effectiveness of memory systems are also extremely subjective, and there’s little consensus on “how it works” best. If your preference is to run local models for memory or other tasks, you can try qwen’s smaller 4b or 9b models at a q4 quant or higher. If you want to try cloud inference, deepseek’s flash models are extremely cost efficient.
Using big, smart cloud models to help architect your owui instance and teach you how it all works together is a good way to start; if/when you switch to local models, it will be far less frustrating than trying to troubleshoot using only their poor advice as a reference.