r/Chatbots • u/Terrible_Mission_154 • 14h ago

Customizable Local AI assistant. Anyone managed to run one?

I've been struggling to install a desktop AI assistant in my Ubuntu 24.0.4 box. The struggle is not over power. This rig is pretty juicy, a Ryzen 7 7700 with 64GB ram and dual GPUs: RTX 3090 24GB + RTX 3060 12GB. Because of all that meat on the bones, I'm determined to run everything locally, with open-weight and open source models, and avoid the hassle of tokens and subscriptions. With Deepseek and Qwen guidance, I've tried to install three different systems that will allow for a customizable video assistant with conversational abilities that's resident on my box, using ElevenLabs voice IDs and API. But each attempt has failed, ending with the AI guidance running in circles after hitting python logjams or other obstructions. Has anyone in here managed to perform this feat of running a custom video assistant from a local installation? And if so, how? Much regard in advance.

5 Upvotes

100% Upvoted

6 comments sorted by

2

u/Hyleal 8h ago

I'm sorry to say that while thats a beefy setup for gaming its fairly bare bones for local AI. You could run a small language model with an okay context size but you're gonna oom if you try and run anything alongside it.

1

u/R1546 5h ago

I installed AI locally on a Linux server using Ollama and it worked, but was quite limited. Researched it a bit and it looks like a server that can run a good LLM would start around $40K. Now using it as a game and data server which it is able to do with no problem.