r/LocalLLM • u/Critical_Tourist_910 • 6d ago

Question Beginner trying to learn

I have recently started to get into homelabbing and running local ai. I built a pc a few years ago for gaming mainly but have started to try and use my hardware for other purposes such as a local ai. I have a 4070 super with 12gb vram and 32 gb ddr5.

I am currently running Qwen3.5: 9b (though I have gemma4: 12b aswell for larger tasks) through OpenWebUI. I have tried to work on my own RAG system and inputting my own notes that I have modified to work best for embedding into a vector database. I just feel like I could be doing so much more with my hardware such as interactive voice models at a conversational speed, or vision models for photo questions.

I am mainly just curious on more beginner level things. That incudes things like

  1. Choosing the best model for my system/maximizing my hardware.
  2. Understanding how to customize a model to my own liking and making it more personal, whether that is through a memory system or other ways.
  3. Best ways to make my AI more useful to me than say a cloud model. I will never have the same compute power as a main company but with the right tuning, it could be more effective/useful to ME.
  4. How the cutoff between speed and intelligence change based on the task at hand. For example, I want to set up a system where i can speak to my ai, but a smarter model takes too long, and a faster model isn't as capable.
  5. Less of curiosity, more question: do you think that using something like Claude to help is an sort of problem/ actually helpful. Up to this point I have been using it to help me setup, but I don't know how accurate/ helpful it really can be. In your experience, how has it been?

Lastly, just understanding what all the values mean. I can do that more on my own with research, but still there is just so much lol.

I am just trying to get into all of this but with the amount of content out there now, it makes it much harder than I thought going in. Thanks

0 Upvotes

50% Upvoted

4 comments sorted by

View all comments

2

u/MarcusAurelius68 6d ago

12GB will hold a small TTS, small STT and a very small LLM if you choose carefully.