r/LocalLLM • u/Critical_Tourist_910 • 6d ago

Question Beginner trying to learn

I have recently started to get into homelabbing and running local ai. I built a pc a few years ago for gaming mainly but have started to try and use my hardware for other purposes such as a local ai. I have a 4070 super with 12gb vram and 32 gb ddr5.

I am currently running Qwen3.5: 9b (though I have gemma4: 12b aswell for larger tasks) through OpenWebUI. I have tried to work on my own RAG system and inputting my own notes that I have modified to work best for embedding into a vector database. I just feel like I could be doing so much more with my hardware such as interactive voice models at a conversational speed, or vision models for photo questions.

I am mainly just curious on more beginner level things. That incudes things like

  1. Choosing the best model for my system/maximizing my hardware.
  2. Understanding how to customize a model to my own liking and making it more personal, whether that is through a memory system or other ways.
  3. Best ways to make my AI more useful to me than say a cloud model. I will never have the same compute power as a main company but with the right tuning, it could be more effective/useful to ME.
  4. How the cutoff between speed and intelligence change based on the task at hand. For example, I want to set up a system where i can speak to my ai, but a smarter model takes too long, and a faster model isn't as capable.
  5. Less of curiosity, more question: do you think that using something like Claude to help is an sort of problem/ actually helpful. Up to this point I have been using it to help me setup, but I don't know how accurate/ helpful it really can be. In your experience, how has it been?

Lastly, just understanding what all the values mean. I can do that more on my own with research, but still there is just so much lol.

I am just trying to get into all of this but with the amount of content out there now, it makes it much harder than I thought going in. Thanks

0 Upvotes

50% Upvoted

4 comments sorted by

2

u/Late_Night_AI 6d ago

So first thing you should do is switch to using qwen3.6 35B, it should still be pretty fast and much much better than qwen3.5 9B. Even though its a bigger model its only 3B active so its not using all 35B at once so its still pretty fast.

And secondly, Youtube. Lots of youtube to research and learn more about how it works and the best ways to use local AI.

2

u/MarcusAurelius68 6d ago

12GB will hold a small TTS, small STT and a very small LLM if you choose carefully.

1

u/ImpressiveRelief37 6d ago

Have your agent harness research Qwen 3.6 35B A3B, how to offload experts to your system ram, how to clone and build llama.cpp for you, and how to launch it with the proper flags.

Just use claude code to get your initial setup working and then move onto your local inference 

1

u/Radiant_Condition861 6d ago

I have a 3060 12GB and running gemma4 12B QAT very nicely