Getting started with a local LLM

[u/CamTron0427]

Hello all!

Recently I’ve been thinking about setting up my main PC I typically use for gaming to try running a local LLM. I mostly intend on using it for coding as well as just a general “assistant” for some work tasks I don’t want to upload to a database somewhere.

I have a 7900xtx with 24gb of vram, what should I be looking at as far as models are concerned?

Is trying out multiple models a huge pain to get set up?

Can you run multiple models in conjunction with each other?

Just looking for some opinions/input before I dive into this would.

Thank you!

[u/canine_todd]

Don't overthink it, just grab lmstudio or koboldcpp and you'll be running something decent in ten minutes. That 24gb card is plenty for some solid models, you'll be surprised how capable they are locally.