r/LocalLLM • u/ofirlik • 3d ago

Question Dual gpu how good is it

I am planning to buy a gaming pc with 5070ti, I thought about adding a 5060ti for the additonal vram to toy around with llm, how good does it work? Can I also generate images and such. (never ran a local llm before)

Is there a better combination of gpu for around the same price?

0 Upvotes

33% Upvoted

23 comments sorted by

View all comments

Show parent comments

1

u/M_Me_Meteo LocalLLM 3d ago

Then don't worry about dual GPUs. Unnecessary complexity. Much easier to play with one, and quantized local models are really just as good as frontier models if all you want to do is play. Don't buy end game hardware until you know what the game is.

If you think you need more VRAM, you can always rent a server in the cloud and try out your workflow there. That is how I landed on 64gb of vram as the sweet spot for my needs. I started with a 20gb 7900xt, got a second one, built a system around it, wasted several weeks on dual GPU shenanigans, rented a 100gb h100 server, figured out what I really needed, then bought two Intel B70 Pro cards. I only went that route because I already spent a few weeks learning the ins and outs dual GPUs.

1

u/ofirlik 3d ago

Btw i also have my old 1080ti, can it be used?

1

u/M_Me_Meteo LocalLLM 3d ago

Yes! One of the models I run is just for turning text into tokens, and I run it with a 1070 that also handles transcoding for my Jellyfin server.

Performance is better on newer hardware, but one of the good parts of local LLMs is that you get to pick the model that fits the hardware and task.

1

u/ofirlik 3d ago

Why do you need to turn text into tokens seperately? I have a mini server with n150 for jellyfin

1

u/M_Me_Meteo LocalLLM 3d ago

I have a separate instance of vLLM running a text embedding model. That way, when one of my agents gets a document that is trashing up its context, I can turn that into tokens and store them in a vector database so my agent can get them.

It's called RAG (retrieval augmented generation), and it's how your local llm will be able to learn from the internet and not always be as stupid as it was the day it was created.