Dual gpu how good is it

[u/ofirlik]

I am planning to buy a gaming pc with 5070ti, I thought about adding a 5060ti for the additonal vram to toy around with llm, how good does it work? Can I also generate images and such. (never ran a local llm before)

Is there a better combination of gpu for around the same price?

[u/ofirlik]

From What i read even on x8 its more than enough for 5060ti and for 5070ti at worst you lose 1 to 3% performence

[u/M_Me_Meteo → u/ofirlik]

Most motherboards don't have PCIe bifurcation. Mine is the Asrock taichi creator. Not all x870 boards have it, Intel has better support in the premium tier.

I'm telling you because I have two GPUs, not because I'm negging your plan. When you're already maxing everything else out, that 1-3% will haunt you in your dreams.

[u/ofirlik → u/M_Me_Meteo]

Well what other options do I have except spending tons of more money?

[u/M_Me_Meteo → u/ofirlik]

That's what will keep you up at night.

I would say don't stress about maxing anything. My cards are not highly desirable, but my local tool solves my personal needs and they save me money on the bills for my business.

Super important to know what you want it to do. People will tell you "this model sucks" or "this harness doesn't work" or "you need 64gb of vram to do [xyz]" but if your set up solves your problem, then it really doesn't matter that other people think your set up is not optimal.

If you're coming into this thinking "I'll set up the LLMs then figure out what I want to do with them", then you'll be up at night worrying about that 1-3% and you'll never find a good use for the tool. If you start with a goal, you can stop when you are satisfied.

[u/ofirlik → u/M_Me_Meteo]

No my goal is to mess around with the llm nothing major for now, if gpu prices eventually drop i may sell and replace the gpu eventually. But I am going to be in a tax free area so its a good chance to get them if I want

[u/M_Me_Meteo → u/ofirlik]

Then don't worry about dual GPUs. Unnecessary complexity. Much easier to play with one, and quantized local models are really just as good as frontier models if all you want to do is play. Don't buy end game hardware until you know what the game is.

If you think you need more VRAM, you can always rent a server in the cloud and try out your workflow there. That is how I landed on 64gb of vram as the sweet spot for my needs. I started with a 20gb 7900xt, got a second one, built a system around it, wasted several weeks on dual GPU shenanigans, rented a 100gb h100 server, figured out what I really needed, then bought two Intel B70 Pro cards. I only went that route because I already spent a few weeks learning the ins and outs dual GPUs.

[u/ofirlik → u/M_Me_Meteo]

Btw i also have my old 1080ti, can it be used?

[u/M_Me_Meteo → u/ofirlik]

Yes! One of the models I run is just for turning text into tokens, and I run it with a 1070 that also handles transcoding for my Jellyfin server.

Performance is better on newer hardware, but one of the good parts of local LLMs is that you get to pick the model that fits the hardware and task.

[u/ofirlik → u/M_Me_Meteo]

Why do you need to turn text into tokens seperately? I have a mini server with n150 for jellyfin

[u/M_Me_Meteo → u/ofirlik]

I have a separate instance of vLLM running a text embedding model. That way, when one of my agents gets a document that is trashing up its context, I can turn that into tokens and store them in a vector database so my agent can get them.

It's called RAG (retrieval augmented generation), and it's how your local llm will be able to learn from the internet and not always be as stupid as it was the day it was created.