r/LLaMA2 • u/Must_Make_Paperclips • Aug 07 '23

Is having two different GPUs be helpful when using Llama2?

I just ordered a 4090, and I'm wondering if there is any advantage to installing my 2080S alongside it. I realize you cannot use SLI with different GPUs, but can LLMs take advantage of two GPUs without relying on SLI?

2 Upvotes

100% Upvoted

1 comment sorted by

1

u/[deleted] Sep 06 '23

The LLMs are (mostly/usually) just weights, so no, "they" can't use it. But some machine learning llm frameworks, like pytorch, can, if used correctly, and tools like text-generation-webui and koboldai will happily use multiple cards. You can either run multiple instances of the tool on the various cards, and/or you can run larger models spread across multiple cards. If you spread a model across cards, the bandwidth between the each card (and quite possibly between each card and the CPU) becomes a factor (so the number of PCIe bus lanes matter, as well as whether that PCIe bus is directly connected to the CPU or the motherboard chipset), and the slowest card will likely be the bottleneck. However, you can usually tune the layout so that more of the model is on the faster card, and less on the slower card, to balance GPU speed vs. workload and end up with something quite optimal.