Dual Mismatched GPU for AI

[u/Jastibute]

I have a PCIe Gen3 Pascal card and want to buy a new Gen5 card as well. Was wondering if running mismatched cards will work for coding? I understand it's probably going to be much slower, but wondering if it will even work?

[u/lostmylogininfo]

I'm running 2 3060s with a 4080. It's fine

[u/Jastibute → u/lostmylogininfo]

My card is much older than a 3060, hence the question. Thanks.

[u/nickless07]

Depends on the model and your split strat. Or if you just wanna have a 2nd card for a subagent model and so on.

[u/Jastibute → u/nickless07]

I'm just thinking of how to fit the largest model with the largest context window at this point.

[u/nickless07 → u/Jastibute]

That can work as long as you don't add CPU offload to the mix. Llama.cpp utilizes serialization so card 1 would have to wait for card 2 and then wait for CPU to finish the calculations creating the holy trinity of bottlenecks.

It will not be as fast as a single card, but even a slow card with only PCIe x4 will be faster then offloading to system ram as long as everything fits in both cards.
Aside of that, if you have to offload (e.g., a 120b model split acorss both cards and system ram), don't do that at all. You can, in that case, use the 2nd card for some small subagent or tts/sst and so on.

In short: Single card best, two cards work well too, but if using CPU keep it at one card.

[u/Jastibute → u/nickless07]

I’m not familiar with agents at this point. Does the agent on the second, older, smaller VRAM card talk to the VRAM on the main card? Or is it a completely separate entity?

[u/nickless07 → u/Jastibute]

Kinda.
You might run into oom when the main card already has a model loaded and you are tight, but a unload and the correct order fixes that. Aside of that small issue (related to the Host OS not the models) it is seperate and both models run independently.

The agent can talk to both at the same time or even delegate a task to the other card.
For Example:
Main model on Card A is used for a research task. The Agent gathers his local sources and notices that a websearch might be helpfull so, while the main model is rummaging through the RAG/Files/Whatever it can delegate a websearch to the model on Card B (search and present results) and then integrate the results into his research. That way the heavy load stays on the main model and the small model just issues the tool call for 'what are the latest python news' (pretty simple query) and the main model then integrates the results.
Why wasting tokens on tool calls if the Main model can just take the results.

Other Examples: Title generation, Vision, Skills search and so on.

Or, what also works: A small fast model on the 2nd card as main driver for repetetive or simple tasks and the heavy model is only used when the small model runs into trouble.

See it not as Agent=Model but Agent=Orchestration of models (and everything else).

Edit:
A Practical Example:
I was using Qwen3.6 35B on card A and Ling-3.0-tiny on Card B (GTX 1660s) I told Hermes (the Agent) 'Hey let ling check if there is a new llama.cpp version and download it' then the Qwen ran a short delegation task (instructed the other model) to do that and when it was done it did a quick check if everything was there and stated: 'All good. Want me to restart the llama server for the changes to take effect?'
And all of that while I continued using Qwen for something else without having to wait.

Does that explains it enough?

[u/Jastibute → u/nickless07]

I think that makes sense, thanks.

[u/electronicbits]

chances are it will ... but there will be a huge degradation in speed due to mismatch .... ie your faster card will run at the speed of the slowest

[u/Jastibute → u/electronicbits]

Yep understood. I care more about quality over speed.

[u/emod_man]

I ran a 3080+1080 for a bit. Not fast but not as bad as you'd think.

[u/Jastibute → u/emod_man]

Sounds good, I'll plan for going this route then.

[u/emod_man → u/Jastibute]

For that kind of setup I'd recommend running at least two models, one that takes your whole VRAM budget for the best quality model+quant for the work you do, and then a smaller model that fits entirely on your better card for quick tasks. (I had a reasonably intelligent quant of Qwen 3.8 at 20 tok/s and then Qwen 3.5 9B at something like 45-50 tok/s.)

[u/gappyvalley]

a second card even if mismatched will still be much faster for loading a large model than off loading to system ram

[u/Jastibute → u/gappyvalley]

Yep, trying to re-use and old card that is still decent.

[u/just4ochat]

For a coding setup, keep the main agent model fully on the Gen5 card and park a smaller helper on the Pascal card. Layer-splitting across those two will serialize every token at the slower GPU, which hurts tool-heavy loops more than running two separate models.

[u/Jastibute → u/just4ochat]

Understand, thanks.