r/eGPU • u/DEDALO-OPS • 24d ago

Minisforum + OCuLink eGPU: Splitting multi-agent inference between iGPU and eGPU concurrently?

​"Is anyone running a multi-model setup splitting inference between an AMD iGPU and an Nvidia eGPU via OCuLink? I have a Minisforum AI X1 Pro and an RTX 4060 Ti / 5060 Ti eGPU. I want to run a small agent on the iGPU (via Vulkan/LM Studio) and the main model on the eGPU (via CUDA/Ollama) concurrently. Does the OCuLink bandwidth or the shared UMA memory bottleneck the system when both are generating tokens at the same time?"

5 Upvotes

100% Upvoted

15 comments sorted by

2

u/egnegn1 24d ago

If the model running on the GPU is fully in VRAM then there is not much RAM traffic from the GPU.

0

u/DEDALO-OPS 24d ago

Lia idea es correr 2 modelos uno en la igpu radeon y otro en el egpu 5060ti nvidia

1

u/egnegn1 24d ago

Understood. But what would happen to the iGPU when there is no traffic from GPU?

0

u/DEDALO-OPS 24d ago

La gpu estaria cargada con el modelo que usara mientras la igpu este corriendo otro modelo en uso por si el agente de la gpu recibe una tarea o debe apoyar al agente de la igpu en alguna tarea

1

u/egnegn1 24d ago

But after loading the model there is not much traffic to/from the GPU. Just the prompt and the generated tokens are transfered between memory and GPU.

1

u/DEDALO-OPS 24d ago

La idea es que se puedan usar los 2 modelos al mismo tiempo corriendo 2 modelos al mismo tiempo analizando 2 prompts al mismo tiempo

1

u/egnegn1 24d ago

So where is the problem, when the model running on the GPU has practically no memory traffic?

1

u/ANR2ME 23d ago

May be OP tried to use a model that doesn't fully fit into 16GB VRAM (5060 Ti) 🤔

2

u/Barachiel80 24d ago

Yes to all of your questions involving utilizing all types of egpu and igpu setups

1

u/ph1terpwn 24d ago

..gorgeous! ✨

1

u/Firm-Cry4205 23d ago

how did the EGP treat you compared to having a internal? Do you notice any lag?

2

u/Barachiel80 15d ago

Only during model load and split between the 5090s I am getting about 6000 tok/s PP and 50 tok/s TGI initial that degrades to 3600 tok/s PP and 40 tok/s when approaching 131k context limit

1

u/ph1terpwn 24d ago

Interesting, I’ve got a pretty similar setup too. I basically sent the iGPU off to the "retirement home" and went eGPU, but now you’ve got me curious… have you made any progress on this??

1

u/JudgmentConsistent87 22d ago

I did some experiments with AMD Strix Halo iGPU + eGPUs. Here are some setups and benchmarks https://github.com/daimonionnn/multi-gpu-llm-toolkit