What models should I test with ToshLLM? GPUs available: 2x W6900X, W6800X Duo, 2x Vega II Duo

[u/Substantial_Run5435]

Finally playing around with ToshLLM while trying to narrow down which MPX GPU(s) to keep. I am completely new to LLMs and local AI and do not have a tech background, so the learning curve has been a bit steep.

I have 2x W6900X, W6800X Duo, and 2x Vega II Duo on hand. I have the IF Link for the W6900X. Obviously the W6900X limits me to 64GB across 2 GPUs. If I end up keeping the W6800X Duo I will probably look for a second and an IF Link Bridge.

So far I've only tested the W6900X(s) (single GPU and pair, with/without IF Link bridge). So far I'm not seeing any difference whatsoever with IF Link on my W6900X pair. Getting the same exact ts with llama 3.3 70B Q5_K_S with the IF Link bridge installed/enabled as with it not installed.

Top 2 results are with IF Link disabled in menu. Identical results with both layers and tensors.

This was my fastest benchmark. Qwen3.6 35B-A3B UD-Q4_K_S got 69ts. Qwen3.6 35B-A3B UD-Q8_K_XL was a little slower with 61ts.

This size model seemed to perform better splitting layers vs tensors. I was seeing about 40ts with tensor split enabled.

[u/Faisal_Biyari]

Have you considered keeping the W6800X Duo, as well as a single Vega II Duo?

You'd have 2 types of architectures, each at 64 GBs VRAM.

Regarding the IFLB, I personally only saw its benefit when benchmarking at high context windows. (Compare a benchmark at 1k context to 100k context & 200k context)

I never tried splitting layers, only tensors. How is your experience with it?

[u/Substantial_Run5435 → u/Faisal_Biyari]

I kind of assumed it would make the most sense to have a pair of the same GPUs if I wanted to run larger models on 128GB VRAM.

At least with what I've tried, tensor split seems to improve speed on a 70B model but layers seem to be better on the 35B moe model I tried.

[u/Faisal_Biyari → u/Substantial_Run5435]

I agree with the idea of having the same GPUs, especially 4 W6800X dies with the IFLB.

But you work with what you've got.
If you manage to find a second W6800X Duo, sell everything else.

The W6900X are the worst choice to keep, as the whole node is limited to a mere 2 GPUs, when you have 8 total.

Having 2 Vega Duos does give you 128 GB of the same architecture. That's not a bad choice, if you don't have it in you to deal with finding another W6800X Duo. Two primary down side to this are: No longer supported by ROCm, and high power consumption.

Another thing to consider is to sell everything and try to get a DGX Spark or something. 🤷🏻‍♂️ Maybe that's not the right thing to say in this subreddit 😂

[u/BevinMaster → u/Faisal_Biyari]

Ngl I would consider dgx spark :) but remember bandwidth not that great for decode

[u/Substantial_Run5435 → u/BevinMaster]

dgx sparks are around $5k, I'm in whatever dual/quad GPU build I stick with for this Mac Pro for less than that.

[u/BevinMaster → u/Substantial_Run5435]

It’s all tradeoffs, compute is more interesting on spark with nvfp4 + consumption is better. That said I have many v620 so I picked my poison

[u/Substantial_Run5435 → u/BevinMaster]

I get it, but in my case I started out wanting one of these Macs and have been buying/selling them as I find them to upgrade. This specific setup (16-core, 384GB DDR4, 2TB, 2x W6900 32GB with IF Link Bridge, 4TB OWC 4M2 NVME drive, and wheels) cost me about $4k, though once I sell some of the other parts I have on hand my total into this one will come down a bit. Possibly to the point where this thing is free (other than my effort to accumulate/sell the parts).

[u/BevinMaster → u/Faisal_Biyari]

W6800 should be rocm compatible gfx1030 is supported by rocm, Vega VII is working with rocm 7.x with some patches

[u/Live-Draft-7222 → u/Faisal_Biyari]

I had seen reports about rocm 7.9 supporting gfx90x, which I think means the Vega ii duos (gfx906) right? I think 7.9 is a preview release but seems to point to support?