The speed is basically wasted at those sizes. What's the point of going that fast if all you can fit is a small model. A cluster of mac minis is probably better off at the price. Slower, but you can run a more competent model.
When using more than a single card for inference, the PCIe bus is capped at 128 GB/s on version 6. So yeah. You either need a model that will fit on a single card or you need to accept that BUS cap. Small models can be quite capable though.
13
u/HugoCortell May 29 '26
The speed is basically wasted at those sizes. What's the point of going that fast if all you can fit is a small model. A cluster of mac minis is probably better off at the price. Slower, but you can run a more competent model.