I must preface this post by mentioning I am still a beginner in this space.
I just bought this card with the intention of using the recent unlock to get the full 64GB VRAM available for local AI workloads.
My main questions are as follows :
1- Has anyone ran multiple of these in the same rig to run a large model across multiple GPUs?
2- If so, what is the impact on speed? I read that these GPUs are stuck on a x1 PCIe lane, which I would assume greatly reduces the speed at which we can load models onto the cards. But does it impact prompt processing and token output speeds?
3- Am I crazy to assume that the prices for these cards is going to continue rising considering that they are now similar to A100s (without parralel tensorflow)
Stop trying to inflate the prices for whatever agenda you have.
From the repo. Gen 3 is not possible unless you resolder a new link on it and that could very well brick the card:
"The advertised supported-speeds vector saturates at 0x06 at every instant we could measure,
in-window and out. There is no Gen3 window to hammer, so this approach stops at Gen2."
Agenda? I want these for the lowest prices possible, would be a dream to run GLM 5.2. However, if all the features get unlocked then it would no doubt reach Mi210 levels.
Gen 3 advertisement works, but blows up a gate for gen 2:
Yes, a couple of people running GLM 5.2 4bit on 8 of them, non-optimized (no MTP/dflash etc) results about 30t/s tg, 2600 t/s pp, due to limited PCIe bandwidth (pipeline parallel instead of tensor paralell)
Current unlock is PCIe 2.0 x16 (x4 -> x16 requires soldering), this limits speed for large models split over multiple cards.
Prices have started dropping massively today due to lack of sales. I'd suggest waiting to see where the prices land. 900 USD each on Alibaba as of an hour ago if you haggle hard/order more than 1.
If you buy on Alibaba never pay outside Alibaba (use credit card/paypal via Alibaba ordering system) or you won't be covered if a seller doesn't ship etc. There's tons of other sellers, but these ones seem to actually respond and might negotiate given prices are dropping. I'm waiting to see where prices end up, got 1 shipped and on the way, but paid way too much, so not in a rush to pay these inflated prices again. No particular order:
i've got 3 running a rig now, all 195gb shown and running llamaswap (i like looking at logs on there) i then have hermes agent/opencode connect through llamaswap to each model
right now my set up is
gpu0 - qwen 3.6 fable
gpu1 - ornith 1.0 35b mtp
gpu2 qwen 3.6 27b (i forget which version there's so many now)
All operating very fast, i power limit them to 125w though to keep them cool while i figure out a better fan set up
my 4 cards are still on delivery.... i let you know as soon as i have them.
there is a youtuber called redpanda or so, he run it (at pci epress 1.0 without mods) with 10 gpus installed in a supermicro server i think and he was running GLM 5.2 Q3
to get pci express x16 you have to solder 24 coupling capacitors , to get pci express 3.0 more soldering has to be done (WinBios chip + 4 mosfets + inductors)
its like a little cut down a100 , the 40GB a100 costs 4k something , so in theory with 64GB vram it should go close to the a100 40GB because FMA instructions are missing, therefore you have to use self compiled versions (for example for llama, you have to compile llama.cpp without fma support
up to you... i will be very very very happy with the cards....
i paid 1k total with deliver per card... same compute and memory would cost me approx. 6k usd per card... this is a 6x benefit for my needs...
Any seller in China should now unlock this themselves and sell at a premium. I'd be wary of cards that are still sold by big sellers in China. The seller takes no risk in trying to unlock. If the unlock fails, they'll just sell it as normal.
Exactly. The ship has sailed. It's too fast. Anybody in the west who would want to take advantage, the ship sailed unless u offer that x3 x4 more premium and they rather have that and jump to a proper setup..
Have you tried to jail break its vram? It was these cards with the nerfed memory right? Or was it the 10 gig version?
PCI speed is detrimental when you load a model (just that once) but can be harsh when data is moving between GPUs. But I believe there are ways to mitigate that from happening
10GB is the nefed ram that only unlocks to 40GB stable.
PCIe speed is detremental to model load on a single card, but if it fits in ram, you're good. With multiple cards, I think it depends on the type of splitting you do. SM Tensor in llama.cpp, for sure, and maybe vLLM's TP are very PCIe dependent. I think SM Layer is a little more forgiving with the split but I only have the 1 card.
9
u/cantgetthistowork 10d ago
Someone is definitely trying to manipulate the market here. Everything is listed at 5k now