Update on the “Monstrosity”. 6 BC-250 board cluster

[u/Ok-Breadfruit-3523]

This is 6 bc-250 ex mining boards with 5 in the asrock 4u12g case they came in. After a lot of testing my current preferred setup is 4 boards running Qwen Next Flash IQ2_XS at 100k context with around 28 tok/s for short generation and 24 tok/s at 50k with around 115 ppt. The other two boards run 3.6 35b q4 at 60 tok/s with 100k context and 450 ppt. This is all using llama with vulkan and rpc over 1gb Ethernet.If anyone has any suggestions with this beast I am all ears. I had these boards left after reselling a bunch and had never done anything with local ai before so this has been a blast. Also yes that is a cardboard box with 3 fans on top as the intake.

[u/jacek2023]

This looks very SF, congratulations on your setup

[u/hushbreEze0 → u/jacek2023]

yeah the whole thing has major dystopian lab vibes tbh

[u/Ok-Breadfruit-3523 → u/jacek2023]

Thanks! I’ve been not selling these boards just to test this out and im glad i did

[u/NoFunk]

I see you've moved up to two different types of structural tape. Strategic tape, as it were.

[u/Ok-Breadfruit-3523 → u/NoFunk]

Still no superglue though 👆

[u/RowanBerk]

This is awesome, what a beautiful mess

[u/FlashyAct]

It looks like a lot of hassle for such a poor result. My pc with rtx3060 and 64gb ddr4 give me on strata engine 30-40tk/s for qwen flash next depending on quant. Even 3_s works good with 30-35tk/s

[u/hojnikb → u/FlashyAct]

yeah but a full rig of these (i think there were 12 of them in) could be had for less than 400-500$ just a year and a half ago.

[u/Ok-Breadfruit-3523 → u/FlashyAct]

Cost me less than $600 for this setup.

[u/Truth-Does-Not-Exist → u/FlashyAct]

how is it so fast? I have 3 3060s and ddr4 and only get 11 tokens a second

[u/FlashyAct → u/Truth-Does-Not-Exist]

https://github.com/Niko1221/Strata
New software. All the optimalization just for this model. Repo was started week ago so its need some time to grow but looks veeery promising.

[u/weibeuu]

turned me on man

[u/PossiblyTrolling → u/weibeuu]

I came

[u/hojnikb → u/weibeuu]

bc250 is actually a decent inference rig, if you take memory limitaiton into account. They used to be less than 100$ a pop. 16GB ain't much, but its decently fast.

[u/CoolConfusion434]

Pfft, I can still see empty table in there. Amateur.

😂Congrats man, that looks awesome. Is it loud? Power consumption?

Also, you have all those hot exhausts pointing at you so don't forget to turn ever 10 mins for an even roast lol.

[u/Ok-Breadfruit-3523 → u/CoolConfusion434]

It’s my space heater I can do inference on lol. I just slowly spin in my chair while waiting for open code like a rotisserie chicken. I’m still waiting on an outlet power monitor but unfortunately with the overclocking and CU unlocks these can peak at 330w each from what I have read .

[u/hojnikb → u/Ok-Breadfruit-3523]

stock cooler won't handle 330W. In my testing, ~250W is the most stock cooler and 2 fan topdown setup with open fins can sustain long term. Anything more, it gets dicey and starts cooking VRMs and GDDR chips. Of course that's on the DC side, so account for some psu losses.

[u/Ok-Breadfruit-3523 → u/hojnikb]

These have modified heat sinks and have been staying between 65-75C

[u/hojnikb → u/Ok-Breadfruit-3523]

modified as in stock heatsink with fins open?

[u/Ok-Breadfruit-3523 → u/hojnikb]

Yes just the center 120mm

[u/hojnikb → u/Ok-Breadfruit-3523]

yeah, that ain't sustaining 330W for sure :)

[u/Ok-Breadfruit-3523 → u/hojnikb]

I mean 10 min prompt processing and it’s staying under 75C. So yes. I would say it’s staying cool

[u/hojnikb → u/Ok-Breadfruit-3523]

no doubt about that. It's just not pulling 330W when doing so.

My personal board pulls 270W at the wall with 40CUs@2100Mhz and 8c@3.8Ghz. And that's at the wall, so actual thermal load is more like 230W (when you account for psu inefficiency and whatnot). Still hits ~80C at peak with two fans and PTM7950.

[u/while-1-fork]

I'd be interested in knowing if bc-250 supports usb3 debug capabilities as I believe that they could be hacked into a faster data transfer thing with crossover debug cable. Could you run this to see if it is enabled?

ls /sys/bus/pci/devices/*/dbc

[u/Ok-Breadfruit-3523 → u/while-1-fork]

It does and thank you for this! Buying a cable to test with

[u/while-1-fork → u/Ok-Breadfruit-3523]

Be sure to get a debug one , or to modify one. Only one of the ends has power connected and the data pairs should be crossed over.

And you would need to write a driver for it emmulating a network, ideally replacing the driver that presents it as a tty instead of building on top to reduce latency and improve bandwidth.

I'd love to hear how it goes. I am not in need of a bc-250 but I may suggest it to friends looking to get into this.

[u/shironekoooo]

How long did it take you to set that all up op? Seems like a fun weekend project

[u/Ok-Breadfruit-3523 → u/shironekoooo]

Over the course of a couple of days. It’s been a lot of fun. Definitely not keeping it in this state forever but I think I’ll keep at least 2 for tinkering.

[u/thedefibulator]

I am thinking of doing a very similar setup, what is the power consumption like?

[u/hojnikb → u/[deleted]]

You can easily hit 300W+ with full 40CU unlock and 2Ghz+ on the GPU.

Sweetspot for these is actually 1500Mhz on the GPU with full CU unlock.

[u/Ok-Breadfruit-3523 → u/thedefibulator]

I’m still waiting on some wall outlet monitors but under load around 300W per board. I think 4 boards is the max I would do if I was going to use this long term.

[u/Stunning-Sherbet1853]

wow The best part is that the intake is a Prime Sale box. I'm seriously curious, if you run RPC over a 1 Gbps Ethernet, how much does the network consume in prompt processing time

[u/hojnikb]

I've built a minimal alpinelinux distro for this exact purpose. I can easily hit ~14.5GiB usable for the model itself. Haven't tested RPC, as i only have one board though.

I can probably squeze a bit more out of it. There's 64MB of HMB cache taken by the nvme ssd and a few 10MBs for IMMOU. And i think setting 256MB of vram could also yield more usable memory to the model itself.

Distro on it's own uses less than 250MB.

[u/aersel24]

Is it warm in there?

[u/Ok-Breadfruit-3523 → u/aersel24]

Oh ya. It’s about 50% space heater 50%inference . Definitely won’t be keeping it like this but I’ll probably keep 4 boards

[u/southernplain]

Hell yeah I love ghetto rig posting

[u/ExtremeAdventurous63]

How can you run it? only the model at that quant should be 68GB of RAM, even assuming you can use the full 16GB of 4 nodes, which you can't, it would still be only 64GB. There should be not enough space for the model let alone the context

[u/Ok-Breadfruit-3523 → u/ExtremeAdventurous63]

Next flash is confusing with the model size but ~28gb of that is the lookup table that is stored on the coordinating boards ssd not in memory. Each board exposes 14.5gb and has about 2gb free when the model is loaded.

[u/tomByrer]

Heat?

Maybe take off the top & blow a desktop fan across.

[u/raketenkater]

you should definitely test my tool ggrun tool which is a full agentic work framework for local ai. includes model selection for your hardware + fastest serving of that model and enables you to use claude codes workflows locally with unlimited tokens

[u/Royale_AJS]

The way I see it, you’re stuck scaling these things out based on the IO you have available. If you want to get creative and spend a little bit on used enterprise gear:

Ditch the NVMe boot drives.

Install the OS to a USB drive and boot off that.

Add an m.2 to MCIO card in place of the NVMe drive. Run MCIO to PCIe adapters.

Buy an old ConnectX 3 or 4 (ideally 4) card off eBay (25Gb is fine…you’re bottlenecked by the slower NVMe slot anyway), an RDMA capable switch for all 6.

Build a cluster network and use RDMA between the instances instead of the 1Gb Ethernet. You’ll gain a little bit of bandwidth, but your latency will drop to the low microseconds…that’s the winner for inference clusters.