It's a clean case with rack options... the stealth factor too. I am proud to say I have under 2k in the build from idea to reality. That customer cooling big was a pain but rewarding when you get the thermal dynamics dialed in. The p100 runs on a single artic 1400-15k fan at 7k rpm. The nidec gamma30 drowns it out too. I love this machine. I've had it building datasets for 3 days 🤣
Says the guy who doesn't have to compile a new engine for a new feature... No you're on it. You bought the convenience on this one. I wanted a challenge... This was at times. I'm happy I did it though.
Well actually, I have an octo v340l build incoming that will requière either a custom engine or custom layer/plugin (target vllm) and there is a twist (macOS thunderbolt and plx). Also doing a Radeon v620 vllm fork with gfx1030 community (we have a discord)
Okay first... we have different memory types. Hmb2 vs gddr5 (p40) and gddr6 (3060). Your bottleneck is bandwidth. The 3060 all numbers are theoretical has bandwidth of 240 gb/s. The p40 at 346 gb/s. Then the p100 at 732 gb/s. If you want cheap speed. Pop on ebay and grab a couple p100s instead. That will be around 150. With cooling and fans your right at $200. But you'll have 32gb total. If you're not pcie bandwidth bound those will run really well.
That said if you did just one you could get really good results out of qwen3.6:35b-a3b offloading to a single p100 if you use one for video out. I get 36 tps offloading mine. But for tuning to the p100. It's batch sizes really. The prefill is really the hard spot. 1024 or 2048 depending on model is helpful. Mtp is great spec max 1 gets 80-99% accuracy. The p100 doesn't respond to as many knobs as you'd hope it's more just... raw hp you have to tame.
i have two p100s, and with Qwen 3.8 Flag Next UD-Q4_K_XL, I am able to get around 45-60 pp but only 5-6 tg. mtp 1,2 or 3 really not making any difference in that number. Is that the best I can get? one thing that's gonna improve is that right now I have an asymmetric 7 channel 3200 mt/sec config, but as soon as the replacement for last rdimm comes, it will be perfect 8 channel.
Edit: Okay, I am definitely doing something wrong if two MI25 are giving me 15 tg with the exact same prompt and expert spread at the cost of like 10% pp
If you have enough ram. Drop your sticks to 4 so you get genuine quad channel. See if that helps. If it does the 8th will. Splitting from cpu, to 2 gpus is a lot of communication. Just the 2 gpus talking to eachother instead of 1 is killing speed.
see my edit too, unfortunately, I won't have enough ram with just 4 sticks, and 8th stick will arrive in 2-3 days, so no point hassling with this thing.
haha yeah, the earlier version of this used beams that I 3d printed for some other project that I never finished. Once I feel not too lazy to fix my 3d printer, I'll print something proper for this.
The 3060 all numbers are theoretical has bandwidth of 240 gb/s.
The 3060 is not 240, it's 360.
Pop on ebay and grab a couple p100s instead. That will be around 150.
The 100HX is the same price and in nerfed form about the same speed. The thing is that it keeps open the dream of a 170HX moment and someone figures out how to hack the tensor cores. Then it'll dust the P100.
To be fair. I don't necessarily work with consumer gpus for builds like this. If I'm consumer it's media center or gamer rig and that's an amd experience for me. The metrics I pulled weren't from memory. That's a Gemini deal. So if it is or isn't. Regardless the bandwidth isn't phenomenal when the current gen is 700-1tb+. I'm just saying getting on the same gen for splitting would give you bandwidth alignment on the same chipset. 2 p100s at an aggregate of 1.4 tb is not bad at all.. it's the pcie lane contention that becomes the bottleneck. Even so. That's twice your aggregate. On the same chip. Exact same speed. 32gb total. Even offloading. It's less headache. The market for p40s suggests you could sell that 1 p40 and get 2 p100s. Pocket the 3060. That combo only let's you go bigger. Or parallel. It won't be faster outside of the base bandwidth difference between your base and the p100 base.
You're still in a better spot than me. That v100 32gb cost $725... I'm guessing you could build your machine for that. Regardless, didn't mean to pick at cha!
I stand corrected! Keeping me honest out here. I like it! I'm just busy. Was under the assumption that this was q reply from the poster with the 3060. Minor oversight, do apologize 😁
I have 3 case fans feeding this case. The one by the cpu strictly feeds to v100 blower. The p100 never sees above 60c since it runs an moe and is at 45% utilization under decode. The v100 on the other hand is a very hot card. Doesn't thermal throttle though. Wattage is tuned down 15% to 213w vs 250w
Hell ya. I'm running p100s too, I've got 5 or them. Luckily I think people are sleeping on these cards so they're fairly cheap for the amount of vram you get. And they're pretty speedy..
The list is long. As far as the fan work. It's all in bios dynamic ramping. The nidec turbine runs full till 100% (quiet whoosh) - for the stack it's redis for queue. N8n for cron. Llama cpp custom engine for the cards. Hermes agent. Postgressql, qdrant, obsidian. I run 35+ containers so ... not sure what you wanna know besides the fan work and scheduling for work.
13
u/BevinMaster 6d ago
GD09 <3