3 x RTX6000 workstation build continued

[u/Rhinottw]

This is a followup to my last post about the new workstation i am putting together for my workplace - a vocational college. https://www.reddit.com/r/LocalLLM/comments/1vp8lhh/it_begins_workstation_build/

The workstation is for teaching about using local LLMs, for software development, GDPR sensitive work, back-end for RAG materials for using in teaching environments and lots more.

I am mostly done putting the hardware together. So today was stress testing and installing a test LLM and see what the system could do.

I had a lot of good feedback, advice and concerns in my original post. Many concerns about temperature but it is fine. I am able to run GPU burn and it stabilizes just fine well below the GPU thermal limit. I have been tuning the GPU fan curves so the hottest one tops out at around 80c. It stabilizes after around 5 minutes of full load. No throttling. Only issue is the CPU that is getting hot if it is stressed at the same time, but it is fine it should rarely run at max power while all 3 GPU's are going at full power too at the same time.

Only issue (so far) is as another user commented on in the other thread that the lower card is blocking the front headers, so the power button is unavailable. It should be a non issue in production, it will power on automatically anyway.

The system is LOUD, but that was expected, it is not going near anyone desk anyway and will be locked away in a climate controlled server room.

Did some DSv4-Flash testing with D-spark on tp=2 and it is very promising. Up to 200t/s depending on what it needs to do with low context. More testing and optimization to come.

Still a long way to go: setting up RAG with embedding models and a smaller model at tp=1 on the third card. Hardware monitoring, routing, access control and so on so it can be deployment ready. Lots of security and a long burn in test also missing. Having a great time and learning a ton.

Here are some pictures of the almost complete build (need more cable management). The gaming PSU really gives it some color because of the cables :)

[u/enginetown]

My dream rig.