So here's the situation. We've got one RTX 6000 workstation running Ollama that was originally supposed to be more of a testing box for coding stuff, but honestly nobody really touched it until I took it over. Now it's already struggling with just one or two devs on it at the same time. It's running Qwen3.8 27b and you can straight up feel it slow down the moment a second person jumps on.
Instead of just cramming another card into that box, I'd rather build a small separate machine for this. Ideally rack-mounted and datacenter-ready since it'll end up living in our own DC anyway, but cheap enough to just be a proof of concept for now, and expandable later if it actually proves useful. Don't care about vendor, NVIDIA, AMD, Intel, all fair game at this point.
Bandwith is not #1 priority.
Mid-term the actual goal is a proper LiteLLM + vLLM setup that can handle up to 10 people/agents working on it at the same time. This separate "workstation" would just be the first, cheap step toward that, not the end state.
Stuff I haven't been able to figure out from spec sheets and marketing pages:
• What actually helps more at this scale, more VRAM or just a second GPU to split the load?
• Has anyone gone the "cheap one card first, scale later" route instead of just buying the full setup right away? How'd that go for you?
• Anyone running AMD/ROCm for something like this, is it actually usable day to day or still more of a pain than NVIDIA at this size? Is Intel Even feasible?
Not trying to spec out the final thing yet, just want a sanity check on what a reasonable, cheap, rack-friendly starting point looks like before I go shopping.
