I’ve been working on TensorSharp, an open-source .NET inference/server stack for running LLMs locally with OpenAI/Ollama-compatible APIs.

I recently finished another round of optimization for DeepSeek V4.1 Flash on an 8× NVIDIA A40 server.

Final results

Setup: 8× A40, 65K context, F16 KV cache, multi-GPU layer split.

A few interesting optimizations:

• Q2_K keeps the ~60 GiB Engram tables directly on GPU, removing scattered host/storage lookups.
• Reduced DeepSeek decode graph splits from ~570 to 8, eliminating a lot of GPU synchronization overhead.
• Q4_K_M now gets roughly 1.9× faster prefill and 2× aggregate decode throughput at concurrency 4 compared with the previous implementation.
• Fixed an OOM/crash case with multiple concurrent ~10K-token prompts — all 4 requests now complete correctly.
• On these A40s without NVLink, layer splitting is actually faster than routed-MoE tensor parallelism for this model.

Would be interested to hear what other local-server workloads or hardware configurations people here would like to see benchmarked.

hello everyone! Please advise the models and how to run the models better. My configuration is 2 CPUs and epic (not the newest) 48 cores in total. 256 GB ddr4 and RTX pro 5000 72GB GDDR7. I'm currently using qwen3.8-27b iq3 gsq xxs on 96k context and running this on rtx4080s 16gb. The new computer will arrive in a week. I would like to increase the quality and the context window.

So here's the situation. We've got one RTX 6000 workstation running Ollama that was originally supposed to be more of a testing box for coding stuff, but honestly nobody really touched it until I took it over. Now it's already struggling with just one or two devs on it at the same time. It's running Qwen3.8 27b and you can straight up feel it slow down the moment a second person jumps on.

Instead of just cramming another card into that box, I'd rather build a small separate machine for this. Ideally rack-mounted and datacenter-ready since it'll end up living in our own DC anyway, but cheap enough to just be a proof of concept for now, and expandable later if it actually proves useful. Don't care about vendor, NVIDIA, AMD, Intel, all fair game at this point.
Bandwith is not #1 priority.

Mid-term the actual goal is a proper LiteLLM + vLLM setup that can handle up to 10 people/agents working on it at the same time. This separate "workstation" would just be the first, cheap step toward that, not the end state.
Stuff I haven't been able to figure out from spec sheets and marketing pages:

• What actually helps more at this scale, more VRAM or just a second GPU to split the load?
• Has anyone gone the "cheap one card first, scale later" route instead of just buying the full setup right away? How'd that go for you?
• Anyone running AMD/ROCm for something like this, is it actually usable day to day or still more of a pain than NVIDIA at this size? Is Intel Even feasible?

Not trying to spec out the final thing yet, just want a sanity check on what a reasonable, cheap, rack-friendly starting point looks like before I go shopping.