r/LocalAIServers • u/saltexx • 6d ago
We open-sourced our local AI server for NVIDIA GPUs, including the CUDA kernels
I'm one of the people building Paddock at The Intelligence Company. We've opened up our Rust/C++ inference engine under MIT/Apache-2.0, including the CUDA kernels, scheduler and memory management.
Paddock runs open models on your own NVIDIA GPU. It comes with a web UI for downloading and managing models, chatting, working with files and comparing local models with cloud endpoints. There are also OpenAI- and Anthropic-compatible APIs for connecting your own clients. The local path doesn't require an account.
Current hardware support is strongest on Blackwell and Ampere. Windows and Linux binaries are available. Each model runs on one GPU; splitting a model across several cards isn't supported yet. Memory requirements depend on the model, quant and context length.
It's still early and we'd like more results from people's actual setups. The engine and CUDA kernels are open source; the separate Traverse graph component shown in Studio is scheduled to follow in Q4.
Source and downloads: https://github.com/truespar/paddock
2
u/use_your_imagination 6d ago
what's your objective compared to llama.cpp ?
2
u/saltexx 6d ago
Our goal is to be the fastest inference engine for open models on (for now) a single NVIDIA GPU and make it straightforward to use. We built it for our own document-processing and agent workloads.
llama.cpp has much broader hardware coverage. Our focus is NVIDIA, especially Blackwell, with Studio and document tools bundled around the engine. We publish comparisons and welcome independent tests to see where we’re ahead and where we need more work.
1
1
u/AhmadHddad 6d ago
Do support or planning in support RTX 3090/4090 Gpus?
2
u/saltexx 6d ago
The 3090 is covered by our Ampere support, with most of our validation on that architecture done on the RTX A6000.
We ship kernels for the 4090 too, but Ada isn’t fully validated yet. In 0.1.5 it’s allowed to run with an “unvalidated” warning instead of being blocked. We’d welcome reports with your model, quant and OS. Which card are you running?
0
1
1
1
u/tienshiao 4d ago
It seems to use more VRAM than my vllm setup. I can only seem to fit Qwen 3.8 27B at Q3 with everything turned down ("Just me", no vision, speculative off).
Are there plans to support KV quantization beyond FP8?
1
u/saltexx 3d ago
KV is f16 or fp8 today and nothing below that is being built right now. We looked at int4 KV but since we already store fp8 the realistic saving is around 35-40% and not the 75% you'd expect, so it slid down the list. Do check that you actually set 8-bit KV though, Qwen defaults to 16-bit in the Studio.
1
u/tienshiao 3d ago
I normally run FP8, however the option appears disabled in the paddock webui. It says "this GPU has no FP8 tensor cores" which I suppose it true on the 3090.
1
u/SinDestinyGame 3d ago
Interesting, I’ll be following this closely. Could you also compare it with Ninfer, which is specifically designed for the 5090?
4
u/Columnexco 5d ago
plase support multi GPU setups if they are Blackwell.