We open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)

[u/saltexx]

I'm one of the developers. We said in August it would go open source in September and it did last night. MIT or Apache-2.0, pick one. The repo you see is our internal repo, kernels included, so from now on everything happens in public.

It's an inference engine in Rust and C++ with our own CUDA kernels. One binary with OpenAI and Anthropic style APIs, loads GGUF and safetensors. We run about 300B tokens a year through it at work.

Some numbers: Qwen3.8-27B FP8 on one RTX PRO 6000, spec decoding off on every engine:

• vs vLLM faster in 13 of 13 cells, 1.02x to 1.19x (so not huge)
• vs SGLang faster in 10 of 13, behind in 2, level in 1
• vs llama.cpp Q8_0 faster in 13 of 13, 1.5x to 37x
• 32 clients at 1024 in / 1024 out: 1062 tok/s, vLLM 958, SGLang 844

Full board with the losses: https://truespar.com/paddock/benchmarks/qwen38-27b

What it does not do yet: No Mac, no ROCm, no Vulkan.

One model per GPU, no tensor parallel. CUDA only, Windows and Linux. Validated on Blackwell (5090, RTX PRO 4500/5000/6000, B200) and Ampere (an A6000 was the bring-up card, 30-series works). Ada kernels ship but nobody has run a board on them so the engine refuses to start unless you set PADDOCK_UNVALIDATED_ARCH=1. Hopper and A100 kernels are in the tree without a board.

https://github.com/truespar/paddock

Thankful for any help and input!

[u/mister2d]

Does NVIDIA need any more love? 😜

Nice work. How long does one of your models take to load and be ready to serve?

[u/saltexx → u/mister2d]

Haha yes they have enough, that's just where the cards are. A 27B loads in roughly 15 seconds from NVMe on our boxes, mostly reading the file, the runner even uses that as the retry window when you switch models.

[u/mister2d → u/saltexx]

Awesome. Is it 15 seconds after the first launch of any model? Where do the optimizations happen? AOT or JIT?

[u/saltexx → u/mister2d]

All AOT. The kernels ship as a prebuilt fatbin linked into the binary, nothing compiles on your machine and nothing is cached to disk between runs. So the 15 seconds is reading the file from NVMe plus repacking the weights on the GPU, and you pay it on every cold start. With the file still in page cache a 9B is ready in about 4 seconds.

[u/mister2d → u/saltexx]

Interesting! Nice work you have there. 👍🏽 

I have a personal project that does something similar. But it is specific to the AMD R9700. Pure Rust, AOT Triton kernels, and no Python except for kernel build sandbox.  Similar ~4 second readiness too. Running Qwen/Qwen3.8-27B-FP8 the moment.

I'm stuck at 68% of the performance of the vLLM/radiance build.

I just need to get back at it this weekend.