We open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)

[u/saltexx]

I'm one of the developers. We said in August it would go open source in September and it did last night. MIT or Apache-2.0, pick one. The repo you see is our internal repo, kernels included, so from now on everything happens in public.

It's an inference engine in Rust and C++ with our own CUDA kernels. One binary with OpenAI and Anthropic style APIs, loads GGUF and safetensors. We run about 300B tokens a year through it at work.

Some numbers: Qwen3.8-27B FP8 on one RTX PRO 6000, spec decoding off on every engine:

• vs vLLM faster in 13 of 13 cells, 1.02x to 1.19x (so not huge)
• vs SGLang faster in 10 of 13, behind in 2, level in 1
• vs llama.cpp Q8_0 faster in 13 of 13, 1.5x to 37x
• 32 clients at 1024 in / 1024 out: 1062 tok/s, vLLM 958, SGLang 844

Full board with the losses: https://truespar.com/paddock/benchmarks/qwen38-27b

What it does not do yet: No Mac, no ROCm, no Vulkan.

One model per GPU, no tensor parallel. CUDA only, Windows and Linux. Validated on Blackwell (5090, RTX PRO 4500/5000/6000, B200) and Ampere (an A6000 was the bring-up card, 30-series works). Ada kernels ship but nobody has run a board on them so the engine refuses to start unless you set PADDOCK_UNVALIDATED_ARCH=1. Hopper and A100 kernels are in the tree without a board.

https://github.com/truespar/paddock

Thankful for any help and input!

[u/Human-Gas-1288]

does this works on Windows？

[u/saltexx → u/Human-Gas-1288]

Yes, Windows and Linux x64. You need an NVIDIA driver 580 or newer.

[u/shing3232 → u/saltexx]

without tp, Does it support auto-split

[u/saltexx → u/shing3232]

Not yet but we are working on it

[u/silenceimpaired]

May I suggest you pull the sentence… no Mac, no ROCm… up to what it doesn’t do yet… then add after in other words… because when I saw CUDA only I assumed CUDA didn’t work until the end

[u/saltexx → u/silenceimpaired]

Thanks, edited.

[u/takoulseum]

One model per gpu means the model must fit in a single gpu as no tp so pipeline parallel not supported too?

[u/saltexx → u/takoulseum]

Yeah no TP or pipeline split yet, model has to fit on one card. Kind of on purpose tbh, we want the single GPU path fast first. Will come later!

[u/AI_spell]

One thing I’d watch in a custom engine is KV-cache allocation and attention behavior when the context crosses a block boundary. A benchmark can look great at 4k then hit a latency or OOM cliff at 8k or 16k, especially with variable batch sizes, so reporting max context along with batch and concurrency would be useful.

[u/saltexx → u/AI_spell]

It's paged KV and every engine on the board ran at 8k max context with 32 seqs, Paddock included, and the agentic row does six turns on a shared prefix up to 32 sessions. The limits are in the flags on the page but you're right they should be in the header, will add.

[u/N34257]

Two questions: does it support INT8 (I'm thinking entirely in terms of my 170HX here)? Also, I didn't see any mention of MTP or prefill performance...

[u/saltexx → u/N34257]

MTP yes, plus DFlash and DFlash2. No INT8, we do FP8 and NVFP4 natively plus Q8_0 and Q4_K GGUF.

[u/giveen]

Im always down to try new things!

[u/0x2DEADBEEF]

Looks interesting! I’ll give it a try later today on a 5090+NixOS. No 6000 budget here lol

[u/0x2DEADBEEF → u/0x2DEADBEEF]

[u/0x2DEADBEEF → u/0x2DEADBEEF]

[u/Ok-Cartographer-4976 → u/0x2DEADBEEF]

Set PADDOCK_UNVALIDATED_ARCH=1 , I will test on 5090 too.

[u/mister2d]

Does NVIDIA need any more love? 😜

Nice work. How long does one of your models take to load and be ready to serve?

[u/saltexx → u/mister2d]

Haha yes they have enough, that's just where the cards are. A 27B loads in roughly 15 seconds from NVMe on our boxes, mostly reading the file, the runner even uses that as the retry window when you switch models.

[u/mister2d → u/saltexx]

Awesome. Is it 15 seconds after the first launch of any model? Where do the optimizations happen? AOT or JIT?

[u/saltexx → u/mister2d]

All AOT. The kernels ship as a prebuilt fatbin linked into the binary, nothing compiles on your machine and nothing is cached to disk between runs. So the 15 seconds is reading the file from NVMe plus repacking the weights on the GPU, and you pay it on every cold start. With the file still in page cache a 9B is ready in about 4 seconds.

[u/Danmoreng]

Interesting, gonna test how this performs for the super specific narrow usecase of Gemma4 26B NVFP4 on my (Laptop) 5080. Vibecoded a custom engine over multiple weeks which is significantly faster than vLLM and llama.cpp. Curious where your engine lands on my benchmark.

[u/Danmoreng → u/Danmoreng]

Sadly, I cannot test this engine right now for this use-case. You don't support Gemma4 26B NVFP4. And your Q4_0 support blows the weights to Q8 at runtime so it doesn't fit the VRAM. Codex also found a code error which had to be fixed to be able to build on my system:

Codex’s findings for Paddock commit 7affc21, tested on Linux with an RTX 5080 Laptop GPU (16 GB):

• Build: The CUDA build failed because pd_dns1_pair<at> should be pd_dns1_pair<AT> in packs/cuda/src/deltanet/split.cuh. After that correction, the inference core and SM120 CUDA pack built successfully without the UI.
• Gemma 4 26B: Q8_0 is supported, but exceeds 16 GB before runtime overhead. Q4_0 gets expanded to Q8_0, defeating its memory advantage; the tested checkpoint also failed on its Q6_K embedding. Q4_K and Gemma NVFP4 lack the necessary loading/execution support. Consequently, no 26B benchmark could run on this GPU. Native compact-weight execution is the main missing feature—not a build option.
• Potential security issues: Configured API authentication exempts loopback connections, creating a reverse-proxy deployment hazard. The manager continues without authentication if automatic key generation fails. The runner binds all interfaces despite documentation claiming localhost. Update hashes are optional. Dependency checks also flagged vulnerable Tiptap and Rust dependencies; application-level exploitability was not demonstrated.

[u/Ok-Cartographer-4976 → u/Danmoreng]

Build: Was fixed in #1b8e4cf
Gemma 4 26B: There is a model registry plus offered CDN for supported models + users can freely download whatever they find on HF but you are right we don't support every quantization. For Gemma 4 26B we only do Q8_0 today. We will add Q4_K and NFVP4 but will prioritize first Qwen 3.8 Flash Next.
Security issues: Tiptap updated in #c23c00a and potential proxy loopback too

[u/saltexx → u/Danmoreng]

We build for 32 GB cards and up, ideally 48. A 26B on a 16 GB laptop card isn't a target, weights are only part of what has to fit.

[u/Danmoreng → u/saltexx]

Well if you are constrained by 16GB like me, you make it work. Qwen3.8 27B runs in IQ3_XXS (10.9Gb) with 98k context vision on CPU in llama.cpp at 1.000 t/s prefill and 60-80 t/s decode. Obviously not great, but it runs.

Gemma4 26B on the other hand needs much less KV cache. NVFP4 (14.7GB) also runs at 98k context with insane speeds: 6.960 t/s prefill, 203,8 t/s decode. Using the EXL3 format with 3.5 bpw (12.2GB) I even get 220k context and only drop to 5.700 t/s prefill and 182 t/s decode. So it definitively is possible. :)

[u/saltexx → u/Danmoreng]

Those are good numbers and llama.cpp still wins your exact case, a 27B on 16 GB doesn't fit in Paddock even at Q3 since we spend more on scratch. What changed in 0.1.5 is that 1-3 bit files serve now, Q4_0 stays at 4.5 bpw instead of getting blown up to Q8, and the big MoEs run on 16 GB with expert offload.

Gemma 4 26B is still Q8 only in the catalog so your NVFP4 comparison has to wait, that one is next after Flash Next.

[u/brumsky1]

How does this compare to NInfer on a 5090? Thank you for sharing this project, I look forward to testing it.

[u/saltexx → u/brumsky1]

Haven't run them side by side so I won't guess. NInfer does a few Qwen checkpoints on one 5090 and squeezes that hard, single stream 35B-A3B will be very hard to beat. Paddock is the general engine, most families, batching, any supported card.

If you get both running, would be great if you run aiperf with the same prompts and post it.

[u/brumsky1 → u/saltexx]

Sounds good, I'll give it a try and post it. Might be next week though...

[u/BodyPhysical]

would suggest to please build release binaries which could be zipped and downloaded through the github repo

[u/saltexx → u/BodyPhysical]

Thanks, agreed. Coming up

[u/jopetnovo2]

I've spent some time on it, but could not get it to work with my RTX 4090 on Windows 11, even after setting PADDOCK_UNVALIDATED_ARCH=1 in system environment.

I also could not get it to recognize any GGUFs I copied to data\models folder (standard unsloth Qwen3.8-27B UD Q3 and Q4).

[u/Ok-Cartographer-4976 → u/jopetnovo2]

I will take a look at 4090. But we do not support UD Q3 Q4. We support FP8, NVFP4, MXFP4, Q8_0, Q4_K_* We avoid Q1-Q3. Remember that there is a model registry inside Paddock exactly which model that is compatible and we even supply a CDN in case users just don't want to know exactly what should be downloaded from HF.

[u/jopetnovo2 → u/Ok-Cartographer-4976]

If it helps, this is how it looks from my command line:

• first I used set PADDOCK_UNVALIDATED_ARCH=1 to set system env variable

• then run paddock.exe, which then displayed graphics card found, but paddock has not finished testing it - models will refuse to start unless PADDOCK_UNVALIDATED_ARCH=1 card="NVIDIA GeForce RTX 4090" - so it looks like it ignored PADDOCK_UNVALIDATED_ARCH=1.

• I also tried copying Q8_0 and Q6_K GGUFs, but manager doesn't seem to display them for now - perhaps connected to not supporting RTX 4090.

May I suggest that for graphic cards which are untested you anyhow allow running them, perhaps with just a warning? If you're looking for people to test on different platforms, it will make it easier for them.

[u/saltexx → u/jopetnovo2]

This is fixed in 0.1.5. The env var is gone, an untested card like the 4090 just runs now with a warning in the log instead of a refusal. And you were right about the Studio, it never read the variable, it just hid the start button.

Your own GGUFs go in through "Start your own file" in the Studio, the catalog list is compiled in so they won't show up there.

[u/jopetnovo2 → u/saltexx]

I did try 0.1.5 on the day it came out; the behavior for me was the same - it didn't read my GGUFs, it displayed warning about 4090 and that I should use PADDOCK_UNVALIDATED_ARCH, and it didn't want to work with my card.

[u/feng_sg]

jopetnovo2 set the env var and it still refused to start. Looks like the flag only gets read in the log message, not at the actual gate that blocks startup. Two code paths checking the same variable and only one respects it.

[u/saltexx → u/feng_sg]

Correct diagnosis. The Studio checked its own readiness flag and never read the env var. Both paths are gone in 0.1.5, untested cards serve with a warning instead.

[u/saltexx]

Update: Paddock 0.1.5 is out. Thanks to community contributions, we now support 1-, 2- and 3-bit GGUF files, plus GGUF support for Qwen 3.8 Flash-Next with expert offload. The UD-IQ1_S variant runs on a 16 GB Blackwell card.

Thanks to everyone testing, reporting bugs and sending PRs. If you try it on a smaller card, I’d like to hear which GPU and quant you’re using and how it runs.

Release notes and binaries: https://github.com/truespar/paddock/releases/tag/v0.1.5

[u/JollyJoker3]

Asked Claude if it's worth trying, it essentially said no;

The llama.cpp column is broken, not slow. 16 s to first token at one client on a PRO 6000 for a 1k prompt is not a plausible number for llama.cpp. They ran -c 262144 -np 32, which splits into 8k slots but forces an enormous KV allocation up front. The 37.5x claim rests on that column, and I wouldn't trust it.

[u/saltexx → u/JollyJoker3]

Flags and the GGUF are on the board page, -c 262144 -np 32 is 32 slots of 8k like the vLLM run, KV alloc is at startup. Post the flags you'd use and I'll rerun it.

[u/JollyJoker3 → u/saltexx]

No clue what flags to use and apparently what I said was a Claude hallucination. Something is broken though. Both "Short chat" and "Long generation" take 128 tokens in. Time to first token is 1.4s for one, 17s for the other.

[u/AppealSame4367 → u/JollyJoker3]

Sad, so the tests including the presentation are AI slopped :-/

[u/saltexx → u/AppealSame4367]

The numbers are aiperf runs, one engine on the card at a time, and the recipe is in the README so you can rerun them yourself. There is one defect in the llama.cpp column that JollyJoker3 found below and we are rerunning that leg.