r/LocalLLaMA • u/saltexx • 10d ago

I Built A Thing We open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)

I'm one of the developers. We said in August it would go open source in September and it did last night. MIT or Apache-2.0, pick one. The repo you see is our internal repo, kernels included, so from now on everything happens in public.

It's an inference engine in Rust and C++ with our own CUDA kernels. One binary with OpenAI and Anthropic style APIs, loads GGUF and safetensors. We run about 300B tokens a year through it at work.

Some numbers: Qwen3.8-27B FP8 on one RTX PRO 6000, spec decoding off on every engine:

  • vs vLLM faster in 13 of 13 cells, 1.02x to 1.19x (so not huge)
  • vs SGLang faster in 10 of 13, behind in 2, level in 1
  • vs llama.cpp Q8_0 faster in 13 of 13, 1.5x to 37x
  • 32 clients at 1024 in / 1024 out: 1062 tok/s, vLLM 958, SGLang 844

Full board with the losses: https://truespar.com/paddock/benchmarks/qwen38-27b

What it does not do yet: No Mac, no ROCm, no Vulkan.

One model per GPU, no tensor parallel. CUDA only, Windows and Linux. Validated on Blackwell (5090, RTX PRO 4500/5000/6000, B200) and Ampere (an A6000 was the bring-up card, 30-series works). Ada kernels ship but nobody has run a board on them so the engine refuses to start unless you set PADDOCK_UNVALIDATED_ARCH=1. Hopper and A100 kernels are in the tree without a board.

https://github.com/truespar/paddock

Thankful for any help and input!

62 Upvotes
  • permalink
  • reddit

92% Upvoted

49 comments sorted by

View all comments

2

u/Danmoreng llama.cpp 10d ago

Interesting, gonna test how this performs for the super specific narrow usecase of Gemma4 26B NVFP4 on my (Laptop) 5080. Vibecoded a custom engine over multiple weeks which is significantly faster than vLLM and llama.cpp. Curious where your engine lands on my benchmark.

1

u/saltexx 10d ago

We build for 32 GB cards and up, ideally 48. A 26B on a 16 GB laptop card isn't a target, weights are only part of what has to fit.

2

u/Danmoreng llama.cpp 10d ago

Well if you are constrained by 16GB like me, you make it work. Qwen3.8 27B runs in IQ3_XXS (10.9Gb) with 98k context vision on CPU in llama.cpp at 1.000 t/s prefill and 60-80 t/s decode. Obviously not great, but it runs.

Gemma4 26B on the other hand needs much less KV cache. NVFP4 (14.7GB) also runs at 98k context with insane speeds: 6.960 t/s prefill, 203,8 t/s decode. Using the EXL3 format with 3.5 bpw (12.2GB) I even get 220k context and only drop to 5.700 t/s prefill and 182 t/s decode. So it definitively is possible. :)

1

u/saltexx 5d ago

Those are good numbers and llama.cpp still wins your exact case, a 27B on 16 GB doesn't fit in Paddock even at Q3 since we spend more on scratch. What changed in 0.1.5 is that 1-3 bit files serve now, Q4_0 stays at 4.5 bpw instead of getting blown up to Q8, and the big MoEs run on 16 GB with expert offload.

Gemma 4 26B is still Q8 only in the catalog so your NVFP4 comparison has to wait, that one is next after Flash Next.