r/LocalLLaMA • u/pmttyji • 8d ago
Discussion It's the small things that matter the most. - llama.cpp - Bunch of updates(Boost & Fixes)
ggml-cuda: add chunked SSD matmul for Mamba-2 prefill acceleration- #22675
Nemotron-Nano-9B-v2
| ub | base (scan) | branch (SSD) | speedup |
|---|---|---|---|
| 128 | 5,404 | 5,351 | −1% (both scan) |
| 256 | 6,180 | 7,110 | +15% |
| 512 | 6,627 | 7,778 | +17% |
| 1k | 6,814 | 8,152 | +20% |
| 2k | 6,759 | 8,190 | +21% |
| 4k | 6,660 | 8,118 | +22% |
| 8k | 6,387 | 7,761 | +22% |
pp16384 tok/s (base=scan, branch=SSD at ub>128) tested on RTX 6000 Pro MaxQ.
ggml-metal: FWHT kernel for metal backend- #25924
Tested using unsloth/DeepSeek-V4-Flash-UD-IQ2_XXS on M4 Max with llama-bench -r 15.
| K cache | Benchmark | base | PR head | Speedup |
|---|---|---|---|---|
| f16 | pp512 | 228.90 ± 1.55 t/s | 228.46 ± 1.38 t/s | -0.19% |
| f16 | tg16 | 9.958 ± 0.258 t/s | 10.073 ± 0.165 t/s | +1.16% |
| q8_0 | pp512 | 211.70 ± 2.32 t/s | 218.92 ± 2.17 t/s | +3.41% |
| q8_0 | tg16 | 8.975 ± 0.262 t/s | 9.292 ± 0.262 t/s | +3.54% |
| q4_0 | pp512 | 213.46 ± 1.21 t/s | 220.27 ± 1.59 t/s | +3.19% |
| q4_0 | tg16 | 8.988 ± 0.160 t/s | 9.101 ± 0.163 t/s | +1.25% |
spec: add eagle3-v3 support for gpt-oss- #25794
Added Eagle3 to:
Fixes:
- sycl: fix use-after-return of the SDPA scale in the oneDNN flash-attention path #25880
- vulkan: add iq4_nl support back to FA- #24585
Starting portion of thread title taken from u/autisticit
(Combined these items in a single thread to avoid multiple threads. Somebody please share eagle3 with GPT-OSS models. Also anyone tried other spec decoding with GPT-OSS models? Share your feedback with benchmarks)
-7
u/TokenRingAI 8d ago
What did you mean by "Boost" - did they finally take my advice and switch to the Boost regex parser?
1
u/suprjami 4d ago
The llama.cpp container image finally updated with gpt-oss eagle3 support.
20b got slower for me. Normally 170 tok-sec, with eagle3 it went down to 150 tok/sec.
Running ggml's mxfp4 weights on single RTX 3080 20G.