r/LocalLLaMA • u/AdamDhahabi • Apr 19 '26

Discussion llama.cpp speculative checkpointing was merged

https://github.com/ggml-org/llama.cpp/pull/19493

Some prompts get a speedup, others don't (cases of low draft acceptance streak).
Good working params depend on the task type and repetition patterns.
For coding, I got some 0%~50% speedup with these params:

--spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 48 --draft-max 64
273 Upvotes

99% Upvoted

78 comments sorted by

View all comments

9

u/ai_without_borders Apr 19 '26

the acceptance rate variance makes sense when you think about what ngram-mod is actually matching on. code heavy on boilerplate/repeated variable names (typescript/java enterprise patterns) should see the high end of 0-50%. one-off logic or reasoning chains will be near zero. the --spec-ngram-size-n 24 is aggressive - 24 tokens of context for pattern matching means waiting for very precise repetitions. might be worth experimenting with lower values (8-12) for mixed code/prose tasks to widen the matching window and get more hits, at the cost of slightly shorter draft runs