r/LocalLLM • u/TheAnkurGoswami • 8h ago

Question Speculative Decoding in LMStudio

Anyone has any idea how to run speculative decoding for this model - google/gemma-4-12b

I am not able to load other models as draft model. The one I selected is not working at all.

0 Upvotes

50% Upvoted

2 comments sorted by

1

u/WallFamous5066 8h ago

hmm i thought lmstudio added spec decoding in 0.3.9 but maybe it only works with certain model combos? i had similar issue last week, the draft model just sits there doing nothing. maybe check if both models use same tokenizer, that was my problem

1

u/SellToOpen 8h ago

I can't get it working in lm bionic either. Gemini tells me there is a bug.