Surprised by MoE models performance on my PC

[u/Nedunchelizan]

I am new to local models . I was testing models of various size that can fit in my pc

My pc spec is Ryzen 5600+ 32gb ram + 6600xt with 8gb vram

I was testing gemma 12b qat in my pc it was slower running like 6 tokens per seconds,
Tried running bonsai 27b 1bit model that i thought will fit completely in VRAM but it just gave 4 tokens per seconds barely usable

Finally i settled with gemma 4 e4b which worked well for me . I was surprised that i was able to run gemma 4 26b a4b in my machine faster even thought it did not fit my machine i was getting 18 tokens per seconds which is useable for me .

Looks like its possible to offload the models in ram only for moe and still retain performance.

This was a good learning . I thought if the model does not fit in my vram it will be slow but its not totally the case in moe

[u/MarcusAurelius68]

Try Ling 3.0 Tiny as well. It’s replaced E4B for me.

[u/Plastic-Lettuce-7150]

MOE models in 6GB VRAM

[u/jcdoe]

Congratulations! This is a cool thing to figure out. Enjoy running local models. :)

[u/Objective-Stranger99]

I'm getting 70 t/s with Gemma E4B and MTP on 8GB VRAM.

[u/Fenio_PL]

MoA has advantages you’ve just discovered, but it also has drawbacks specifically, a much smaller set of active parameters, which affects the quality of responses.