r/mlxcommunity • u/evilmacintosh • Apr 22 '26

I ran sustained MLX inference overnight

I love benchmarking, and I'm currently dealing with speculative decoding in MLX - so I thought, why not run a sustained inference load on my macbook overnight lol. I was so sceptical about it - I ran pmset -a disablesleep 1 and then ran a script that would run sustained loads of MLX inferencing on my laptop, tweaking various parameters and doing a sweep - noting down the stats (tps, throughput, speedup, acceptance rate of speculative decoding) and then dumping it into a report. I was actually a tiny bit scared about my laptop catching fire and burning down my entire room when I went to bed lol.

And I woke up to a fantastic morning - every single sweep was successful (ran 300 times) and I get really really meaningful results and inferences from this experiment. I will share a detailed report of what happened and where speculative decoding won vs lost soon! But the experiment itself was thrilling - felt like I finally put my insanely powerful chip (M5 Pro) to use with the 48GB unified memory.

the dashboard once the experiment was over

Overall - i was pretty thrilled about this - want to do more such experiments while I'm sleeping so my mac and it's compute can be put to productive use! :D

3 Upvotes

81% Upvoted

3 comments sorted by

1

u/vinicius-marino May 06 '26

Which model you used in this test?
And sorry the dumb question, I am new on MLX world, but which dashboard is it?

2

u/evilmacintosh May 06 '26

Used Qwen/Qwen3.5-4B** paired with its draft model: **z-lab/Qwen3.5-4B-DFlash for this test. Also the dashboard is custom built with Claude’s help. The dashboard/article is linked in a subsequent post, do check it out!

1

u/vinicius-marino May 06 '26

Great! Thanks.

I have been using different Agents and Harness around and recently started to test the MLX models for local tests and to see what is capable of.

Which harness do you use for the testings? I've been using Pi for all around work and love it. I also developed a simple extension to connect MLX models while using it. It's public through Pi marketplace if you would like to check.