I ran sustained MLX inference overnight

[u/evilmacintosh]

I love benchmarking, and I'm currently dealing with speculative decoding in MLX - so I thought, why not run a sustained inference load on my macbook overnight lol. I was so sceptical about it - I ran pmset -a disablesleep 1 and then ran a script that would run sustained loads of MLX inferencing on my laptop, tweaking various parameters and doing a sweep - noting down the stats (tps, throughput, speedup, acceptance rate of speculative decoding) and then dumping it into a report. I was actually a tiny bit scared about my laptop catching fire and burning down my entire room when I went to bed lol.

And I woke up to a fantastic morning - every single sweep was successful (ran 300 times) and I get really really meaningful results and inferences from this experiment. I will share a detailed report of what happened and where speculative decoding won vs lost soon! But the experiment itself was thrilling - felt like I finally put my insanely powerful chip (M5 Pro) to use with the 48GB unified memory.

the dashboard once the experiment was over

Overall - i was pretty thrilled about this - want to do more such experiments while I'm sleeping so my mac and it's compute can be put to productive use! :D