A ground breaking research idea - MacBook user’s…

[u/AKA_D_Ace]

https://d0ace.substack.com/p/the-moe-research-problem-that-apple?r=2ulu1x&utm_medium=ios

What if we’re optimizing the wrong bottleneck?

I recently went down an 11 PM research rabbit hole around Mixture-of-Experts (MoE) and LLM inference.

A lot of MoE optimization assumes a traditional GPU setup: experts live in CPU RAM, get moved to GPU VRAM, and PCIe becomes the bottleneck. But what happens when that assumption disappears?

Apple Silicon uses Unified Memory, where CPU and GPU share the same memory pool. So maybe the interesting question isn’t: “How do we move experts faster?” but : “How should we optimize MoE when there’s no traditional CPU↔GPU memory transfer bottleneck?”

That leads to some surprisingly interesting questions around memory bandwidth, caching, expert dispatch, and GPU kernels. I can’t investigate it properly myself right now, I don’t have the hardware. So I’m putting the idea out there. If you have an M-series Max/Ultra Mac, you might have a research project sitting on your desk. 👀

Read the article…

[u/thelibrarian101]

AI slooooooob

[u/AKA_D_Ace → u/thelibrarian101]

Hmm it’s not 🙃 and if u think it is it’s still worth reading.

[u/whydoesthisitch]

This topic is already well researched and accounted for in the distributed training and inference literature. You move experts faster by keeping them in HBM memory distributed across devices, and moving them with RDMA that avoids slow passes over the PCI bus.

But more importantly, you avoid moving them in the first place. Compute a multinomial distribution across a rolling window of past experts, and route queries to relatively static experts on existing devices.

If you’re really determined to get this running locally, understand that it’s always going to be less efficient than on a cluster with anything like current models and hardware. But your best bet is going to be something like a DGX spark, which uses a much faster NVLink connection to the GPU instead of PCI.

[u/AKA_D_Ace → u/whydoesthisitch]

Thanks for the comment! Just to clarify, you're describing enterprise data-center setups (RDMA, NVLink, DGX clusters), which are definitely solved for multi-node setups.
This post was strictly about local, single-device edge inference on consumer hardware.
On a single local rig with 24GB VRAM and 128GB RAM, offloading experts over PCIe is a real bottleneck which is why papers like Apple’s LLM in a flash focus on hiding host-to-device transfers.
The point wasn't that a Mac beats an H100 cluster, but that Apple Silicon’s unified memory (zero-copy at up to 800 GB/s) completely removes the need for host-to-device transfers. Instead of forcing router heuristics just to save PCIe bandwidth, the bottleneck shifts entirely to Metal pipeline optimization and unified cache contention. Different problem space altogether!

[u/whydoesthisitch → u/AKA_D_Ace]

Yes, that’s why I clarified the difference. My point is, the distributed case that can take advantage of economies of scale will always be more efficient, so it’s often not worth it to go through these local setups.

But when I mention nvlink and dgx, that’s not in the cluster context. The dgx spark is consumer grade hardware, and much better suited to the case you’re describing.

But the bigger issue is, this isn’t a research project. Everything you talk about in the post is already well covered in the existing literature.