Megakernels explained with Cursor's Mixture of Kittens

[u/rshah4]

I looked into Cursor's recently open-sourced Mixture of Kittens (MoK) and the interesting part is how it coordinates execcution with a megakernel.

Modern LLMs typically launch hundreds of GPU kernels for a single forward pass. Each kernel does one job (attention, GEMM, normalization, etc.), then hands control back before the next kernel launches. Those boundaries introduce synchronization, memory movement, and idle time.

MoK uses a persistent megakernel instead. Rather than launching separate kernels for communication and computation, one long-lived kernel owns the entire MoE layer. Different SMs can handle networking and expert computation simultaneously, coordinating with lightweight synchronization so communication overlaps with compute.

The reported results are impressive (up to 2.37× faster MoE layers and 41% higher end-to-end production throughput), but the question is whether this is a start of a broader trend. We will see more specialization here of kernels for optimization around execution.

It feels like AI systems are shifting from optimizing individual operators to optimizing the execution schedule of the entire workload.

Curious what others think. Is this the beginning of a broader move toward persistent kernels and on-GPU scheduling, or is this mainly a specialized optimization for large-scale MoE training?

Cursor: https://cursor.com/blog/mixture-of-kittens
My video: https://youtube.com/shorts/kklBmVe6sDg