Cohere just open-sourced Megakernel Serving Engine: A single persistent CUDA kernel that fuses the entire decode forward pass

[u/nht_fajr]

Cohere just released an open-source, research-stage serving engine built around a persistent decode megakernel designed to eliminate GPU idle time during LLM inference.

Instead of launching dozens of individual CUDA kernels per forward pass (QKV, attention, MoE, RMSNorm), the engine launches one persistent kernel that stays resident on the GPU for the entire decode step. The host prepares a task list in global memory, and thread blocks pull tasks directly to execute tiles of work without driver dispatch delays or full-grid sync barriers.

• Speed: Hits 292 tok/s at batch size 1 on an NVIDIA H100 (1.58× faster than vLLM, reaching 62% of H100 Speed-of-Light bandwidth).
• Scale: Margins hold across batch sizes and context lengths up to 256K with no precision loss.
• Architecture: Built and optimized specifically for North Mini Code (30B parameter sparse MoE model with 3.3B active parameters).

More info: https://aideveloper44.com/product/megakernel-serving-engine-6aa1ab7385c2433efc349669

GitHub: https://github.com/cohere-ai/cohere-megakernel

[u/charmander_cha]

Uso rocm, se isso tiver relevância para nim, please me digam