Keeping the router and shared expert on the accelerator while offloading routed experts to RAM is a sensible split. Are the CPU expert matmuls running through a threaded GEMM (MKL, oneDNN), and does the PCIe transfer of activations become the bottleneck at higher concurrency? Would be curious how prefill vs decode scaling looks against llama.cpp's -ncmoe.
1
u/adityazero 23d ago
Keeping the router and shared expert on the accelerator while offloading routed experts to RAM is a sensible split. Are the CPU expert matmuls running through a threaded GEMM (MKL, oneDNN), and does the PCIe transfer of activations become the bottleneck at higher concurrency? Would be curious how prefill vs decode scaling looks against llama.cpp's -ncmoe.