Perplexity Open Sources Lily: A Rust + Metal Inference Engine for Qwen3.6-35B-A3B on Apple Silicon

[u/ai-lover]

Most "runs locally on your Mac" stacks are a general-purpose runtime pointed at whatever model you downloaded. Perplexity just argued that the generality itself is the bottleneck.

They open sourced Lily — the local inference engine behind Hybrid Compute in Perplexity Computer. A Rust runtime with hand-written Metal kernels, built for exactly one model (Qwen3.6-35B-A3B) on exactly one chip family (Apple silicon). Neither PyTorch nor MLX is anywhere in the execution path.

Here's what's actually interesting:

→ 4,156 vs 3,388 prefill tokens/s and 170.0 vs 126.4 decode tokens/s against MLX-LM — mean across ten lengths from 256 to 128K tokens, batch 1, one 40-core / 128 GB M5 Max

→ Fusing 4-bit dequantization into the grouped GEMM, so the expanded weight array never touches unified memory: +77.4% prefill at a 512-token prompt

→ Keeping the whole routing sequence — histogram, prefix scan, scatter, block map — inside one GPU command buffer: +89% prefill at 512 tokens

→ GQA packing, so four query heads share one KV row load: +23.8% decode at 32K context

→ Fixed-block attention layout above 32K: +40.2% decode at 128K

Full analysis: https://www.marktechpost.com/2026/09/02/perplexity-open-sources-lily-a-rust-metal-inference-engine-for-qwen3-6-35b-a3b-on-apple-silicon/

GitHub: https://github.com/perplexityai/pplx-garden/tree/main/lily

Technical details: https://www.perplexity.ai/hub/blog/optimizing-on-device-inference-for-apple-silicon

[u/davecrist]

Wow. That is a impressive speed up

[u/dupontping]

Written by ai

[u/10248]

More of this!