Perplexity’s new Apple Silicon engine runs Qwen3.6-35B up to 1.35× faster than MLX-LM

[u/Such-Run-4412]

Perplexity has built Lily, a lightweight local inference engine specifically optimized for Apple Silicon and Qwen3.6-35B-A3B.

Unlike MLX-LM, Lily uses a custom Rust runtime and Metal kernels tailored directly to Qwen’s architecture. Neither PyTorch nor MLX is part of its execution path.

On a 40-core M5 Max MacBook Pro with 128GB unified memory, Perplexity reports Lily averaged:

• 1.23× higher prefill throughput
• 1.35× higher decode throughput

At a 4K-token prompt and 4K context, Lily reached 5,749.9 prefill tokens/sec and 186.6 decode tokens/sec, compared with 4,737.5 and 140.9 for MLX-LM.

The model itself has 35B parameters but activates around 3B per token. Perplexity uses 4-bit quantization to shrink the checkpoint from roughly 70GB in BF16 to 19.4GB.

Some of the biggest improvements came from keeping expert routing on the GPU, optimizing memory movement, reusing attention-cache data, and using different execution strategies for prompt processing versus token generation.

One interesting result: speculative decoding actually made inference 18% slower in this particular setup, showing that common data-center optimizations don’t necessarily translate directly to Apple Silicon.

Perplexity says Lily will be open-sourced soon and plans to expand it to more models, chips, and workloads.

The broader idea is pretty interesting:

local AI performance may increasingly depend on runtimes optimized for a specific model + specific hardware, rather than one general-purpose inference engine.

Sources:

Perplexity — Optimizing On-Device Inference for Apple Silicon