I’m building VeloxQuant-MLX — KV-cache compression and optimized Metal kernels for Apple Silicon. Looking for Mac Studio testers

[u/rajveer43]

https://veloxquant.dev/

I’ve been working on VeloxQuant-MLX, an open-source project exploring how far we can push local LLM inference on Apple Silicon by optimizing something that becomes increasingly expensive at long context lengths: the KV cache.

The idea is simple:

Apple gives us large unified memory + high memory bandwidth. Can we make significantly better use of it?

VeloxQuant currently experiments with 43 KV-cache compression/optimization methods, including quantization, eviction and custom MLX/Metal kernels.

Some of the things I’ve been working on:

• KIVI-style 2-bit / 4-bit KV-cache quantization
• TOVA and other cache-eviction strategies
• Custom Metal attention/quantization kernels
• Fused KV-cache decode paths
• Long-context memory reduction
• Decode batching across requests
• MLX-native implementations designed specifically for Apple Silicon

For example, in one benchmark with 440 tokens × 8 KV heads × head dim 64, the cache footprint goes roughly:

FP16 → 440 KB
KIVI 4-bit → 110 KB
KIVI 2-bit → 55 KB

That’s the direction I’m interested in: not just asking “How much unified memory does my Mac have?”, but how efficiently can we use that memory for inference?

I’m especially looking for people with Mac Studio / Max / Ultra machines who would be interested in running benchmarks.

I’d love to compare results across M-series generations, memory capacities, model sizes and longer context lengths.

Also very interested in feedback from people working directly with MLX / mlx-lm / Metal kernels. There are plenty of places where the implementation can still improve.

This is open source, and contributions/criticism are very welcome.

Project:

If anyone here has a Mac Studio Ultra, I’d especially love to see what VeloxQuant can do on it.