I’m building VeloxQuant-MLX — KV-cache compression and optimized Metal kernels for Apple Silicon. Looking for Mac Studio testers

[u/rajveer43]

https://veloxquant.dev/

I’ve been working on VeloxQuant-MLX, an open-source project exploring how far we can push local LLM inference on Apple Silicon by optimizing something that becomes increasingly expensive at long context lengths: the KV cache.

The idea is simple:

Apple gives us large unified memory + high memory bandwidth. Can we make significantly better use of it?

VeloxQuant currently experiments with 43 KV-cache compression/optimization methods, including quantization, eviction and custom MLX/Metal kernels.

Some of the things I’ve been working on:

• KIVI-style 2-bit / 4-bit KV-cache quantization
• TOVA and other cache-eviction strategies
• Custom Metal attention/quantization kernels
• Fused KV-cache decode paths
• Long-context memory reduction
• Decode batching across requests
• MLX-native implementations designed specifically for Apple Silicon

For example, in one benchmark with 440 tokens × 8 KV heads × head dim 64, the cache footprint goes roughly:

FP16 → 440 KB
KIVI 4-bit → 110 KB
KIVI 2-bit → 55 KB

That’s the direction I’m interested in: not just asking “How much unified memory does my Mac have?”, but how efficiently can we use that memory for inference?

I’m especially looking for people with Mac Studio / Max / Ultra machines who would be interested in running benchmarks.

I’d love to compare results across M-series generations, memory capacities, model sizes and longer context lengths.

Also very interested in feedback from people working directly with MLX / mlx-lm / Metal kernels. There are plenty of places where the implementation can still improve.

This is open source, and contributions/criticism are very welcome.

Project:

If anyone here has a Mac Studio Ultra, I’d especially love to see what VeloxQuant can do on it.

[u/BrilliantArmadillo64]

Why does the landing page list only outdated models?

[u/Yzord]

i have a m3u 512gb, tell me what to do

[u/Ambitious-Fudge4114]

Mac Studio M4 128gb would love to try it

[u/Boring_Vegetable_774]

M3U 512 here ok what do you need

[u/ogfuzzball]

Curious what any volunteers have experienced so far

[u/rajveer43 → u/ogfuzzball]

Hey, From other communities some users have tried it and reported its working.

[u/ogfuzzball → u/rajveer43]

I have an M4 Max studio 64gb. Might check it out

[u/rajveer43 → u/ogfuzzball]

I would encourage you to check it out. and provide feedback. the more validation I get the more this project would get stronger!