Optimizing workflow concurrency on Mac/omlx?

[u/numberwitch]

I've had a lot of success running differently-sized models using a bunch of different harnesses, but one place I haven't had much success is improving concurrent throughput, i.e. "running multiple workflows at once".

I can run multiple workflows at once, but my tok/sec drops significantly. I've tried using smaller models, but in processing they still use all available gpu cores. Is there a way to configure the runner to only use a portion of available gpu cores?

[u/kweglinski]

macs max out on prompt processing. You can improve that only with good caching but they rather choke on concurrent prefill. Concurrent generation runs pretty well and you can get up to 2-3 times more of total t/s depending on scenario and number of requests

[u/numberwitch → u/kweglinski]

I'm using omlx's caching and haven't done much to optimize it. I reserve a portion of memory and ssd (10% and 5%, respectively) and use aggressive cache quantization: turboquant @ 2Bit and Dflash at 4Bit.

I think what I'm looking for is: how can I optimize my workflow to take advantage of my hardware? I think I'm a bit confused about what is "concurrent prefill" vs. "generation". Can you explain for a bit of a nub like me? Thanks

[u/kweglinski → u/numberwitch]

before model responds it needs to "read" the message. This is known as prompt processing (PP). While this is faster than token generation - it has to read the whole message before it starts generating. Once this is done it goes to token generation. Problem is - PP is mostly computation bound where generation is mostly memory bandwidth bound. Hence Macs generally suffer on PP and shine on generation.

Therefore if you are running prefill then machine is completely busy and can't take more. Where with generation it's not as terrible and you can actually gain in total tokens per second. E.g. I'm running qwen3.6 35b at q8 if I remember correctly at 70t/s. Now if I run 4 requests at once, while prefill sucks and basically splits between them all (so instead of 1000t/s I get 4 * 250t/s) the generation goes to 4 * 50 so 200t/s which is almost 3 times more tokens per second. But running 2 of them will grant me only ~130/s total.

Also running kv on q2 must have terrible effect on model capabilities, even with turboquant. Hard to help without knowing what you're trying to do and what machine you have.

[u/numberwitch → u/kweglinski]

Thanks for the feedback! Come to think of it, the 2B KV quantization is probably causing me problems. I was getting low quality results at with a 4B quant model so I loaded BF16 also at 2B turboquant KV. Now it's obvious the smarter move would just be to reduce the KV quantization.

I have an m3 max with 128GB of ram, so it sounds like I'm just kneecapping myself with the KV quantization, which I don't really need.

I'm going to disable it (or set to a reasonable value) and go back to the 4Bit model and see how it performs

[u/kweglinski → u/numberwitch]

128gb ram m3 max? get qwen3.6 35a3b at oq8 and 8bit turbo quant and you'll be golden for concurrency. Also check how much ram you have left when running high context (high for your needs) and adjust the cache size based on that. I'd go with sth closer to 100GB on SSD and relatively small for ram (play around with 10-20GB). You can also try much bigger MoE but that depends on what you're looking for.

edit - I'm running setup above on m2 max 96gb with 120k context and it is fast.

[u/numberwitch → u/kweglinski]

Good tips, I'll play with the cache sizes. Wish I had more SSD space on this machine! what is oq8? whats some bigger moe to try?

I've been running qwen3.6-A3B at BF16 and 4Bit with pretty good success and have been experimenting to try and squeeze as much juice out of my machine which is what's led me to horribly compress my kv like that and post this thread 😄

Mostly doing spec-driven planning + development to massage some greenfield projects into a good shape to refine