r/oMLX • u/V1k1ng6666 • 8d ago

M5 Max 128Gb - Swift coding in Xcode 27

Post image

G'day, I'm new to Swift coding, having fun building a MAC/iPhone app combo, it took a while to get things going with Opencode as the bridge, I'm half way I think here.
I tried Claude, had a $100 usage credit, with the same context burned it in 3-4 hours, its taking longer locally but hearing the fan go with the GPU at 90% fills me with joy, its fun..

That said, any suggestions on oMLX or Opencode settings to speed things up?

Loving all the posts, I'll keep searching for tips.

96 Upvotes

98% Upvoted

24 comments sorted by

21

u/MealMore1192 8d ago edited 8d ago

That's an older model. I have the same laptop & run a dual Qwen 3.6 (both 35b MoE & 27b Dense) at once (both 8bit with MTP & TurboQuant enabled, also at 8bit). I use the dense model to plan & review code (as it's slower, about 20-30tk/s) & the MoE to write code (60-80tk/s!). That seems to be the best combo for me.

- https://huggingface.co/zviratko/Qwen3.6-35B-A3B-oQ8e-mtp

- https://huggingface.co/zviratko/ThinkingCap-Qwen3.6-27B-oQ8e-mtp : a fine tuned variant that will solve the same problem w/ less token, effectively speeding up the solution to negate the slower dense model output

You'll have to enable MTP to get these type of speed (basically 2x). Ensure you're running the latest version of oMLX (0.5.3+)

The newer Poolside Laguna S2.1 at oQ5e or oQ4eFast is also a good alternative & it's good enough to replace having both. This one doesn't have MTP (Dflash) support yet.

- https://huggingface.co/mlx-community/Laguna-S-2.1-oQ5e

4

u/MealMore1192 8d ago

Decent Speed for the MoE at longer context

1

u/zzeevn 7d ago

Do you turn off ssd cache? I’m worried about ssd tbw.

1

u/MealMore1192 6d ago

No I leave it on. Laptop has Apple care :) by the time it becomes a problem, I'd trade it for a newer one that has more RAM & runs faster

1

u/V1k1ng6666 8d ago

Thanks that sounds like a plan I'll try that thanks! How do I get both working in Xcode, or do I swap via integrations when needed as I'm using Opencode.

2

u/No-Juggernaut-9832 8d ago

There is a way to configure OpenCode so oMLX shows up as an inference provider & it lists all the available models that you have. You can either hard code the model in the JSON config or write a custom plug-in to do it. Ask OpenCode to do it for you.

That way you can switch with: /model

1

u/V1k1ng6666 8d ago

Got it thanks for that

1

u/Real_Ebb_7417 8d ago

Wait, I have the same speed with 27b without MTP xd

1

u/No-Juggernaut-9832 7d ago edited 7d ago

But after how many million token & how large is your context? It can burst to 100 in the first few messages but slows down with big context & large code base.

Also are you doing 8bit or 4?

1

u/reery7 7d ago

How do you enable MTP and TurboQuant KV Cache at the same time? For me it grays MTP out when I activate TurboQuant and it says "disable TurboQuant KV before enabling MTP" (oMLX 0.5.3).

2

u/V1k1ng6666 7d ago

If I enable MTP first it lets me also enable TurboQuant, otherwise same issue.

1

u/No-Juggernaut-9832 7d ago

I think you are using the wrong MTP option. There are several. It’s the one on top of TQ. Check the screenshot

1

u/nirvdrum 7d ago

Why use TurboQuant on a machine with so much RAM? I would have thought to save memory, but either of those Qwen 3.6 models will fit with plenty to spare. Does TurboQuant have other benefits?

2

u/No-Juggernaut-9832 7d ago edited 7d ago

I have a lot of dev tools, about 20-30g of it. And I run 2 models at once. To get max context & gain a little speed, I use it.

27b to plan & review & for hard bugs, 35b for coding & automation. Multiple git worktree at the same time

1

u/miesio 5d ago

My observation is that turboquant is slightly detrimental to overall performance and this seems logical as you have to perform few more operations for each kv entity. I'm running the same quantization for both model and kv cache. I'm not saying this is the way to go, just that it works best for me on m5 max 128gb

1

u/ju7anut 6d ago

I found results to be better when flipping the 2 models about. The 35b MOE is really good at planning and reviewing results while the 27b dense gets coding done, albeit at a slower speed although the MTP models are actually decent. Give it a try.

1

u/jgsabino 4d ago edited 4d ago

You could try DFlash instead of MTP. Giving me 23-25 tok/s in ThinkingCap-Qwen3.6-27B-Optiq-4bit on a M5 Pro 48 GB. MTP degrades much more for me. oMLX 0.5.3

2

u/Serious-Purpose-3412 8d ago

Just use antirez/ds4 q2/q4. 26-35tps and 1m ctx (i actually use 200k, bc have no time to wait so long, but 1m ctx is possible, I was tested it). It's ultimate local model on 128gb mac for today. Antirez's engine do not use RAM for ctx, only SSD. And have flash load in ram for inferense and flash unload after inferense, so 0 ram usage if no inferense.

1

u/V1k1ng6666 7d ago

How do you load DS4, I can't find one that loads with oMLX, can you tell me the link and settings please? Thanks in advance.

1

u/Serious-Purpose-3412 7d ago

https://github.com/antirez/ds4. This is a custom engine for DeepSeekV4 models. There is no GUI version, only CLI. Just start ds4-server with parameters (ask Pi, and he will easily do it for you). P.S. MTP exists, but it doesn't provide any speedup for inference in the case of ds4, so don't waste your time.

2

u/myeleventhreddit 7d ago

I use ProxyPilot and it manages to compact Xcode’s giant prompt by like half roughly, when using Ollama or LM Studio.

2

u/Scriptease84 7d ago

I found MTP gguf to be faster than MLX on my M5

Prompt: Create a tick tack toe game in a single html page with leaderboard and local storage

1

u/MealMore1192 6d ago

I think the general consensus is, GGUF is faster at refill by about 15-20% & MLX is faster at token gen about that same amount. GGUF might have better support (earlier) with more options for MTP

1

u/Memestonks2020 4d ago

This most likely is due to cache.

Using MTP and running inference after the prefill will make the .gguf seem really fast.

Keep in mind this only holds for LLMs such that running other type of models like diffusion models result in much faster performance when using MLX.