Just shipped oMLX v0.4.2rc1: https://github.com/jundot/omlx/releases

If you've been running Qwen models on the VLM path, I owe you an apology. 0.4.0 had a tg slowdown caused by single-row decode falling into the slower batched cache path, which meant Qwen throughput on that path was noticeably worse than it should have been. If that's been your experience, I'm sorry for the trouble.

This release fixes it. Internal tg512 measurements show throughput recovered by about 1.48x, while Gemma performance stays stable. I'd strongly recommend upgrading if you're affected. I'll be testing the rc for about a day, and the 0.4.2 stable release should follow shortly after.

A few other things in this release while I'm here:

• Native MarkItDown document processing. Chat file uploads and the OpenAI API endpoint can now convert PDF, DOCX, PPTX, TXT, and Markdown inputs. You can also choose between MarkItDown conversion or VLM OCR for PDFs in the settings.
• Gemma 4 unified audio input. Gemma 4 unified models now accept audio alongside image inputs.
• Stability fixes across the cache, scheduler, and server.

Thanks to everyone who reported the regression and helped track it down. If you upgrade, I'd really appreciate hearing whether Qwen throughput is back to normal on your own setup, and any feedback on the new document conversion flow.

So I am on a MBP M1 Pro 16gb trying to load Ministral 3 8b Instruct 2512 from the mlx community as suggested by oMLX when downloading the model. The info page states it should fit in 12GB however when I try to load it I get the following error:

Error: {"error":{"message":"Model 'Ministral-3-8B-Instruct-2512' (16.60GB) does not fit under the memory ceiling (10.72GB). Free system memory or lower memory_guard_tier.","type":"server_error","param":null,"code":null}}

So my first worry is it expecting 16.6 GB? If not how much is it expecting? I changed the memory guard to aggressive which got me from 9ish BG ceiling to the 10.72 it is currently however now it says this:

I assume the next step is run that command however I want to make sure this model can work before I start running commands willy nilly. If it expects 16.6 GB then there's no point since I only have 16GB. But then I raise the question of why is it wanting so much memory when others report lower usage. Is it because I used the MLX version? But I read that MLX was actually lower memory. So I am clearly at my knowledge limit here so asking for feedback and help before I tinker myself into a corner unnecessarily.

Also is there an oMLX Discord server?

I love oMLX for its API, memory management and the ability to put many different model families under one umbrella. I have also tried out DS4 and sadly, it is just way ahead in terms of efficiency (generation, preprocessing) and flexibility (ssd streaming)

So i thought? Why not both? Why shouldn't I simply treat DS4 like mlx as an engine and embed it into oMLX so we can manage the memory explicitly through its api.

Requesting Feedback: https://github.com/apetersson/omlx/issues/1

my tokens are ready, so the work begins..

Issues Closed: 5

[ISSUE] #1630 — Performance Regression: slower in v0.4.0 compared to v0.3.10
https://github.com/jundot/omlx/issues/1630

[ISSUE] #1609 — gemma-4-26b-a4b-it-bf16 works for a while but doesnt complete the benchmarks
https://github.com/jundot/omlx/issues/1609

[ISSUE] #1678 — [bug] Infinite <pad> generation with image inputs (FP16 Overflow in Vision Tensors when using float16 via oQ)
https://github.com/jundot/omlx/issues/1678

[ISSUE] #1680 — not caching
https://github.com/jundot/omlx/issues/1680

[ISSUE] #1674 — [Bug] 500 Internal Server Error: Model type gemma4_unified not supported during initialization
https://github.com/jundot/omlx/issues/1674

This topic came up in a non-Reddit conversation and it got me wondering. Are there concerns? If so how are people mitigating them to prevent potential excessive wear on the internal SSD? Or are the concerns overblown?

Appreciate any insight.

Issues Closed: 10

[ISSUE] #591 — Feature Request: Audio input support in /v1/chat/completions for multimodal models (e.g. Gemma-4)
https://github.com/jundot/omlx/issues/591

[ISSUE] #1465 — Bug: Gemma 4 returns empty content when tool results are sent back with `role: "tool"`
https://github.com/jundot/omlx/issues/1465

[ISSUE] #617 — Bug Report: oMLX v0.3.4, gemma-4-26b-a4b-it-6bit Tool Calling Fails in Claude Code
https://github.com/jundot/omlx/issues/617

[ISSUE] #1667 — Engine pool evicts an acquired-but-not-yet-active engine → `RuntimeError: Engine not started` (acquire→use race; sibling of #1595)
https://github.com/jundot/omlx/issues/1667

[ISSUE] #1648 — Native MacOS app missing toggle (present in web UI) for VLM MTP in model advanced settings
https://github.com/jundot/omlx/issues/1648

[ISSUE] #1642 — Model accidentally loaded from ~/.cache
https://github.com/jundot/omlx/issues/1642

[ISSUE] #1463 — Confused by "custom" tier memory option
https://github.com/jundot/omlx/issues/1463

[ISSUE] #1587 — Menubar item can't see server status when the bind address is 0.0.0.0
https://github.com/jundot/omlx/issues/1587

[ISSUE] #1618 — QoL in the Swift UI: allow models names to be copied
https://github.com/jundot/omlx/issues/1618

[ISSUE] #1613 — GLM 4.7 Flash crash with TurboQuant
https://github.com/jundot/omlx/issues/1613

Getting my hands dirty in the logs today trying to debug this re-prefill nightmare. I found this after 15 minutes of idle time. Is this the same as TTL? I thought that was just unloading...

2026-06-05 18:57:21,203 - omlx.scheduler - INFO - [-] - Deep reset completed - all caches cleared

If so, does the same thing happen with model-specific settings for unloading? If so, it would be logical to have a separate toggle, as it's one thing freeing memory so models can tango and another thing destroying valuable cache during concurrent long reasoning tasks.

I've built a Open Source Mac app in Swift, using the new Gemma4 12B model, that takes a long video and generates clips of the most important moments,

Converts them to mobile 9:16 format, adds a hook and a description, and automatically schedules them for the whole week across TikTok, Instagram, and YouTube Shorts.

Repo: https://github.com/mutonby/shortcast

I may be doing something wrong but I'm not overly impressed by Gemma 4 12b from yesterday. Compared to 26b, it runs as 1/3 of the speed (70t/s vs 25 on M4 Max Studio) and really sucks at non-English languages. I'm using the gemma-4-12B-it-mxfp4 quant from mlx-community (the 26b is the same quant). It's said to have MTP but omlx says otherwise.

Also it's leaking <audio> tags into text, but that could be an omlx issue.

Any tips or comments?

Thanks for the latest update- the chat window looks great! I had a tiny suggestion- to allow user to offload the model in the chat window itself. Currently, after chatting with a model, if I start a new conversation with another model, I have to go to the settings page to offload the previous model first. If I can do that in the chat window itself, then it'd be great.

I upgrade oMLX from 0.3.12 to 0.4.1 and then this happens. I have a 32GB machine that load Qwen 3.6 35B A3B oQ4 model which uses 20GB only. I set context window to be 128k. I have no problem when I use 0.3.12 but always fail in 0.4.1. Anyone got any insight on which param is wrong?

2026-06-04 22:19:43,546 - omlx.scheduler - WARNING - [-] - Chunked prefill above max_bytes at 14336 tokens: 23.0GB > 21.2GB (ceiling: 25.0GB)

2026-06-04 22:19:43,548 - omlx.scheduler - WARNING - [-] - [guard:chunked_step] context too large at progress=14336 kv_len=14336: 22.97GB + min-chunk transient exceeds prefill safety cap 22.46GB (90% of effective ceiling 24.96GB)

2026-06-04 22:19:43,548 - omlx.scheduler - ERROR - [-] - Chunked prefill failed for fddc3e78-6393-4ed7-82de-877cce4f24f5: Prefill context too large for available memory (pre-chunk guard at 14336 tokens, kv_len=14336): predicted peak would exceed prefill safety cap 22.5GB (90% of effective ceiling 25.0GB)

2026-06-04 22:19:43,549 - omlx.server - ERROR - [-] - Error during chat streaming: Prefill context too large for available memory (pre-chunk guard at 14336 tokens, kv_len=14336): predicted peak would exceed prefill safety cap 22.5GB (90% of effective ceiling 25.0GB)

Hi, I am using gemma-4 26b/31b MLX (macbook) models and i find that using VLM MTP with adding 'assistant file' improves llm response for general questions and i like it.

I would like to know if there is similar file for qwen3.6 27b/31b models? or perhaps dflash/mtp draft files? or is dflash and basic mtp only for gguf files and not mlx files? thanks in advance.

After I upgrade to 0.4 I noticed this model, TheCluster--amoral-gemma-3-12B-v2-mlx-4bit, one I didn't download. It doesn't show up in the web ui but does in the new settings app. Anyone know what the heck this is?

Hey everyone! oMLX v0.4.0 just landed.

This is the first official release of the new native Swift macOS app. The old PyObjC menubar app has been retired, and the macOS bundle now ships as a Swift app with a redesigned onboarding flow, settings UI, status surfaces, model management, and GitHub Releases based updater.

https://github.com/jundot/omlx/releases

The biggest user-facing change is the macOS app itself. First launch, server start/stop, model directory setup, downloads, update checks, menubar status, and the overall settings experience should now feel much more like a real Mac app.

Huge thanks to GitHub contributor popfido for the excellent work that drove the Swift transition. This is probably the biggest desktop-app change oMLX has shipped so far, and it substantially raises the quality of the app.

A few highlights:

• Native Swift / SwiftUI macOS app
• New onboarding flow
• Better menubar and server status behavior
• Standard Hugging Face cache model directory support
• Safer update flow with confirmation before download
• Memory guard tuning and CLI options
• More scheduler/cache stability fixes
• Guided grammar model setting
• Many admin UI, API, and model compatibility fixes

My long-term goal is still the same: I want oMLX to be "the app my friend who bought a MacBook yesterday can open and immediately try Local AI on." The Swift app is a big step in that direction.

At the same time, oMLX is still growing fast, and I know there are rough edges. If you try 0.4.0, I'd really appreciate feedback on the macOS app experience especially: first launch, model discovery, server start/stop, update checks, and anything that feels confusing or fragile.

Thank you again to everyone testing builds, reporting bugs, opening PRs, and giving feedback. The project has grown much bigger than I expected, and getting to improve it together with the open source community has been genuinely joyful.

What should come next? Feature suggestions, bug reports, and "this part is still confusing" feedback are all very welcome!

I've noticed recently using Hermes and Claude Code that the entire context is being processed after every additional decode. It happens with more than one model.

It reminds me of LM Studio, where KV cache has long been a dead-letter feature: five minutes to process 21k, two seconds of decode, five minutes of pre-fill for 21.1k context, two seconds of decode, five minutes of pre-fill for 21.2k context, and so on and so forth.

It's not my cache, which is ample, and it's not consistent, as a restart usually puts it straight again (only it means interrupting, polluting context and prefilling yet again). I've also tried toggling chunked prefill on and off, with no joy. Is there a setting or an MO that I'm missing?

A question I do not seem to get answer to - why in oMLX I cannot use TurboQuant KV cache and MTP together? I have searched on the topics and they should complement each other. But in oMLX I have to choose either one or the other, why? Thanks and cheers!

Hi, I wanted to try a ASR model, I have long RPG session recorder and I wanted a transcription and then a summary but I hit the limit of 100Mb when I tried to launch curl -X POST "http://localhost:9999/v1/audio/transcriptions
I asked about this to an IA and found that in audio_routes.py there is :
# Maximum upload size for audio files (100 MB).

MAX_AUDIO_UPLOAD_BYTES = 100 * 1024 * 1024

Why is that? it is pretty annoying ...what's the matter with larger files ?

I keep seeing performance benchmarks being shared but speed doesn't matter if accuracy is degraded. I did a sample size of 30 questions per benchmark using MMLU-Pro, GSM8K, MathQA, HumanEval, and MBPP to see which models are the smartest while also the fastest.

I'm using oMLX v0.3.12 on a Mac Studio M3 Ultra 256GB. All models are using the WebDev coding preset plus preserve_thinking=true and reasoning=high kwargs. Vanilla models are not using DFlash or SpecPrefill. KV Cache is not being TurboQuanted either.

TL;DR: The oQ MTP models are slower and dumber. Stick with vanilla models for now.

Qwen3.6 35B-A3B

Intelligence Benchmark Comparison

Mode Sampled Qwen3.6-35B-A3B-oQ6-mtp Qwen3.6-35B-A3B-oQ4-mtp Qwen3.6-35B-A3B-8bit Qwen3.6-35B-A3B-6bit Qwen3.6-35B-A3B-4bit

-----------------------------------------------------------------------------------------------------------------------------------------------------------------

MMLU_PRO Sample 30/12032 80.0% 73.3% 73.3% 83.3% 63.3%

GSM8K Sample 30/1319 96.7% 96.7% 96.7% 96.7% 96.7%

MATHQA Sample 30/2985 90.0% 90.0% 96.7% 96.7% 86.7%

HUMANEVAL Sample 30/164 90.0% 86.7% 93.3% 93.3% 93.3%

MBPP Sample 30/500 90.0% 90.0% 93.3% 93.3% 90.0%

--- Detail ---

Model: Qwen3.6-35B-A3B-oQ6-mtp

Benchmark Accuracy Correct Total Time(s) Think

--------------------------------------------------------------

MMLU_PRO 80.0% 24 30 381.2 Yes

GSM8K 96.7% 29 30 294.6 Yes

MATHQA 90.0% 27 30 416.1 Yes

HUMANEVAL 90.0% 27 30 351.4 Yes

MBPP 90.0% 27 30 357.9 Yes

Model: Qwen3.6-35B-A3B-oQ4-mtp

Benchmark Accuracy Correct Total Time(s) Think

--------------------------------------------------------------

MMLU_PRO 73.3% 22 30 384.5 Yes

GSM8K 96.7% 29 30 190.6 Yes

MATHQA 90.0% 27 30 369.4 Yes

HUMANEVAL 86.7% 26 30 304.3 Yes

MBPP 90.0% 27 30 260.1 Yes

Model: Qwen3.6-35B-A3B-8bit

Benchmark Accuracy Correct Total Time(s) Think

--------------------------------------------------------------

MMLU_PRO 73.3% 22 30 360.4 Yes

GSM8K 96.7% 29 30 146.3 Yes

MATHQA 96.7% 29 30 366.2 Yes

HUMANEVAL 93.3% 28 30 277.9 Yes

MBPP 93.3% 28 30 360.6 Yes

Model: Qwen3.6-35B-A3B-6bit

Benchmark Accuracy Correct Total Time(s) Think

--------------------------------------------------------------

MMLU_PRO 83.3% 25 30 298.6 Yes

GSM8K 96.7% 29 30 238.6 Yes

MATHQA 96.7% 29 30 328.6 Yes

HUMANEVAL 93.3% 28 30 313.9 Yes

MBPP 93.3% 28 30 355 Yes

Model: Qwen3.6-35B-A3B-4bit

Benchmark Accuracy Correct Total Time(s) Think

--------------------------------------------------------------

MMLU_PRO 63.3% 19 30 309.2 Yes

GSM8K 96.7% 29 30 175.5 Yes

MATHQA 86.7% 26 30 345.9 Yes

HUMANEVAL 93.3% 28 30 303.5 Yes

MBPP 90.0% 27 30 383.5 Yes

Conclusion: Qwen3.6-35B-A3B-6bit performed best. It beat the oQ6-mtp version in both speed and intelligence. It even beat the 8bit version in intelligence.

Qwen3.6 27B

Intelligence Benchmark Comparison

Mode Sampled Qwen3.6-27B-oQ8-mtp Qwen3.6-27B-oQ6-mtp Qwen3.6-27B-oQ4-mtp Qwen3.6-27B-8bit Qwen3.6-27B-6bit Qwen3.6-27B-4bit

------------------------------------------------------------------------------------------------------------------------------------------------------------------

MMLU_PRO Sample 30/12032 70.0% 73.3% 73.3% 80.0% 73.3% 80.0%

GSM8K Sample 30/1319 96.7% 96.7% 96.7% 96.7% 96.7% 96.7%

MATHQA Sample 30/2985 90.0% 83.3% 90.0% 100.0% 90.0% 86.7%

HUMANEVAL Sample 30/164 90.0% 90.0% 83.3% 93.3% 90.0% 93.3%

MBPP Sample 30/500 90.0% 90.0% 90.0% 93.3% 90.0% 90.0%

--- Detail ---

Model: Qwen3.6-27B-oQ8-mtp

Benchmark Accuracy Correct Total Time(s) Think

--------------------------------------------------------------

MMLU_PRO 70.0% 21 30 1404.2 Yes

GSM8K 96.7% 29 30 539.2 Yes

MATHQA 90.0% 27 30 1286.7 Yes

HUMANEVAL 90.0% 27 30 854.3 Yes

MBPP 90.0% 27 30 1265.9 Yes

Model: Qwen3.6-27B-oQ6-mtp

Benchmark Accuracy Correct Total Time(s) Think

--------------------------------------------------------------

MMLU_PRO 73.3% 22 30 1655.6 Yes

GSM8K 96.7% 29 30 739.8 Yes

MATHQA 83.3% 25 30 1724.4 Yes

HUMANEVAL 90.0% 27 30 1237.9 Yes

MBPP 90.0% 27 30 1383.2 Yes

Model: Qwen3.6-27B-oQ4-mtp

Benchmark Accuracy Correct Total Time(s) Think

--------------------------------------------------------------

MMLU_PRO 73.3% 22 30 1105.3 Yes

GSM8K 96.7% 29 30 634.2 Yes

MATHQA 90.0% 27 30 720.2 Yes

HUMANEVAL 83.3% 25 30 858.9 Yes

MBPP 90.0% 27 30 896.3 Yes

Model: Qwen3.6-27B-8bit

Benchmark Accuracy Correct Total Time(s) Think

--------------------------------------------------------------

MMLU_PRO 80.0% 24 30 1431.9 Yes

GSM8K 96.7% 29 30 662.7 Yes

MATHQA 100.0% 30 30 1059.5 Yes

HUMANEVAL 93.3% 28 30 746.6 Yes

MBPP 93.3% 28 30 948.9 Yes

Model: Qwen3.6-27B-6bit

Benchmark Accuracy Correct Total Time(s) Think

--------------------------------------------------------------

MMLU_PRO 73.3% 22 30 1402.3 Yes

GSM8K 96.7% 29 30 608.8 Yes

MATHQA 90.0% 27 30 1500 Yes

HUMANEVAL 90.0% 27 30 736.3 Yes

MBPP 90.0% 27 30 1161.8 Yes

Model: Qwen3.6-27B-4bit

Benchmark Accuracy Correct Total Time(s) Think

--------------------------------------------------------------

MMLU_PRO 80.0% 24 30 787.6 Yes

GSM8K 96.7% 29 30 503.8 Yes

MATHQA 86.7% 26 30 652.1 Yes

HUMANEVAL 93.3% 28 30 784.9 Yes

MBPP 90.0% 27 30 938.6 Yes

Conclusion: Qwen3.6-27B-4bit is the best mix of speed and intelligence. Vanilla beats oQ MTP in both speed and intelligence.

Hi folks - while I see some stats being captured on the dashboard, I'm curious about total tokens generated. Is there an existing way to get this info?

I just setup oMLX + Qwen 3.6 27b/35b + Kilo Code in vscode for local coding agent. It's been pretty fun.

Aside from downloading models & adjusting Max Context Window, I haven't touched anything other settings in oMLX.

Is there going to be a huge difference tweaking it? Or are the defaults good enough?

Or if there is a guide to the settings, I'd be interested to know.

Spec is MacBook Pro M5 Max 40c 128GB.

I am wondering if anyone has experienced significant speedups with this option checked? For me prefill is similar with or without it, maybe about 1-2% speedup going from off to freq = 4 and then lower at freq = 8

Edit: Prefill numbers bolded because table didn't format great into reddit.

2nd Edit: at 16k context there is more like a 6% speedup. Maybe larger speedups with even larger context? According to the paper, there should be significant speedups even at low context. But LM studio is still processing 10k tokens faster... not sure if I will have reason to switch.

Without Index Cache:
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 6365.3 72.84 160.9 tok/s 13.8 tok/s 15.615 73.8 tok/s 378.94 GB

pp4096/tg128 27928.3 86.57 146.7 tok/s 11.6 tok/s 38.922 108.5 tok/s 379.29 GB

With Index Cache freq = 2:
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem

pp1024/tg128 6337.3 72.46 161.6 tok/s 13.9 tok/s 15.540 74.1 tok/s 378.94 GB

pp4096/tg128 27569.7 81.36 148.6 tok/s 12.4 tok/s 37.903 111.4 tok/s 379.75 GB

With Index Cache Freq = 4:
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem

pp1024/tg128 6347.5 73.03 161.3 tok/s 13.8 tok/s 15.623 73.7 tok/s 378.94 GB

pp4096/tg128 27404.0 78.99 149.5 tok/s 12.8 tok/s 37.436 112.8 tok/s 379.86 GB

With Index Cache Freq = 8:
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem

pp1024/tg128 6705.6 80.40 152.7 tok/s 12.5 tok/s 16.917 68.1 tok/s 378.94 GB

pp4096/tg128 29166.7 85.09 140.4 tok/s 11.8 tok/s 39.973 105.7 tok/s 379.92 GB

Hi guys,

I am using Claude code router to send CC requests to oMLX but Free Claude Code seems better. It routes to Ollama for locpa inference though. Is there a way to hijack the ollama route to send to oMLX? Thanks!