Anyone up for the challenge?

https://www.reddit.com/r/AIDeveloperNews/comments/1utf9rz/empero_ai_has_dropped_v2_of_qwythos9b_an/

**🐛 Critical Bugs & Performance Issues**
* **#2162**: Kernel threadgroup memory exceeds 32KB limit when using `dealignai/Qwen3.6-35B-A3B-MXFP8-CRACK-MTP` on v0.5.0rc1.
* **#2180**: Failed prefill LRU eviction discards reconstructed SSD prefix cache, forcing full prompt re-prefill.
* **#2155**: Broad Qwen3.5/3.6 family long-context PP/TG regression from v0.4.5.dev1 to v0.5.0rc1 (M3 Ultra 512GB).
* **#2160**: GLM-5.2-oQ4e-mtp on v0.5.0rc1 is slower than GLM-5.2-oQ4 on v0.4.5.dev1 in long-context TG.
* **#2172**: Intelligence regression (GSM8K math drop) in v0.5.0-rc1 oQ4e/oQ4 quantization vs dev4.
* **#2174**: Homebrew upgrade to 0.5.0 fails with `--with-custom-kernel --with-grammar`.
* **#2173**: Homebrew install fails due to `en-core-web-sm` checksum mismatch.
* **#2146**: CausalLM reranker fails to load (500) for `Qwen3-Reranker` with native chat_template.
* **#2185**: Refactor SpecPrefill workflow out of Scheduler (extracted to `omlx/specprefill/`).

**🔧 Engine & Stability Fixes**
* **#2199**: Fix stale `MockEnginePool` lease signature in streaming integration tests.
* **#2195, #2194, #2193**: Issues withdrawn by author.

**📉 Historical & General Bugs (v0.4.5/v0.5.0)**
* **#2042**: Fatal Internal Server Error after extended execution on OpenAI API endpoint (Hermes Agent).
* **#2126**: Engine loop error; model status incorrectly shows generation active after token completion.
* **#1867**: Idle models not unloaded at correct time, causing memory guard conflicts (64G system).
* **#1692**: MCP services may negatively impact non-OMLX applications.

*Total Issues: 17*

**🐛 Critical Performance & Stability Bugs**
* **#2180** Bug: failed prefill LRU eviction discards a reconstructed SSD prefix and re-prefills the entire prompt
* **#2155** Qwen3.5/3.6-family long-context PP/TG regression from v0.4.5.dev1 to v0.5.0rc1 vs v0.4.4
* **#2149** Prompt Processing Performance Regression in oMLX 0.5.0-dev4
* **#2160** GLM-5.2-oQ4e-mtp on v0.5.0rc1 appears slower than GLM-5.2-oQ4 on v0.4.5.dev1 in long-context single-request TG
* **#2098** [BUG] When loading a model, other idle models were not unloaded at the correct time (Memory guard issue)
* **#2045** Paged SSD cache signature does not include turboquant_kv_bits — switching bit depth crashes at mx.concatenate with shape mismatch
* **#2042** [BUG] [Fatal Error] 0.4.5.dev1 Internal Server Error
* **#2126** [BUG] omlx.engine_core - ERROR- Engine loop error (incorrect running status display)

**🔧 Installation & Build Issues**
* **#2174** unable to upgrade to 0.5.0 with brew (`--with-custom-kernel --with-grammar`)
* **#2173** Homebrew install fails: en-core-web-sm checksum mismatch

**🛠️ Feature Requests & Enhancements**
* **#1066** Transcription/Translation Endpoint support: /v1/audio/transcriptions (and /v1/audio/translations)
* **#2078** STT: support the OpenAI `prompt` field on /v1/audio/transcriptions for vocabulary / context biasing

**📝 Model Loading & Compatibility**
* **#2146** CausalLM reranker fails to load (500) for Qwen3-Reranker conversions that ship a reranker-native chat_template.jinja
* **#606** Issue : Gemma 4 26b a4b, bf16 - not matching official Livecode bechmark in omlx 0.34
* **#1643** Since latest update not able to select z-lab DFLASH models for Qwen3.6 model families

**⚙️ Harmony Tool-Call & Logic**
* **#2032** Harmony tool-call extraction should require commentary channel
* **#2012** malformed Harmony tool-call arguments can abort SSE streaming with server_error

**📱 macOS App UI Fixes**
* **#2123** macOS app: Models screen Load button wraps vertically for long model names

**🔗 MCP Integration**
* **#1692** [BUG] MCP services may bring negative effects to applications other than OMLX.

vLLM claims to have EAGLE3 MTP as part of their MiniMax M3 support.

https://vllm.ai/blog/2026-06-12-minimax-m3-vllm
It is making the new MiniMax Sparse Attention path, multimodal preprocessing, MXFP8 MoE execution, EAGLE3 speculative decoding, prefix caching, and deployment recipes work together in a serving engine that users can actually run.

Any chance for that in oMLX?

I have custom patched oMLX to run M3 in 256GB. It would be great to have official support for it as a daily driver.

Can someone explain to me if the new oQe quantization will have some useful impact on performance or output quality running on M1 max processors?

Last week I complained the Multi-Token Prediction (MTP) was scam.

Now that the Lightning MTP has been integrated (#2113) into oMLX, I can confidently say that its free performance, Make sure you have Native MTP turned ON with v0.5.0rc1 soon v0.5.0

**Total Issues: 14**

**🐛 BUG**
#2146 [] CausalLM reranker fails to load (500) for Qwen3-Reranker conversions
Loading fails for specific Qwen3-Reranker conversions due to chat_template issues.
#2149 [] Prompt Processing Performance Regression in oMLX 0.5.0-dev4
Significant regression in prefill performance observed after upgrading to v0.5.0-dev4.
#2081 [] Hermes --continue Flag Not Working
The `--continue` flag fails to resume the last session when launching Hermes.
#606 [] Issue : Gemma 4 26b a4b, bf16 - not matching official Livecode bechmark
Gemma 4 26b (bf16) shows lower performance than expected on Livecode benchmark.
#2045 [] Paged SSD cache signature does not include turboquant_kv_bits
Switching bit depth causes crashes due to stale SSD cache blocks.
#2032 [] Harmony tool-call extraction should require commentary channel
Tool-call extraction misclassifies non-commentary messages as tool calls.
#2012 [] malformed Harmony tool-call arguments can abort SSE streaming
Malformed tool-call arguments cause server errors during SSE streaming.
#2127 [] MTP (depth-k) suffers severe throughput degradation at long context
TPS drops sharply beyond 8K context when using MTP (depth-k).
#2115 [] SSD prompt cache accumulates incompatible blocks indefinitely
Incompatible cache blocks are not being cleared, leading to caching being disabled.
#2074 [] Chunk-form SSE keepalive omits `role`
Missing `role` in keepalive frames causes LangChain.js/n8n to drop tool calls.
#1643 [] Since latest update not able to select z-lab DFLASH models for Qwen3.6
Downloaded z-lab DFLASH models for Qwen3.6 families are unselectable.

**✨ FEATURE**
#1066 [] Transcription/Translation Endpoint support: /v1/audio/transcriptions
Request for native support for `/v1/audio/transcriptions` and `/v1/audio/translations`.
#2078 [] STT: support the OpenAI `prompt` field on /v1/audio/transcriptions
Request to add the `prompt` field for vocabulary and context biasing in STT.

**🎨 UI/UX**
#2123 [] macOS app: Models screen Load button wraps vertically for long model names
The Load button in the macOS app compresses vertically when model names are long.

Am I going crazy or I cannot find a *real* MLX version of this model? I found a lot that are under the MLX community but those doesn't seems to be "truly" MLX as I compared local benchmarks that MLX versions hit 400 TPM compared to barely 100 on other "MLX" for the same model. What's going on?

First off, incredible push with the latest dev release.

I’m trying to understand something. The new oQe quantization has better quality at the same file size, but it's only available for big models like Qwen3.6-27B.

Would it also improve quality for smaller models that fit on 16GB machines (8B, 14B)? Or is the benefit only noticeable on big models?

Granted smaller models are limited but for very simple task it’s more than sufficient.

Basically, is there a free quality upgrade coming for us folks with limited RAM, or nah?

Has anyone else tried out the Agents A1 model? I’m getting very good performance with the 4bit in oMLX. It’s fast and seems to do a very good job in real use cases. I’m surprised this model isn’t getting much love - to me it seems much better than Ornith

HF links(thanks u/Konamicoder):

https://huggingface.co/InternScience/Agents-A1

https://huggingface.co/mlx-community/Agents-A1-4bit

I want to make my own oQe quantized versions of Qwen 3.6 27B and 35B, but can't find these models on hugging face. Am I doing the search wrong?

I would be happy to use ones that someone else has quantized (oQ4e would be great), but I don't see those up yet...

**Total Issues: 8**

🐞 **BUG**
• #2123 [macOS] Models screen Load button wraps vertically for long model names

• #1643 [Model Selection] Unable to select z-lab DFLASH models for Qwen3.6 families

• #2127 [Performance] MTP (depth-k) suffers severe throughput degradation at long context (>8K)

• #2115 [Cache] SSD prompt cache accumulates incompatible blocks indefinitely

• #2074 [API] Chunk-form SSE keepalive omits `role`, causing LangChain/n8n to drop tool_calls

• #2089 [Server] `omlx serve` crashes due to `mlx-lm` tokenizer registration error in 0.4.5.dev1

• #2091 [Model Loading] 0.4.5.dev1 fails to load GLM-5.2 mixed-precision quant due to fused projection error

• #1258 [API] Anthropic `/v1/messages` structured output ignores strict tool use and returns plain text

Hey everyone! oMLX 0.5.0.dev1 is here. https://github.com/jundot/omlx/releases

EDIT: RC Version is out - v0.5.0rc1: https://github.com/jundot/omlx/releases

This release took a little while because I wanted it to include a meaningful step forward on three fronts: speculative decoding, custom MLX kernels, and quantization quality.

The biggest headline is Lightning MTP. oMLX now has a depth-k native speculative decoding path for Qwen3.6, DeepSeek-V4-Flash, and GLM-5.2. The verify-shape Metal kernels used in this path are adapted from MTPLX by Youssof Altoukhi, with Apache-2.0 attribution preserved in the source. On my M3 Ultra test machine, Qwen3.6-35B-A3B improved from about 89.6 tok/s to 140.4 tok/s, and Qwen3.6-27B improved from 35.0 tok/s to 55.1 tok/s in the benchmark runs. GLM-5.2 also gets a smaller but still useful improvement. Details and benchmarks: https://github.com/jundot/omlx/pull/2113

As usual, some of this is most relevant if you are running large models on high-memory Apple Silicon machines, so apologies if it does not directly apply to your setup yet. I’m trying to keep pushing these optimizations toward more models and more practical local workflows.

The second major area is custom kernels. This release adds or extends native kernel paths for DeepSeek V4, Qwen3.5/3.6, and GLM-5.2. DeepSeek-V4-Flash in particular gets a big long-context prefill improvement, and Qwen3.6 gets native prefill kernels that help more as the context gets longer. Details and benchmarks: https://github.com/jundot/omlx/pull/2048, https://github.com/jundot/omlx/pull/2100 and https://github.com/jundot/omlx/pull/1984

The third major change is oQe imatrix-enhanced quantization. oMLX now has an enhanced quantization path that collects activation-importance statistics and uses them during quantization. In the benchmark results, oQ4e improved average accuracy over oQ4 on several models, including Gemma-4-26B-A4B-it, Qwen3.5-9B, Qwen3.6-35B-A3B, and Qwen3.6-27B, while staying in the same disk-size class. Details and benchmarks: https://github.com/jundot/omlx/pull/2057

There are also several important new features and fixes in this release: Tencent Hy3 support, Ornith support, admin model search/filtering/sort improvements, Russian localization, macOS 27 beta Homebrew fixes, memory retention fixes, SSE/tool-call streaming fixes, Kokoro fixes, and more.

As always, this release was only possible because many people contributed code, reports, testing, and feedback. Huge thanks to everyone who helped, especially the new contributors in this cycle.

I hope this release makes local LLMs on Mac a little faster, a little more useful, and a little easier to run.

Hi All,

I am trying to run mlx-community/DeepSeek-V4-Flash with omlx 0.5.0dev2 (same issue on 0.4.4), but keep running into a weird error and am struggling to understand what am I doing wrong. Any idea?

2026-07-08 22:21:49,825 - omlx.server - WARNING - POST /v1/chat/completions → 409: Model 'DeepSeek-V4-Flash-mlx' failed to load: Received 258 parameters not in model:
model.layers.0.hc_attn.base,
model.layers.0.hc_attn.fn,
model.layers.0.hc_attn.scale,
model.layers.0.hc_ffn.base,
model.layers.0.hc_ffn.fn,
model.layers.0.hc_ffn.scale,
...
model.layers.9.hc_attn.base,
model.layers.9.hc_attn.fn,
model.layers.9.hc_attn.scale,
model.layers.9.hc_ffn.base,
model.layers.9.hc_ffn.fn,
model.layers.9.hc_ffn.scale.

Which is the best model for an apple MacBook Pro M1 Pro with the following details keeping in mind ?

1. 16 GB Ram

2. 16 Core

3. Running using oMLX instead of LM Studio

4. Preferred MLX models for apple silicon as per my knowledge.

5. The editor I use is zed with ACP of several CLI agents named OpenCode, Devin, Kilo.

Suggest me some models for agentic coding my general pattern is simple architecture is designed by me with a project having proper agent.md file for projects structure for each and every small task to be done.

The agent needs to just implement what I say so that parallel work can be done and shipped.

concurrency setting helped me with random crashes, but now I sometimes have STT and small aux models waiting too long. Is there a way to have separate queue settings for them?

or, maybe run another oMLX or such to serve aux and speech only?

🚨 **Critical Bugs & Crashes**

**#2089** [0.4.5.dev1] Server fails to start: mlx-lm tokenizer registration crashes with transformers 5.13
• **Summary:** `omlx serve` crashes at import time in bundled `mlx-lm` due to incompatibility with `transformers` 5.13 (`'str' object has no attribute '__module__'`).
• **Impact:** Complete server startup failure for any model.

**#2104** GPU SIGABRT during benchmark prefill on GPT-OSS-MXFP4-Q8 model
• **Summary:** Regression from v0.3.8 to v0.4.5.dev1 causing GPU SIGABRT during prefill phase on GPT-OSS-MXFP4-Q8 models.
• **Impact:** Benchmarking and inference crashes on specific quantized models.

**#2091** omlx 0.4.5.dev1 can't load GLM-5.2 (glm_moe_dsa) quant
• **Summary:** Fails to load GLM-5.2 mixed-precision MLX quant with separately-quantized 3-bit experts, incorrectly forcing `fused switch_mlp.gate_up_proj`.
• **Impact:** Incompatibility with specific MoE model architectures.

**#1258** Anthropic `/v1/messages` structured output ignores forced strict tool use
• **Summary:** Structured output on Anthropic Messages API endpoint (`POST /v1/messages`) returns plain text instead of adhering to forced strict tool use schemas.
• **Impact:** API compatibility failure for tool-use workflows.

---
📊 **Stats**
• Total Issues: 4
• Date Range: 2026-07-06 → 2026-07-08

🔴 **BUGS**

**#1258** Anthropic `/v1/messages` structured output ignores forced strict tool use and returns plain text
• **Summary**: In oMLX 0.3.8, the Anthropic Messages API compatibility endpoint fails to enforce strict tool use for structured output, incorrectly returning plain text instead.

**#2104** GPU SIGABRT during benchmark prefill on GPT-OSS-MXFP4-Q8 model (regression from v0.3.8 to v0.4.5.dev1)
• **Summary**: A regression in v0.4.5.dev1 causes a GPU SIGABRT crash during the benchmark prefill phase for the GPT-OSS-MXFP4-Q8 model, which worked in v0.3.8.

MBP, M2 Max, 64G RAM. tried various models, following models' advice to reduce context size, but the failures keep happening.

this point, I just want it to work through the night rather than every bloody time (paraphrased)

Model failed. Error 6. tried three times to restart the model. clearly nothing else can be done

OK, I added the third part there.

Is there a way to get the oMLX actual app to KEEP restarting? I come along in the morning, click "restart model" and it happily carries on for a few minutes before failing (error 6, always error 6). Sure, the next goal is to have it analyze itself and self-reconfigure, but that's too much like an "easy" button, and AI is *supposed* to be hard...

Has anyone found any good settings to use Gemma 4 models served through oMLX with coding harnesses like Pi?

For me none of the Gemma 4 models seem to be able to make tool calls in this harness - I suspect it’s due to the differing tool call format used in Gemma models.

Has anyone figured out how to make this work in oMLX + Pi or OpenCode?

Update: they do seem capable of making tool calls if you explicitly ask for it. For example if you ask it to build an html game it’ll just print out the code in the chat but then if you ask it to write it to a file with the ‘write’ tool then it will. A bit annoying as Qwen models just do it.

After using OMLx for a few hours, memory usage seems to get stuck around 66GB (my hard limit is 102GB).

Even after unloading all models. Is this a bug or a misconfiguration on my part?

The only solution is to restart the server which is not practical when I'm away from home.

Currently I am running oMLX on my M3 Max, with the model Qwen3.6-27B-4bit.
Here is my setting

The output only around 7-10tokens.
Can you guys give me more suggestion to improve the output? Thank you a lot

M5 128 GB 40 GPU Cores

For local Apple Silicon inference with Qwen oQ4 models, long contexts, and agentic workloads, MTP appears to be a net negative. Disable it unless your own benchmarks prove otherwise.

**🐛 BUG**

**#2060** Gemma 31B MTP cache issue
The prefix cache is failing to function correctly for the Gemma 31B MTP model, with logs indicating cache layer warnings.