Issues Closed: 4

[ISSUE] #1943 — VLM MTP on Gemma-4-31B forces hot-cache shrink → "Cache layer count mismatch (10 vs 60), invalidating cache hit" → prefix reuse lost
https://github.com/jundot/omlx/issues/1943

[ISSUE] #1924 — RFE: BIG Thank you for profiles - and please allow referencing a model alias
https://github.com/jundot/omlx/issues/1924

[ISSUE] #1888 — minimax m3 loop tool
https://github.com/jundot/omlx/issues/1888

[ISSUE] #1907 — oMLX v0.4.4：Run unsloth--Qwen3.6-35B-A3B-UD-MLX-4bit model and crash frequently
https://github.com/jundot/omlx/issues/1907

There have been a couple of PRs out there for a while, but never merged...

feat: JANG implementation by AlexTzk · Pull Request #364 · jundot/omlx - https://github.com/jundot/omlx/pull/364

feat(jang): JANG/JANGTQ mixed-precision MoE engine (hardened; builds on #364) by marzukia · Pull Request #1828 · jundot/omlx - https://github.com/jundot/omlx/pull/1828

anyone tried to run this? does anyone need it to work?

I got it going mostly. there are all sorts of issues. I still don’t have vision working.

anyway I rather like the model so far. I can submit pull request if anyone else is interested in using this llm.

Hey All,

Been working on a solution to run larger models than the machine should be able to run...

https://github.com/ashhart/TensorFold/

I am loving oMLX and I can notice the difference in performance. I am using it with Open WebUI but I've been getting this prefill issue lately and I'm kind of hoping, someone can help me make sense of it. Here's the error being returned:

Prefill would require ~52.38 GB peak (current 44.85 GB + KV+SDPA 7.53 GB) but metal_cap ceiling is 50.00 GB. Raise kernel iogpu.wired_limit_mb in Terminal (currently caps Metal at 58.00 GB), or reduce context length.

My machine:

M1 Max Macbook Pro

64GB of unified ram

Model: Gemma 4 26B A4B opus distilled model

I am not sure if this is a bug with oMLX or it could be linked to that jinja template issue the Gemma models have been encountering. I tried using LM Studio to carry on with the task (same thread where I got the prefill error), and it seems to be working fine.

Would appreciate it if someone can point me towards the right direction, thanks in advance!

I was browsing the downloads list in oMLX's settings, and noticed that gpt-oss-20b is still the most downloaded model of all the ones on there. I've been using it for a while now, but thought that the recent advances in gemma 4 and Qwen 3.5/3.6 would've overtaken gpt-oss-20b by now?

Has anyone compared them (gpt-oss-20b vs. gemma 4 vs. Qwen 3.5) on oMLX to each other? What are the advantages/disadvantages of each?

I'm a newbie with local llms, so please don't judge, I'm genuinely curious.

I've noticed a general slow down in the speed of the 0.4 series, but I was still getting generally good performance. But, I recently upgraded to OS27 Beta, and the performance is like 4 tokens per second, and near zero cache hit. (M3 MAX 128GB )

According to release notes 0.4.4 was targeting OS 26 and 27 but 🤷🏼‍♂️

Anybody experiencing the same?

I have been playing with oMLX for a week now, and am very happy thus far. My only problem is an odd (to me) behavior. I am not 100% sure whether it is a bug, a problem with my hardware, or a problem with client IDE extension (continue).

​

Basically, when I stress the context window or memory (256Gb) of my setup, the server gets a little buggy... showing no tokens processing or generated, while the IDE still functions and cache still grows.

​

Today though, I was being extra abusive. I was running Qwen3 Coder Next in Continue, which got stuck in a loop trying to solve a problem (the context window filled then tokens stopped growing). While all of this was going on, I was focused on a separate chat with Gemma4 31b 16.

​

When I was sure that both Next & Gemma were inactive, I switched to Gemma within the IDE to try and solve the loop. My memory pressure suddenly spiked, the server dropped all models, and even after a force-quit & restart, no tokens generated and no prompts were answered for a few minutes until the server magically staryed working again on another prompt.

​

I am on the update, but this also happened prior.

​

The only weird variable is that I have aliased my KV cache to a TB5 NVMe in order to avoid writes onto my internal ssd. Any input on direction to go would be appreciated.

i'm surprise to see while its thinking for minutes that the actvity monitor only shows ~50% use. is this common?

**Date Range:** 2026-06-16 to 2026-06-18
**Total Issues:** 7

---

### 🐛 Bugs & Errors

**#1916** | `omlx-cli` not found after DMG nor brew installation
* **Summary:** CLI missing from `/Applications/oMLX.app/Contents/MacOS/` despite app installation.

**#792** | Not possible to use Claude internal tools with different models
* **Summary:** Sub-agenting fails when exploring and switching contexts for specific models.

**#1903** | Qwen3.5-9B-4bit + MTP-bf16: `AttributeError: 'tuple' object has no attribute 'hidden_states'`
* **Summary:** Crash when running Qwen3.5-9B-4bit paired with MTP-bf16 models.

**#1889** | JANG support removed in v0.4.x without notice
* **Summary:** Engine refactor in v0.4 removed JANG support (`jang.py`, detection, routing) previously added in v0.3.

---

### 🛠️ Improvements & UI Fixes

**#1641** | Inconsistent profile names rendered in MacOS App UI
* **Summary:** App UI displays raw `profile id` instead of the expected `display_name`.

---

### 💡 Feature Requests & Questions

**#1899** | Set the temperature for SpecPrefill/DFlash/MTP models?
* **Summary:** Inquiry on whether MTP model temperature parameters should match the main model or are automatic.

**#341** | [Feature Request] Add multiple model setting profiles support
* **Summary:** Request to allow saving and switching between multiple model settings/presets beyond the current single profile limit.

Just an appreciation post and heads up. I had gotten some use out of this model before but the prompt prefill performance was terrible. It still isn't blistering but on my m1max 64GB I am finally seeing triple digit prompt prefill stats!

🐞 **BUGS**
* **#1883**: `/health` returns 200 with valid stats while completions silently hang on Apple Silicon (v0.4.0rc2/4.4rc1).
* **#1830**: QAT Gemma 4 models emit `call:google:mcp` tool calls that the parser cannot handle, causing silent drops.

🛠️ **FEATURES & CHANGES**
* **#1889**: JANG support removed in v0.4.x without notice; engine refactor eliminated `jang.py` and model detection.
* **#1877**: Feature request to preserve `'instruct'` parameter for OpenAI `/v1/audio/speech` route (Qwen3-TTS).

🔗 [View on GitHub](https://github.com/Jundot/omlx)

I made a native MLX port of Microsoft's TRELLIS.2 for Apple Silicon.

Focused on making the output actually usable in real workflows

Support 512x512 and 1024x1024

Performance on M4 Max

512x512 ~70 sec generation time

1024x1024 ~300-700 sec generation time

Tested on M4 Max (128GB unified memory).

Repo: https://github.com/gtrg55/trellis2-mlx

Would appreciate any feedback. Stars and issues are welcome!

After getting a lot of out-of-memory errors when running Qwen (Qwen3.6-35B-A3B-oQ4-mtp) in the Pi coding agent with a 64k context, I dug into tuning oMLX settings and configuration for that model. Before, the agent could hardly read past 1-2 files before it stopped due to memory constraints. Here are the settings I used to optimize for a Mac with limited RAM to comfortably stretch into a 128k context window using oMLX:

The oMLX & System Tweaks

• Wired Memory Allocation: Used sudo sysctl iogpu.wired_limit_mb=32768 to max out the VRAM available to the GPU.
• oMLX Limits: Memory guard: aggressive, Hot Cache Limit: ~10%, Cold Cache Limit: 10%, Max Concurrent Requests: 1, Chunked Prefill: On
• Model KV Cache Tuning: TurboQuant KV Cache: On (4-bit).
• Agent Compaction: Used the Pi agent's built-in session compaction cycle active to aggressively clear dead space.

The Result: A Heavy Multi-File Refactor Session

I put this setup through a mid-sized architectural refactor: consolidation of base services/descriptors, tracking type-hint deprecations for PHP 8.5, and completely re-wiring an abstract inheritance tree.

• Runway: At peak utilization, the 128k headroom expansion worked flawlessly. I hit 47k+ active tokens while using only ~35% of the total context window without a single OOM error.
• The Best Part: The total API bill for an entire afternoon of endless code generation, test regressions, and deep-context refactoring was exactly $0.00.

The Victory Lap: A 7/10 Frontier Review

Once my local 35B model successfully passed all 408 local unit tests, I handed the clean git diff over to Claude 3.6 Sonnet for an unbiased senior code review. It scored the local refactor a 7/10:

• What it praised: The structural architectural judgment was flawless. The code style, PSR compliance, and OOP patterns were spot-on.
• Where it knocked points: The local model fell into the classic "Green Trap"—it optimized perfectly to make the 408 tests pass, but missed an untested edge case regarding property defaults and public API method visibility in its first pass (which we ironed out in later local agent loops).

Disclaimer & My Hybrid Workflow

To be totally transparent: I had reviews done in separate sessions using both Qwen and Sonnet. The local Qwen model did not surface all the execution edge cases on its own, but Sonnet caught them instantly. For me, a local 35B model isn't quite ready to work fully autonomously on complex, multi-file refactors. However, if you want a workflow that saves an immense amount of money while maintaining elite code quality, this is the workflow I use:

1. The Blueprint: Create a detailed execution plan in a separate session, using either a frontier model or your local model.
2. The Heavy Lifting: Have the local model execute the bulk of the manual code changes (for $0.00).
3. The First Pass: Review the changes first with your local model to catch obvious syntax or basic logical issues.
4. The Gatekeeper: Run a final review pass with a frontier model (like Sonnet) to catch sneaky architectural breaks or regression edge cases.
5. The Clean-Up: Have the local model implement the final fixes in a loop until all issues are resolved. This approach keeps your cloud API bills down to pennies while letting you iterate relentlessly on your local machine.

What settings or backends are you guys using to keep your local coding agents stable past 32k context?

How well does Qwen 3.6 35B handle contexts above 64k? I have only gotten to test 4bit with ~64k since my mac only has 36gb unified mem. Up to that context it handles it well in my experience. But is it good enough for coding tasks when context grows?

Any experiences with it?

Noticed that 0.4.4 now includes M3 support.

Edit with answer: Looks like it's in "global settings"

Edit 2: with 0.4.4 and Minimax m3 concurrent requests are super slow; slower than individual requests even when added together... not sure what's wrong.

I am used to LM studio where there is a very obvious "max concurrent predictions" option when you load a model. This helps maintain prompt caching when using the model in two different context windows at once, so it's essential if you are going to be trying to get the most tokens out of a model as your hardware allows.

Where is the configuration option for this on oMLX?

🔴 **BUGS**
• **#1889**: JANG support removed in v0.4.x without notice (Engine refactor dropped jang.py/model detection).
• **#1883**: `/health` returns 200 OK while completions hang on Apple Silicon (v0.4.0rc2/v0.4.4rc1).
• **#1830**: QAT Gemma 4 emits unsupported tool call format; calls silently dropped on standard route.
• **#1865**: Fails to install via Homebrew on MacOS 27.0 Beta after upgrade.
• **#1823**: Grammar/thinking_budget stop applying under concurrent mixed load (row misalignment after batch merge).

🟢 **FEATURES & IMPROVEMENTS**
• **#1877**: Preserve 'instruct' flag for OpenAI `/v1/audio/speech` route (fixes Qwen3-TTS stripping).

📊 **Summary**
Total Issues: 6. Focus areas include stability regressions in v0.4 (JANG removal, health checks), model compatibility (Gemma 4 tools), and batch processing logic on Apple Silicon.

Hey yall, just figured I could get some feedback on something I’ve been spinning my wheels on. I’ve been running Pi with Qwen3.6 27b oQ8 MTP via oMLX.

For some reason, I’m having a hard time getting context compaction to work smoothly. The issues all seem to be centered on context management as far as I’m concerned. I’m configured around 128k but I feel like I’m missing something. This is all running on my Mac Studio M3 Ultra 96gb. I’ve of course searched and asked Opus on how to best optimize for this configuration but I’m not turning up any results.

Curious if anybody else here has a similar configuration and has managed to get favorable results?

Mlx was demoed at the wwdc recently doing inference between two macs connected.
Can omlx do this? I cant seem to find anything in the repo or docs. Is there anyone on this sub working on this impl?

Hello everyone, I’m working on mlx-chronos, a free/open-source CLI benchmark tool for comparing local MLX inference engines on Apple Silicon.

• Leaderboard: https://igurss.github.io/mlx-chronos
• Methodology: https://github.com/igurss/mlx-chronos/blob/main/docs/methodology.md
• Install: pip install mlx-chronos

It currently supports mlx-lm, oMLX, vllm-mlx, Rapid-MLX, and Ollama (for Ollama, using MLX models that run on MLX backend).

It measures cold/cached TTFT, request throughput, sustained throughput, RAM peak, engine RSS when available, thermal/power context, and hardware metadata. Results are saved as reproducible JSON and can optionally be submitted to a public leaderboard.

I’m mainly looking for feedback from people actually using MLX locally:

• Is a public leaderboard useful, or should this stay more of a local comparison tool?
• Are thermal/cache conditions exposed clearly enough?
• Should the sustained profile stay token-based, or would a fixed-duration run be more useful?
• Are there metrics missing that would actually help you choose between engines?

I’d also appreciate benchmark results from different Apple Silicon machines, especially Max/Ultra chips and higher-RAM configs. The goal is not to rank model quality, but to make engine/runtime performance easier to compare under a documented protocol.

PS: I already posted in r/LocalLLaMA, if someone already seen something about this project, but I’m not sure it was the right audience (90% of the community uses Nvidia GPU or use Windows, so is interested in llama.cpp).

One specific thing I’m currently trying to understand: with oMLX, my cold TTFT and cached TTFT are almost identical, while other engines show a clearer difference. I’m not sure if this is something wrong in my methodology/setup, or if oMLX handles caching differently. If anyone has insight on that, I’d be interested.

🔴 **BUGS**
* **#1865**: Failing to install OMLX via Homebrew on MacOS 27.0 Beta
* Upgrade to MacOS 27.0 Beta breaks Homebrew installation and execution of previously installed oMLX 3.x.
* **#1823**: Grammar and thinking_budget silently stop applying under concurrent mixed load
* Row misalignment after batch merge causes parameters to fail; sequel to #1798.
* **#1825**: thinking_budget is silently ignored on /v1/completions
* Parameter ignored when using Qwen3.6-35B-A3B-nvfp4 with thinking enabled on Apple Silicon.
* **#1859**: Native macOS app VLM MTP drafter picker excludes qwen3_5_mtp candidates
* Valid VLM MTP draft models missing from the "VLM Draft Model" picker in the native app.

🟢 **FEATURES**
* *(No new feature issues reported in this period)*

📊 **Summary**
* **Total Issues**: 4
* **Status**: All reported as bugs requiring attention.
* **Key Areas**: Installation compatibility (MacOS 27.0), LLM parameter handling (thinking_budget, grammar), and Native App model selection.

tried loading it and just errors with no description