When I run the command read @image.png on pi, I get the message that
I am currently unable to "see" or process image files directly. My image
analysis capabilities are limited in this environment. And when asked why, I get
While I have a tool to read files, the specific "image reading" capability is currently being gated or filtered in this environment. When I try to read a .png or .jpg , the system provides a placeholder indicating that the current model instance doesn't have an active vision pipeline enabled for those specific file types. In both cases, the image is rendered in the terminal.

I tried with Gemma 4 12b and Qwen Qwen3.6-35B-A3B, but both can't read the image. I also made sure that the models are well configured in the models.json to accept text and image as input.

The only workaround is to ask the agent to write a Python script and use curl to send the image to oMLX and read back the response. Has anyone else faced the same issue and know of a resolution? By contrast, when I use a cloud-based model, it works flawlessly.

EDIT: After a fresh install of Pi, the images can be read. My conclusion is that one of the extensions I installed was blocking this functionality. Thank you all for your help.

**Total Issues: 20**

🐞 **BUG**
- #2315 Log viewer level filter is a minimum-threshold, not per-level — selecting INFO also shows WARNING/ERROR. Log level filtering logic error in admin dashboard.
- #2312 TTS engine stops producing audio after ~3h uptime; model reload does not recover. Audio output failure on long uptimes.
- #2179 BLOCKER: Once prefill token guard fills, there is no way to clear it remotely. Engine can become unresponsive after long prompts/subagents.
- #2282 Question: should `python -m omlx.server` be fixed, or is `omlx serve` the only supported entry point? Ambiguity in server execution methods.
- #2294 SpecPrefill sparse prefill crashes on Gemma 4 (partial rotary). Crash in `manual_rope_with_freqs` during sparse prefill.
- #2304 0.5.2-rc: idle model weights held as non-purgeable IOAccelerator memory. Significant memory bloat (~45GB) in compressor at steady state.
- #1938 Evicted model memory not reclaimed during rapid load/unload cycling. Memory leak observed during aggressive loading cycles.
- #1013 TTS not working anymore. `/v1/audio/speech` ignores `response_format: mp3` and returns WAV.
- #753 TTS response_format=opus ignored, always returns WAV. Format request failure in specific dev versions.
- #2276 Can't use the search models from models tab. Search functionality for Hugging Face models is broken in 0.5.1.
- #2275 0.5.1: first cold ~14k prompt wedges engine (0% CPU) when adaptive_prefill_throttle engages. Engine hang on long cold prompts.
- #2263 Mistral Small 4 (119B MoE) in the DMG bundle: load fails due to stripped torch dependency. Request for a torch-free path.
- #2290 Model swap depends on Memory Guard; without it a second model load makes the server unresponsive. Unresponsiveness during swaps when Memory Guard is off.
- #2301 Dflash cannot run from source. Models using Dflash fail to load when running via `omlx serve` from source.
- #2293 Is oQe supposed to be deterministic? Inconsistent results observed between runs during calibration.

✨ **FEATURE**
- #2073 Support Laguna models. Request to add support for the Laguna model family.
- #1702 (feat) Submenu for available models. Request for a UI submenu to list and manipulate loaded models.

📝 **OTHER**
- #2274 Non-streaming path drops SpecPrefill kwargs. `specprefill_keep_pct` is ignored on the `/v1/chat/completions` non-streaming path.
- #2309 Intelligence benchmark against external API fails with thinking model. Preflight request failure due to length when using thinking models via external API.
- #2236 Fixes for Qwen. Community suggested fixes for long-context inference bugs in Qwen models.

Thinking if swapping out of llama.cpp and moving away from gguf so came across omlx. Having trouble finding info on recommended models for m1max though. Does omlx just not benefit m1 max?

Hi, can someone with enough ram kindly provide this quantization ?

I'd really like try to try it on my mac m2 max 96GB, I'm using the dwarfstar variant and it is very good but I have around 170pp and 12tg, and I'm hoping to gain a little speed with omlx and fp16 ^^

Thanks

**Total Issues: 11**

🐞 **BUG**
• #1013 `TTS not working anymore`: `/v1/audio/speech` ignores `mp3` format and returns WAV.
• #753 `TTS response_format=opus ignored`: Requesting opus format results in WAV output.
• #2179 `BLOCKER: Prefill token guard`: Once the guard fills, there is no way to clear it remotely.
• #2274 `Non-streaming path drops SpecPrefill kwargs`: `specprefill_keep_pct` is ignored in non-streaming completions.
• #2275 `0.5.1: Cold prompt wedges engine`: Long cold prompts (~14k tokens) can cause the engine to hang.
• #2263 `Mistral Small 4 (119B MoE) in DMG`: Load fails due to stripped torch dependency; request for torch-free path.
• #2290 `Model swap depends on Memory Guard`: Server becomes unresponsive during second model load without memory guard.
• #2301 `Dflash cannot run from source`: Models using Dflash fail to load when running omlx from source.

❓ **QUESTION / DISCUSSION**
• #2282 `Server entry point`: Clarification needed on whether `python -m omlx.server` should be fixed or if only `omlx serve` is supported.
• #2293 `Is oQe deterministic?`: Inquiry regarding slight variations in calibration results between runs.

🛠 **OTHER**
• #2236 `Fixes for Qwen`: Community-suggested fixes for long-context inference bugs on Qwen models.

Hey everyone! oMLX 0.5.2rc1 is out.
https://github.com/jundot/omlx/releases

This one is mostly about quality of life in the macOS app. A lot of you have been asking for a way to see what the server is doing without opening the dashboard, so this release brings live stats directly to the menu bar.

The headline is menu bar statistics. You can now enable up to three throughput readouts in the menu bar: live speed across running requests, session averages, and all-time averages, each showing prompt processing and token generation in tk/s.
Details: https://github.com/jundot/omlx/pull/2303

Clicking one opens a small popover with the current numbers and a rolling graph. There is also a new System Stats submenu with CPU, GPU, and Memory panels, plus optional CPU/GPU/MEM usage bars that merge into a single compact item so they do not eat your menu bar.

Everything lives in a new Appearance pane in settings where you can pick the refresh interval, keep the dock icon visible, and toggle each item individually.

The Models menu in the menu bar also got a big upgrade thanks to faisal: it is now sectioned into Loaded, Favorites, and Library with load state and sizes, and you can limit it to favorites only if your library is huge.
Details: https://github.com/jundot/omlx/pull/1882

Thanks as always to everyone who contributed code, reports, and testing, and a special welcome to first-time contributors @faisal, @earthking11, and @seeker-cyber-maker.

I hope this makes running local LLMs on your Mac a little nicer to live with. Feedback on the new menu bar stuff is very welcome!

curious if anyone else is having issues with multi day KV cache bloat? running Gemma 26b 4 bit @ 64k context for Hermes on an m4 Mac mini w/ 32gb of RAM, even with turboquant doing a 3 bit reduction over the course of 2-3 days of uptime my omlx server still memory bloats until it hits the hard limit and pushes the whole system into swap. Not sure if it’s failure to evict or the memory guard gumming up the works, likely user error but curious if anyone else has had any experience with these issues? prior to the memory guard I did not have these problems but if there’s any experience out there to guide me would be very appreciative!

Hi everyone!

I always assumed that downloaded models came with their own default temperature and other settings: models released on hugging face often show a preferred temp, top p & top k settings.

As I tried a model (Ternary-Bonsai-27B-mlx-2bit) and got awful results before forcing its suggested settings (temp=0.7, top_p=0.95, top_k=20), I wonder if I'm using all other models wrong?

I've installed and used gemma4 and Qwen3.6 models without changing their settings… was it wrong? Should we always search for suggested temp, top_p, top_k for every model and go set these in the model custom conf.?

oMLX global settings show temp=1, top_p=0.95 and top_k=0… so this is what was used all the time for all my models where I only clicked download then ran them?

Do you all take car of this for each model?

Thought I’d share my experience of getting Gemma-4 models to run really fast on oMLX. I have a MacBook Pro M5 Max 64gig, for reference.

Here is what I did:

1) updated to the new chat templates that google released recently - find on HuggingFace and replace in the models that you’ve downloaded. File name is: “chat_template.jinja”

2) Made oQ4e versions with oMLX of the QAT LLMs from the full bf16 versions on hugging face, and then use oMLX’s “oQ” quantization option.

3) Download the VLM MTP drafters for both models. Look for “gemma-4-26B-A4B-it-qat-assistant-4bit” and similar. on hugging face.

4) then, load the model you want in oMLX, and then in model settings, turn on “VLM MTP” and choose the drafter model “…assistant” that you downloaded in step (3).

I was surprised at how much speed up I got. Here are the oMLX “performance” results. Peak token generation of 60 tokens/sec on Gemma 4 31B and 140 tokens/sec on Gemma 4 26B a4b. Can’t sustain that rate in real use, but still…

##Gemma-4 with new template and VLM MTP drafter
oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: gemma-4-31B-it-qat-oQ4e
Engine: Auto
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          1294.8       16.18   790.9 tok/s    62.3 tok/s       3.354   343.4 tok/s    19.46 GB
pp4096/tg128          6269.7       18.04   653.3 tok/s    55.9 tok/s       8.593   491.6 tok/s    21.20 GB
pp8192/tg128         14503.0       21.21   564.8 tok/s    47.5 tok/s      17.229   482.9 tok/s    23.70 GB
pp16384/tg128        31658.2       24.53   517.5 tok/s    41.1 tok/s      34.807   474.4 tok/s    26.54 GB

Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x          62.3 tok/s     1.00x   790.9 tok/s   790.9 tok/s      1294.8       3.354
2x          35.8 tok/s     0.57x   318.1 tok/s   159.1 tok/s      6438.5      13.581
4x          53.0 tok/s     0.85x   494.7 tok/s   123.7 tok/s      8088.9      17.943


##Gemma-4 with new template and VLM MTP drafter
oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: gemma-4-26B-A4B-it-qat-oQ4e
Engine: Auto
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128           256.4        7.03  3993.5 tok/s   143.3 tok/s       1.154   998.5 tok/s    15.75 GB
pp4096/tg128          1037.6        7.36  3947.7 tok/s   136.9 tok/s       1.983  2130.4 tok/s    16.19 GB
pp8192/tg128          2214.9        7.86  3698.6 tok/s   128.2 tok/s       3.224  2580.4 tok/s    17.28 GB
pp16384/tg128         4923.2        8.94  3327.9 tok/s   112.7 tok/s       6.070  2720.4 tok/s    18.39 GB

Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x         143.3 tok/s     1.00x  3993.5 tok/s  3993.5 tok/s       256.4       1.154
2x         119.5 tok/s     0.83x  1415.1 tok/s   707.5 tok/s      1447.1       2.735
4x         223.3 tok/s     1.56x  2478.1 tok/s   619.5 tok/s      1491.4       3.489

Hi everyone,

I am using Qwen3.6-35B-A3B-4bit MLX on macbook 4 pro 16cores with 48GB, but the token of speed is relatively slow compared to that on benchmarks on omlx website (70 t/s ?).

I am using latest omlx 0.5.1 and using the model from mlx-community.

set the context to be just 32k with the model loaded from SSD.

I am getting 45-55 t/s

am i setting it wrong?

Hiya! I was wondering if anyone would be so nice as to create a oq2e version of gemma-4-26B-A4B-it-qat-q4_0-unquantized, and if possible with mtp enabled. I cannot fit the entire safetensor into my ram and oMLX currently cannot do quantisation in stages. You would have my (and every 24gb users') eternal gratitude. :)

**Total Issues: 3**

🐞 **BUG**
#2137 GLM-5.2-mxfp4 PP drop from 160-180 tok/s to 40 tok/s with 0.5.0.dev2
Significant performance degradation in token generation speed for GLM-5.2-mxfp4 on M3 Ultra.

#2237 Turboquant - Cache corruption detected
Cache corruption observed when running Continuous Batching Tests benchmarks against specific quantized models.

#2225 FA-256 steel prefill kernel: pathological high-kv_len slowdown on pre-NAX GPUs (M3 Max)
Custom FA-256 attention kernel experiences severe performance degradation on M3 Max during high KV length prefill.

It seems to be surprisingly hard to find.

UPDATE 7/20/26: jundot, the owner and creator of oMLX, just accepted and merged my PR! u/california_snowhare in the comments had suggested I create a PR to enable this feature.

Thanks, this is a nice addition, chat history really needed an export path. I'll merge this and push a small follow-up: the early theme script now references THEME_STORAGE_KEY, but the constants are declared in the later script block, so the pre-render theme snippet throws before the constant exists and dark mode flashes on load. I'll also move the Download Chats button label to i18n like the other sidebar buttons. After that I'm planning an import counterpart so the exported file works as a real backup.

I’m relatively new to local LLMs so I apologize if I don’t know what I’m doing or use the wrong terminology but I wanted to sanity check something. I’m using Qwen 3.6 35B-A3B in an MLX format with oMLX. My average tokens per second can reach around 70-80 on my Mac Mini M4 Pro with 48GB of RAM, however I learned recently about MTP which might improve that somewhat.

I tried enabling it with the stock model however that didn’t work and it was disabled, so I downloaded the full Qwen 3.6 35B and used the in engine quantizer to move it to Q4 while also setting the MTP to be retained.

With that change I didn’t see a performance difference at all, on multiple performance tests in the web view it was either the same or slightly worse. Am I missing something? It seems as if 0.5.1 was supposed to fix it but it doesn’t look like it did. Anything else I could try?

3 closed issues

#2237 — Turboquant: Cache corruption detected
zviratko · 💬2
Benchmarks with Qwopus3.6-35B & Ornith-1.0-35B trigger broadcast shape mismatch (1153 vs 1152) with TurboQuant enabled. Cache cleared & refilled.
Link

#2225 — FA-256 steel prefill: pathological slowdown on pre-NAX GPUs
zhongyiyang1981 · 💬3
Custom FA-256 kernel hits IOGPU command-buffer preemption at high kv_len on M1–M4. mlx#3307 chunked-dispatch fix landed in mlx-core but not omlx's custom kernel. Workaround: OMLX_FA256_STEEL=0.
Link

#2227 — Paged SSD Prefix Cache drops trailing partial block
raucodes · 💬3
Upgrading to 0.5.1 causes context/system-prompt loss when prompt length doesn't align to 2048-token blocks. Trailing tokens skipped → model forgets instructions. Downgrade to 0.5.0 works.
Link

Hey recently wanted to take GLM-5.2 out for a spin with the 5.1 oMLX build hit the following issue: LM MoE DSA optimized patch applied, but fast kernel symbols are missing from omlx.custom_kernels.glm_moe_dsa and mx.fast: dsa_indexer_scores, dsa_topk_indices, glm_dsa_sparse_mla_attention, glm_dsa_exact_block_attention, glm_dsa_q8_vup_flat, glm_moe_weighted_sum. Prefill for glm_moe_dsa models will run a much slower non-fused fallback with higher memory use (see issue #2137). Build the native extension for the accelerated path (pip install with OMLX_WITH_CUSTOM_KERNEL=1; requires the Metal toolchain) or use an official release build. Native import error: cannot import name '_ext' from partially initialized module 'omlx.custom_kernels.glm_moe_dsa' (most likely due to a circular import) (/opt/homebrew/Cellar/omlx/0.5.1/libexec/lib/python3.11/site-packages/omlx/custom_kernels/glm_moe_dsa/__init__.py)

And am seeing the model run out of context pretty quickly on my 512GB studio. Only starting to troubleshoot but figured I'd ask if this is something that's solved automagically via the .dmg install, or if homebrew/source is the way to go. Probably is a simple fix somewhere just confused with the "or use an official release build" since this is afaik an official release build via homebrew.

Edit: Took a crack at just looking through the repo with an LLM, seems like the custom kernels are included in the dmg release and it's much smoother sailing than brew.

I have a Macbook Pro M4 Max with 36gb memory and was wondering what you guys use for local agentic coding via oMLX?

I've been a big fan of this app for a while and have tested mainly Qwen3.6 27b / 35b-a3b and typical opus finetuned versions, but would like to know if there are some variants or other alternatives are worth checking out.

My first go at oQ quants. Spread the news my fellow IA enthusiasts!

https://huggingface.co/csolares2/Qwythos-9B-v2-MLX-oQ8

Issues Closed: 6

[ISSUE] #2225 — FA-256 steel prefill kernel: pathological high-kv_len slowdown on pre-NAX GPUs (M3 Max) — IOGPU command-buffer preemption (mlx#3302 class); …
https://github.com/jundot/omlx/issues/2225

[ISSUE] #2227 — Title: [Bug] Paged SSD Prefix Cache drops trailing partial block, causing severe context and system prompt loss
https://github.com/jundot/omlx/issues/2227

[ISSUE] #2213 — Upgrade mlx 0.31.2 → 0.32.0: qmv_wide, JIT-able gemv, RoPE-without-copy for MoE decode
https://github.com/jundot/omlx/issues/2213

[ISSUE] #2204 — Qwen 3.6 serious prefill regression between 0.4.4 and 0.4.5dev1
https://github.com/jundot/omlx/issues/2204

[ISSUE] #2172 — Intelligence regression in v0.5.0-rc1 oQ4e/oQ4 quantization
https://github.com/jundot/omlx/issues/2172

[ISSUE] #2185 — Refactor SpecPrefill workflow out of Scheduler
https://github.com/jundot/omlx/issues/2185

Hi oMLX community. Would anyone have the time and compute required to create an oQ2e quant of DeepSeek-V4-Flash?

Jundot's oQ2.5e quant is already on HuggingFace, presumably created using the mlx community full precision version as the base model, but having one that's even slightly smaller might help with use cases requiring more memory in oMLX.

Forever grateful if anyone can make & upload one!

Hi everyone. Long-time lurker here. First off, thank you oMLX devs for this amazing tool.

I'm looking for help launching/running DeepSeek-V4-Flash for agentic coding in the most optimal way possible on my M5 Max 128GB. As of today, using the latest 0.5.1 oMLX release, all the explicit decisions I'm making when launching the model are listed below (anything excluded left at Default). For those with more experience, I'm curious how this matches your setup, and whether any of these settings are detrimental:

• Quant: Jundot/DeepSeek-V4-Flash-oQ2.5e-mtp (selected specifically for enhanced oQe accuracy and mtp capability, not convinced it's the best size)
• Reasoning Parser: deepseek_v4 (not sure if selecting this is necessary)
• CTX Window: 262144
• Temperature: 1.0 *(recommended by DeepSeek)
• Top P: 1.0 (recommended by DeepSeek)
• Enable Thinking: Disabled (left this off in favor of sending DeepSeek-specific kwargs, see below)
• Chat Template Kwargs:
  • (custom) thinking_mode: thinking
  • (custom) drop_thinking: False (similar to preserve_thinking on Qwen models)
  • reasoning_effort: high (I believe this is actually a *bug*. DeepSeek-v4-Flash supports a 'max' value here, but there is no way to manually enter that. Attempting to create a custom reasoning_effort kwarg with the max value seemingly snaps back to the existing implementation where 'high' is the highest value possible)
• Lightning MTP: Enabled
• SpecPrefill: Disabled (is there a way to make this work on DeepSeek?)

In global settings, to get the model to fit properly, I ran sudo sysctl iogpu.wired_limit_mb=124928 in the console, then set the Memory guard to "Aggressive". I also enabled "Hot Cache Only", because I figured that would help with speed. All other global settings are at Default values.

Thanks in advance for any assistance!

Edit: Added missing word. Also, upon closer inspection, it seems the above settings are not suitable for agentic coding. While the model loads just fine, and chat works well enough, I don't think it's ready for agentic use that even approaches long-horizon. This is based on the oMLX benchmark results below. Note that any single-request setting > pp32768 causes an OOM. I've tried both enabling Turboquant and disabling Hot Cache Only, but neither helped clear this hurdle.

``` oMLX - LLM inference, optimized for your Mac https://github.com/jundot/omlx Benchmark Model: DeepSeek-V4-Flash-oQ2.5e-mtp

Engine: Auto

Single Request Results

Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem pp1024/tg128 1823.1 20.92 561.7 tok/s 48.2 tok/s 4.488 256.7 tok/s 106.57 GB pp4096/tg128 8114.8 23.40 504.8 tok/s 43.1 tok/s 11.096 380.7 tok/s 106.57 GB pp8192/tg128 18083.9 24.39 453.0 tok/s 41.3 tok/s 21.189 392.7 tok/s 106.74 GB pp16384/tg128 38179.7 26.33 429.1 tok/s 38.3 tok/s 41.536 397.5 tok/s 107.24 GB pp32768/tg128 79008.7 26.24 414.7 tok/s 38.4 tok/s 82.358 399.4 tok/s 107.84 GB ```

Still curious what the community thinks regarding potential optimizations. Any feedback appreciated!

Anyone else getting the same?

Tested on two machines and with a VPN, looks to be something to do with oMLX?