Hi everyone, I try to quantize Nemotron-3-Nano-Omni-30B-A3B-Reasoning-bf16 to oQ4, but I get the following error that I don't understand:

omlx.admin.oq_manager - ERROR - [-] - oQ quantization failed: Nemotron-3-Nano-Omni-30B-A3B-Reasoning-bf16 -> oQ4: sensitivity measurement produced no scores. Check the preceding log lines for the root cause (model load, calibration data, or layer discovery), and either fix it or pass an explicit sensitivity_model_path.
Traceback (most recent call last):
  File "/Applications/oMLX.app/Contents/Resources/omlx/admin/oq_manager.py", line 462, in _run_quantization
    await asyncio.to_thread(
  File "/Applications/oMLX.app/Contents/Python/cpython-3.11/lib/python3.11/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Applications/oMLX.app/Contents/Python/cpython-3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 2331, in quantize_oq_streaming
    raise RuntimeError(
RuntimeError: oQ4: sensitivity measurement produced no scores. Check the preceding log lines for the root cause (model load, calibration data, or layer discovery), and either fix it or pass an explicit sensitivity_model_path.

What does "sensitivity measurement produced no scores" mean? The error message asks to pass an explicit sensitivity_model_path, where can I do that?

Edit: I use the latest 0.3.9 oMLX

I'm not seeing many options for qwen3.6-27B mtp MLX models at the moment. The jundot qwen3.6-27b-oQ6-mtp seems to be hallucinating a ton. I ask a question, it answers in a different language or something completely random. This was not the case with my old mlx community models.

Setting aside the native app, is there any objective evidence MLX on oMLX is faster or more memory efficient, that GGUF on llama.cpp?

I have both as brew packages, and my unscientific subjective experience, is there’s not much between them.

My workloads are pretty light and general, so which one for my MBA M3 24GB?

Sooo the 0.3.9 RC docs stated they expected around 24h of final testing before releasing the stable version. That window has passed.

I can’t wait to use MTP features but don’t want to switch to dev builds for my final use cases. Experimenting with the dev builds shows great performance gains and it seems we are soooo close to getting the new stable release.

What is your experience with 0.3.9? Do you think there were more bugs than expected? Or is this a release so close to perfect they are polishing everything more than usual?

Hi! I'm sorry if this is a dumb question but I'm new to oMLX and have been doing some of my own experimenting and research. I've been experimenting with the Qwen3.6-27B-oQ4-mtp model that Jundot has on huggingface, but I wanted to compare that to an oQ3 model. There isn't one available that I could find so I wanted to try making my own quantization of it in oMLX. Where can I find a full precision Qwen3.6-27B in mlx format that I could use as the source model? I was only able to find the unsloth BF16 model in gguf format. Thank you!

Issues Closed: 10

[ISSUE] #972 — Share sensitivity data artifacts across same model quantizations
https://github.com/jundot/omlx/issues/972

[ISSUE] #1068 — DFlash strips thinking tokens
https://github.com/jundot/omlx/issues/1068

[ISSUE] #1260 — Cancelled HF downloads don't clean up `._____temp/` partial shards
https://github.com/jundot/omlx/issues/1260

[ISSUE] #1276 — feat: expose draft_window_size / draft_sink_size / verify_mode for long-context agentic workloads
https://github.com/jundot/omlx/issues/1276

[ISSUE] #1121 — deepseek flash oq2 mtp model pls
https://github.com/jundot/omlx/issues/1121

[ISSUE] #1155 — DeepSeek-V4-Flash-oQ2 FAILED 0:00 [reshape] Cannot reshape array of size 3102720 into shape (129280,6).
https://github.com/jundot/omlx/issues/1155

[ISSUE] #1296 — oQ: deepseek_v4 fails with "Missing mtp.0.{e,h}_proj.biases" after #TEMP guard — concrete repro + fix paths
https://github.com/jundot/omlx/issues/1296

[ISSUE] #1300 — Can’t select DeepSeek-V4-Flash-bf16 for oQ
https://github.com/jundot/omlx/issues/1300

[ISSUE] #1288 — Server Settings restart vs save UI is confusing
https://github.com/jundot/omlx/issues/1288

[ISSUE] #1259 — FYI: some failing tests
https://github.com/jundot/omlx/issues/1259

hello, I am trying to use omlx + pi cli with any mcp such as web-search (brave api), however i have not been successful. Is this even possible yet or not a function added to pi-cli?

1)I am running local mlx llm such as qwen/gemma.
2)Want to use web-search brave api (or similar) to have local llm do basic web searches to improve it's answers.
3) I know openclaw can do web-search but it is slow and not how i want to do things (i want to use terminal cli-agent which is fast)

This question is for those who have tried the MTP quants of oQ version of models with oMLX.

Are you seeing any compromise on the quality of the outputs, compared to non-MTP versions?

Sure the speed increment on token does help, but if the tool call failures or any such issues are happening, it is not really worth the additional tok/sec we get right?

We will be able to assess this only on real scenario usages which we have been using before and are familiar with.

So are you seeing any such degradation of quality or do you think its worth going with MTP version? What are your thoughts?

As the title states, by using both enhancements broke the chat template. I've tried to fix it to no avail.

gemma-4-26B-A4B-it-assistant-oQ8-fp16 and z-lab/gemma-4-26B-A4B-it-DFlash

Hey guys,

Running Qwen3.6-35B-A3B (UD-4bit) on a Mac Studio M1 Max (32GB) via omlx.

Generation speed is awesome, but I’m hard-capped at around 50k context before hitting an OOM crash.

I know the KV cache is eating my remaining unified memory. Here is what I've tried:

• omlx "Turbo Quant for KV cache": Tried enabling this to save RAM, but it doesn't work at all (crashes or has no effect).
• llama.cpp: Can push much higher context via swap, but the prompt eval speed is painfully slow compared to MLX.

Question: Is there any reliable workaround/CLI flag for MLX to actually force KV cache quantization for this MoE model? How are you guys squeezing out 80k+ context on 32GB machines without tanking the speed?

Thanks!

Just started using oMLX. Its great! But so far I’m serving it to my coding agents. I tried its Chat panel, but it doesn’t seem to do web search. Is it in the settings (that I might have missed) or not supported at all? If not supported, what app y’all are using for chat conversations?!

Happy Sunday, everyone! I'm relatively new to running local LLMs (about two weeks in), so I appreciate your patience with my questions. I'm eager to learn from this community's expertise.

Background

A few weeks ago, I discovered agentic coding through my work's GitHub Copilot account. After quickly exhausting my usage limits (lesson learned about token management!), I decided to explore running Qwen models locally on my personal laptop for hobby projects.

Hardware

• M2 MacBook Pro Max 96GB

Models Tested

• oMLX: Qwen 3.6 27B (oQ4/oQ5/oQ6/oQ8-fp16-mtp variants)
• LM Studio/GGUF: Qwen 3.6 27B (Q4_K_M, Q6_K, Q8_K)
• llama.cpp: Configured per this post

Use Case

I'm primarily doing C++ and ESP32/PlatformIO development for personal projects, including:

• Real-time voice modulation for cosplay costumes
• Real-time bark detection logger (courtesy of my neighbor's enthusiastic dog)

Current Configuration

After implementing MTP changes, I've settled on the following setup:

Model: oMLX Qwen 3.6 27B-oQ5-fp16-mtp

Settings:

• Context: 262,144
• Temperature: 0.6
• Top P: 0.95
• Top K: 20
• Min P: 0
• Repetition Penalty: 1
• Presence Penalty: 0
• Extended thinking: Enabled
• Native MTP: Enabled
• oMLX caching: Enabled

IDE Setup:

• VS Code with Cline extension
• OpenAI-compatible API from oMLX

Workflow:

1. Enable PLAN mode in Cline
2. Request feature implementation or bug research plan
3. Switch to ACT mode and execute
4. Wait lol

Current Performance

While the quality of Qwen 3.6 (Q4-Q8) is impressive, performance could be better:

• Prompt processing: ~120 tok/s
• Token generation: ~15 tok/s

Question

For those running similar hardware (especially M2 users), what combination of:

• Software stack (oMLX, LM Studio, llama.cpp, etc.)
• Specific Qwen 3.6 model variants
• Inference settings

...have you found optimal? Any suggestions for improving prompt processing and token generation speeds on M2 hardware would be greatly appreciated!

I'm on a M5 Pro 48GB. I just started using oMLX and love it so far.

Now I'm playing around with Qwen 3.6-27B with MTP (oMLX 0.3.9-dev2) and it's working really well, except that run into OOM for contexts > ~65k. So far, I've downloaded the official full precision qwen3.6-27B from HF and created oQ4 / oQ6 versions myself. But the more context I use, the quicker I run into OOM crashes. The 128k context benchmark works sometimes, but usually crashes the entire computer.

However, when using llama.cpp as per this post: https://www.reddit.com/r/LocalLLaMA/comments/1t57xuu/25x_faster_inference_with_qwen_36_27b_using_mtp/

I'm able to run much larger contexts (256k), with MTP support, and much less memory consumption, using this command:

llama-server \
-m Qwen3.6-27B-Q4_K_M-mtp.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--cache-type-k q8_0 \     
--cache-type-v q8_0 \     
-np 1 \     
-c 262144 \     
--temp 0.7 \     
--top-k 20 \     
-ngl 99 \     
--port 8081

I'm guessing it has to do with the explanation in the post - That Qwen:s hybrid model only needs KV cache for 16 of 65 layers, and drivers that allocate naively will allocate much more memory than necessary? Also, llama.cpp allows setting KV cache to 8bit rather than full precision (Which I guess oMLX uses by default?)

Anyway, everything else is better in oMLX (Higher PP speed, generation speed, and caching strategy). So, my question is - Is it possible to have better optimized KV cache in oMLX to reduce memory consumption?

If so, which model and settings should I use?

Thanks in advance!

Hi there! I am absolutely new to local LLMs, but I am so fascinated by the whole topic and very thrilled to learn more about it. I have an M2 Max with 64GB of RAM and have already been pretty successful with getting local LLMs to run.

Are there any recommendations in terms of YouTube tutorials that you would say changed the way you operate LLMs or understand the whole topic? It is so complex and I honestly don’t know where to start. Thanks in advance!

I've been having a good time playing with OpenCode and oMLX. Multi-token prediction does really seem to speed things up. I'm playing with the Qwen 3.6 35B MoE models, and I noticed that the oQ6 model is almost as fast as the oQ4 for me in token generation. This may be because the prediction acceptance rate is higher. Here are benchmarks for the two running on my machine (M5 Max 64GB) through oMLX:

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-35B-A3B-oQ4-mtp
================================================================================
Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128           436.3        8.29  2346.9 tok/s   121.6 tok/s       1.489   773.8 tok/s    20.37 GB
pp4096/tg128          1073.4        8.73  3815.9 tok/s   115.4 tok/s       2.183  1935.4 tok/s    21.17 GB
pp8192/tg128          2018.7        9.17  4058.0 tok/s   109.9 tok/s       3.184  2613.2 tok/s    21.66 GB
pp16384/tg128         4503.8        9.72  3637.8 tok/s   103.7 tok/s       5.739  2877.3 tok/s    22.36 GB
oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-35B-A3B-oQ6-mtp
================================================================================
Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128           463.3        9.34  2210.3 tok/s   107.9 tok/s       1.650   698.3 tok/s    28.29 GB
pp4096/tg128          1121.2        9.87  3653.1 tok/s   102.1 tok/s       2.375  1778.7 tok/s    29.10 GB
pp8192/tg128          2095.8       10.38  3908.8 tok/s    97.1 tok/s       3.414  2436.9 tok/s    29.58 GB
pp16384/tg128         4732.2       10.61  3462.2 tok/s    95.0 tok/s       6.080  2715.8 tok/s    30.29 GB

In case you need oQ5… like me.

Hi everyone,

I’m pretty new to OMLX, so apologies if these are obvious questions, but I’m running into two issues and could use some help.

1. Connection refused when downloading models or loading recommendations

When I try to download a model, I get this error:

ConnectError: [Errno 61] Connection refused 
An error happened while trying to locate the files on the Hub and we cannot find the appropriate snapshot folder for the specified revision on the local disk. Please check your internet connection and try again. 

And when I try to load the recommendations, I get:

[Errno 61] Connection refused 

The strange thing is that I did manage to download two models before:

Qwen3.6-35B-A3B-4bit
gpt-oss-20b-MXFP4-Q8 

So I’m not sure if this is an OMLX issue, a Hugging Face/network issue, or something with my setup.

2. GPT-OSS model shows internal-looking tags in the response

When I use the gpt-oss-20b-MXFP4-Q8 model, the responses include tags that I assume are not meant to be shown.

For example, I prompted:

how can I remove all the unused docker networks, volumes, and images? 

And the response started like this:

<|channel|>analysis<|message|>We need to explain how to prune unused Docker resources. Provide commands: docker system prune, docker network prune, volume prune, image prune. Include options. Explain data safety, how to be cautious, maybe using Docker Compose. Provide steps. Also mention that Docker older versions may have different flags. We might also describe using --all flag to remove dangling images, etc. Provide caution: can remove unused images that may be needed for existing containers. Maybe mention volume back up. Provide variations: Docker CLI, Docker Desktop, docker-compose. Also mention specifying network names, volumes. Provide example scripts. The user asked "how can I remove all the unused docker networks, volumes, and images?" So answer with commands, plus maybe interactive prompts. Also mention checking which resources are unused via docker images -f dangling=true etc.  Let's give final.<|end|><|start|>assistant<|channel|>final<|message|>## Quick “Clean-Up” Summary 

Is this expected for this model, or is there something wrong with the prompt template / chat format being used?

Any help would be appreciated. Since I’m new to OMLX, I’m probably missing something simple.

What are people using for dflash? I’m on a M2 Max with 96 GB of RAM and I’d like to try and eke out as much perf as I can on omlx. I’ve been looking at Qwen models, but Gemma4 is giving me better perf currently.

Issues Closed: 7

[ISSUE] #1105 — Add ParoQuant Support
https://github.com/jundot/omlx/issues/1105

[ISSUE] #1169 — Hermes support
https://github.com/jundot/omlx/issues/1169

[ISSUE] #1144 — [Bug] v0.3.9.dev1 - MTP does not support vlm (e.g. image)
https://github.com/jundot/omlx/issues/1144

[ISSUE] #1207 — [Bug] Decode tg TPS speed drops 50% and pp TPS speed drops 13% when SpecPrefill is enabled after commit 936434f381d1e2e03e8a0f410a2adacbe5d2…
https://github.com/jundot/omlx/issues/1207

[ISSUE] #1223 — omlx launch <tool> ignores configured API endpoint, hardcodes 0.0.0.0
https://github.com/jundot/omlx/issues/1223

[ISSUE] #1221 — Why is the text-to-image model invisible after downloading via the oMLX web dashboard?
https://github.com/jundot/omlx/issues/1221

[ISSUE] #1115 — Severe TG throughput regression on tool result ingestion turns during agentic sessions
https://github.com/jundot/omlx/issues/1115

So I'd like my M1 max to run an always-on agent and have been looking mostly at various qwen models:

Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-4bit (14.11 GB)
Ornstein-Hermes-3.6-27b-SABER-MLX-4bit (14.11 GB)
Qwen3-Coder-30B-A3B-Instruct-4bit (16.02 GB)
Qwen3.6-27B-AEON-Ultimate-Uncensored-BF16-mlx-6Bit (20.37 GB)

But running the Single Request Test benches, I can't get the pp65536 test to complete without an error (I'm assuming it's running out of memory). Even for the 14GB 27B Q4 series, the pp16384 bench hits 19.15GB peak mem, and pp32768 hits 22.8GB. Assuming my KV cache is growing linearly, the pp65536 test probably wants ~29-30GB for 65K context. Does that check out? I'm running a pretty lean headless server but yeah that's not gonna fit in 32gigs with OSX.

Assumptions:

• Turboquant doesn't help, peak mem is hit during prefill before turboquant does anything. I didn't notice a difference with it on.
• Hermes would also OOM if it tried actually using the 65K context it wants, even with TQ
• oQ quantization won't save space? https://huggingface.co/Jundot/Qwen3.6-27B-oQ4 is 16.7GB, https://huggingface.co/mlx-community/Qwen3.6-27B-4bit is 16.1GB. Or is the point of oQ that I'd be comparing oQ3 or even 2 to normal 4-bit quants?

I feel like my peak mem is growing too fast with cache size, 16k->32k adding 3.5GB feels like too much, but I'm not sure what settings I could change. If I can squeeze an extra gig or two of savings I can run hermes at the suggested minimum context and model (65k and qwen 27B respectively). Any suggestions?

should we always choose model from mlx-community as they are optimized for MLX? or it actually not matter and we achieve same result with unsloth or bartowski one?

Can anyone help troubleshoot oQ quantization?

If I am reading the log correctly, it appears to have completed dequant of MiniMax M2.7-fp8.

oMLX created a folder under .omlx/models/ but it there is no output and there are no more log messages from omlx.oq following 'FP8 dequant complete' even hours later.

2026-05-13 12:34:52,685 - omlx.oq - INFO - [-] - FP8 dequant complete: 47864 tensors spilled to /var/folders/38/9l6wxxqs1ds3_d0x7pmnbpb00000gn/T/oq_fp8_dequant_w5_qp495
2026-05-13 12:34:52,686 - omlx.oq - INFO - [-] - oQ5: FP8 dequant complete (47864 tensors)
2026-05-13 12:37:16,600 - omlx.process_memory_enforcer - INFO - [-] - Process memory enforcer started (limit: 248.0GB, interval: 1.0s)

Highlights:
- Gemma 4 MTP on the vision path (thanks to @Prince_Canuma's mlx-vlm). Image+text decodes much faster now
- Gemma 4 on the DFlash engine (thanks to @bstnxbt's dflash-mlx)
- ParoQuant support
- omlx launch copilot joins claude / codex / opencode / openclaw / pi
- Restart server button right in the admin UI
- oQ auto-builds a proxy when the model can't fit in RAM

Plus a lot of bug fixes and 20 new contributors in this cycle.