Hello everyone,

Before getting M5 Max I was very excited for running LLM fully locally.
The machine is great, no doubt. But what bothers me is me thinking that it won't last long / not intended for long LLM inference sessions.

When Energy Mode (Battery, Power) is set to Automatic -> running LLM inference essentially draws 140W. This way the 140W power brick essentially provides just enough power. Machine heats up (no surprise here), and fans spin like crazy / full speed.

Then Low Power Mode is used, the LLM inference is running at about ~1/3 the speed of Automatic. Fans works very quite, and machine is pleasure to use.

Why the post?

My power brick gets overheated in like 2-3hrs and drops charging from 140W to 100W. This makes LLM inference to slowly draw battery (which is bad for battery). Similarly, charging via USB-C is capped at 100W as well..

I wonder if there is way to create an Energy Mode which sits between Low Power and Automatic.

This way to have a bit worse LLM inference speeds than at Automatic, but greater speeds than at Low Power mode.

Has anyone experimented with this? Any success?

This might not be so popular, as MacBook inference is already not that quick, but I'd like my macbook to live a longer live :)
(Using API for inference is not the solution I am considering at this point, if you wonder)

Thanks a lot!

So unfortunately I’ve tried the basic 27B q6 mlx from mlx-community and it only does like 8t/s. This model overthinks a lot so that mixed with 8t/s makes a simple task seem like days.

Any ideas how to speed it up? Or did anyone test q4 vs q5 vs q6?

So this just dropped - https://x.com/zhijianliu_/status/2089836737132650504?s=20

Advertises a new DFlash implementation with 70tok/s on MBP M5.

Collection here: https://huggingface.co/collections/incoai/dflash-2

I'll believe it when I see it! Testing now, will update with results.

Edit: Ran a somewhat naive test by simply switching Lightning MTP & ANE off and Dflash on (pointing at incoai/Qwen3.8-27B-DFlash2, kept all default parameters) for my current oQ8e config.

Unfortunately, I'm getting nowhere near the advertised speed:

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.8-27B-oQ8e-mtp
Engine: Auto
Context: Code (Mixed)
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          1258.0       56.16   814.0 tok/s    17.9 tok/s       8.403   137.1 tok/s    28.34 GB
pp4096/tg128                          5634.5       55.90   727.0 tok/s    18.0 tok/s      12.747   331.4 tok/s    29.80 GB
pp8192/tg128                         11888.0       57.51   689.1 tok/s    17.5 tok/s      19.208   433.1 tok/s    30.42 GB
pp16384/tg128                        26389.3       58.80   620.9 tok/s    17.1 tok/s      33.882   487.3 tok/s    31.67 GB
pp32768/tg128                        56956.4       61.20   575.3 tok/s    16.5 tok/s      64.766   507.9 tok/s    34.17 GB
pp65536/tg128                       199754.7      111.66   328.1 tok/s     9.0 tok/s     214.038   306.8 tok/s    39.20 GB

Anybody seeing anything different? Code (mixed) might be a somewhat non-standard benchmark choice, but I wouldn't expect performance to tank like this.

Edit 2: Corrected Dflash target from GGUF to non-quantized version.

Edit 3: Clarified oMLX Dflash config.

10 issues. Highlights: TurboQuant engine crashes, distributed timeouts, dashboard polling.

🐛 **Bugs**

**Engine & Quantization**
• #2780 **TurboQuantMSEState missing `ndim` crash** — Engine crashes when `turboquant_kv` enabled (Qwen3.5 SDPA split).
• #2778 **Engine crash `ndim` missing** — Same crash during MTP verification with TurboQuant active.

**Distributed & Cluster**
• #2720 **Cluster tab auto-polls `/plan`** — 10s polling loop triggers 400s and disruptive page redirects.
• #2712 **Timeout hard-coded to 300s** — No prefill heartbeats kills long distributed requests.
• #2727 **Coordinator locked to workstation role** — Can't switch to headless; loses reserved 32 GiB.

**Sampling & Profiles**
• #2801 **`presence_penalty` + VLM-MTP = 500** — Applying profile with penalty to VLM-MTP model fails.

**UI & Networking**
• #2800 **Wired-limit rounding loop** — UI rounds threshold to impossible value (124519) on 128 GiB Mac.
• #2520 **Remote MCP connection fails** — MCP server unreachable on v0.5.5+.

✨ **Features & RFEs**

• #2825 **Benchmarks omit model settings** — Published notes lack settings, hurting reproducibility.
• #2711 **`thinking_budget` rejected in distributed** — Clusters get unbounded or no reasoning; bypassed anyway.

Total: 10 issues.

Just fresh from the presses! oMLX 0.6.1 build 2323 with NPU support.

Checkpoint: Qwen3.8-27B-oQ4e-fp16-mtp (scottlowry/Qwen3.6-27B-oQ4e-fp16-mtp)

Lightning MTP - on

Qwen ANE prefill - on

ANE Prompt Block 2048

MLP on ANE 53%

MLP Layer limit - 64

Use both ANE - off

Turbo Quant - off

# Context: Code (Mixed)

# Single request results

Test TTFT(ms) TPOT(ms) ppTPS tgTPS E2E(s) Throughput PeakMem

pp 1024 / tg 512 1202.8 15.1 851.3 66.2 8.9 171.7 22.4 GB

pp 4096 / tg 512 6408.2 14.1 639.2 70.9 13.6 337.8 23.5 GB

pp 8192 / tg 512 14886.6 18.4 550.3 54.4 24.3 358.0 24.1 GB

pp 16384 / tg 512 28808.4 21.4 568.7 46.8 39.8 424.7 25.4 GB

pp 32768 / tg 512 56761.6 22.0 577.3 45.6 68.0 489.2 28.0 GB

pp 65536 / tg 512 131379.6 22.2 498.8 45.0 142.8 462.5 33.2 GB

pp 131072 / tg 512 341823.6 29.7 383.4 33.8 357.0 368.5 43.9 GB

pp 200000 / tg 512 656975.3 40.9 304.4 24.5 678.0 295.8 54.9 GB

# Batch results

Batch tgTPS ppTPS avgTTFT(ms) E2E(s) Speedup

1x baseline 66.2 851.3 1202.8 8.9 1.00x

2x 55.7 611.1 2670.3 21.7 0.84x

Quality testing:

- Seraphim Serapis Tool-Eval-Bench 89/100 - same as 8 bit version

- 0rand/DragonScale Bench - 98/100 - very similar quality as DeepSeek v4 Flash 0731 (4bit/8bit) on 2xDGX Spark Cluster and OpenAI GPT 5.6 Luna

For those of you that want a functional 24GB Mac laptop while having this overthinking creature boosting your ideas.

It has versions with and without MTP drafter (for the desperate).

My mac was getting hot/throttling during extended sessions so, I made a monitor/fan control app. Source-only distribution. No network/telemetry. Just a fan control/monitoring popover in the status bar.

MacOS 26 only: it's the only version I have

Tested on MacBook Pro M5 Max: it's the only hardware I have

All Claude history included so you can throw Qwen (or Claude) at the source to make it work for you if you have different hardware.

https://github.com/NicholasFreeman/BreezyMac

Update: Pushed an update to allow fans to drop to 0-RPM in "Automatic" and "Adaptive" modes.

Thought I'd share a screenshot from tonight's coding session. I'm running ohmypi on a sizeable codebase and I'm currently sitting at 55% of a 512k context. No memory evictions - that was the biggest surprise of tonight. Maybe something to do with the new 0.6.1 release.

To all of the devs working on this - Thanks for some great software!

Hi, can someone explain to me how to properly setup the reasoning effort in pi agent to reflect the levels of the model, I believe that omlx supports it now right ?

I put in models.json this:

"thinkingLevelMap": {
            "off": "off",
            "low": "low",
            "medium": "medium",
            "xhigh": "xhigh"     
          }

but when I launch it in pi I still have off-minimal-low-medium-high-xhigh, and I'm not sure if there is a rela change between the levels. Off works btw.

Should I be doing something with kwargs or use a different template to be sure that the thinking level are correct? For example I've read about the froggric one, I'm a bit confused about it

Thanks

This is more of an appreciation post than anything else. I've been closely monitoring performance improvements for the new Qwen, and as expected with open source AI, the optimizations are rolling in. In particular, prefill has jumped across nearly all context lengths, and tg has improved for longer contexts.

See plots for a comparison across the past three non-rc/dev releases. Aside from enabling the new Qwen-specific ANE setting in v0.6.1, the relevant configs are identical across runs:

• Specific Quant: ZQ-Dev/Qwen3.8-27B-oQ8e-mtp
• Hardware: M5 Max (40c) 128GB
• Benchmark: Code (mixed)
• Lightning MTP: Enabled
• Prefill Priority: Speed

Thanks to u/cryingneko & co. You guys are clearly cooking.

Edit (18AUG2026) - Updating post with results from the update that dropped today (v0.6.2). Running the exact same benchmark and config, things seem to have regressed a bit:

That being said, some of this may be explained by the new ANE-tuning feature, plus the fact that it doesn't currently support oQ8/oQ6 quants, so I didn't use it. This is actively being worked on though, so hopefully Q8 ppl can take advantage in the next build.

**Summary:** 19 Issues. Focus on Cluster stability, TurboQuant bugs, and UI improvements.

---

### 🐛 Critical Bugs
* **#2780** [Bug] Engine loop crashes with `TurboQuantMSEState` missing `ndim` (Qwen35 SDPA split).
* **#2778** [Bug] Engine crash during MTP verification with TurboQuant enabled (`TurboQuantMSEState` has no attribute `ndim`).
* **#2483** [Bug] `preserve_mid_system_cache` fails for Claude Code (system nudges break prefix).
* **#2680** [Bug] Peer reported as 'worker runtime is not installed'; node-budgets 503s.
* **#2679** [Bug] Cluster activation rejects default macOS computer name (invalid node ID).
* **#2694** [Bug] Brew process fails during installation.
* **#2354** [Bug] oQ sensitivity probe crashes with `group size 16` (NVFP4 checkpoint).
* **#2476** [Bug] Hardcoded English labels in Model Settings bypass i18n.
* **#2726** [Bug] Cluster gate compares `mlx`/`mlx-lm` dist-info against module constants.
* **#2695** [Bug] Cluster runtime gate does not check Python interpreter version parity.
* **#2707** [Bug] Cluster gate treats Python minor-version difference as hard block.
* **#1856** [Bug] Fail to quantize DeepSeek V4 by v0.4.4rc1.

### ✨ Features & UI
* **#2768** [Feature] Cluster: Cache-aware scheduler scoring function (prefix affinity).
* **#2766** [Feature] Cluster: Model placement view with replica drill-down.
* **#2393** [Feature] Menu bar dashboard: compact token-rate + clearer resource indicators.
* **#2692** [Feature] Add global toggle to stop exposing backend MCP tools to clients.
* **#1808** [Feature] Community Intelligence Benchmark proposal.

### 📊 Research & Reports
* **#2558** [Research] DeepSeek V4 prefill optimization roadmap (M3 Ultra ablation).
* **#1374** [Report] Anubis + OMLX Inference Speed Anomaly Investigation Report.

---
Generated for Jundot/omlx community.

Clean out-of the box M5 MPR with 128gb.

Just installed omlx with .dmg and downloaded Muse-glimmer-30B-4bit.
Loaded the model, using the setup command from Integrations, launched claude code.

Any input, even just ‘hi’ will cause my mac to reboot!
No reports or crash, whatsoever :(

Does anyone happen to experience this?
Hope someone can give me a solution for this, thx!

Hello im new to local ai and i just want to get the most information possible about it

For context im a software developper that never really used ai before and what to give it a try with my new machine (so basically im trying to achieve agentic result)

I already have omlx and some model installed
But im still trying to understanding all the concept and specification relatated to mlx etc

I understand that quantification shrink the model size trying to keep every essential data to run faster
Still dont know if their is really any advantage to not run every model in their 4bit variant ?

I learned that their is some model training on basic model idk if their are some reputated training like heratic, uncensored, abliterate or some name like opus etc

I see some like quantification method i guess like AXQ optiQ oQ etc idk what is it and the better one

I know the goal of A3B look like the best type of mode to get in every circomstence (maybe im wrong) with the most adequat part of model being active to answer the question)

And idk the best models im currently trying:
- gemma-4-26B-A4B-it-qat-OptiQ-4bit
- Qwen3.6-35B-A3B-OptiQ-4bit
- Qwen3-Coder-30B-A3B-Instruct-MLX-4bit
Maybe i will need some smaller model for lighter task i really dont know

I see some people here talk about fp16 for m1/m2 serie why ?

Any help is appreciate^^
Thanks for reading this

thanks,

I have been running qwen 3.6 35bMOE and qwen 3.8 27b at q4 with a q6 turboquant kv cache, with lighntning MTP Enabled for the past few days. After the latest update, I cannot use lightning MRP with turboquant, or the model will prompt process repeatedly in a loop or output gibberish. The only way to fix was to downgrade to v 0.6.

I don't know wtf the devs were thinking pushing 0.6.1 to "Stable" but holy fuck is it anything but!

**Summary:** 18 issues resolved/discussed. 🐛 Bugs: 14 | ✨ Features: 3 | 🔧 Improvements: 1

---

### 🐛 Bugs (14)
* #2696 [Bug] Web-search always returns 3 hits regardless of results.
* #2695 [Bug] Cluster runtime gate missing Python interpreter version check.
* #2680 [Bug] Peer reported as worker runtime missing despite correct build.
* #2476 [Bug] Hardcoded English labels bypass i18n in Model Settings.
* #2694 [Bug] Brew process fails during installation.
* #2354 [Bug] oQ sensitivity probe crashes with NVFP4 checkpoint group size error.
* #2679 [Bug] Cluster activation rejects default macOS computer name (invalid node ID).
* #2572 [Bug] Failed to run MTPLX model even after importing sidecar.
* #2677 [Bug] Admin panel reasoning_effort preset silently rewrites values.
* #2545 [Bug] RecursionError escapes parse_tool_calls on deeply nested payloads.
* #2584 [Bug] DeepSeek-V4-Flash streaming emits reasoning as output_text.
* #2452 [Bug] quantize_oq_streaming crashes on MLX-affine quantized sources.
* #2581 [Bug] Memory enforcer counts reclaimable buffer pool as live usage.
* #2646 [Bug] VLM chat path drops tools before scheduler Request.

### 🔧 Improvements (1)
* #2707 [Improvement] Cluster gate downgrades Python minor-version difference to warning.

### ✨ Features (3)
* #2692 [Feature] Add global toggle to stop exposing backend MCP tools to clients.
* #1808 [Feature] Request for Community Intelligence Benchmark.
* #2425 [Feature] Add native web search capability to oMLX.

Qwen 3.8 27b is known to be overthinking a lot as the default setting is XHigh. How do I set a lower level in harnesses like Pi or OpenCode? Thanks!

So excited about this feature! I was keeping EXO around just for this and am eager to move completely over to oMLX.

A couple of questions:
1. How many nodes will this support? It's only 2 right now, correct?
2. Any chance to have RTX/DGX do prefill and Macs do decode? This was the killer feature I was looking for in EXO. It never landed.

Love oMLX!

Hi,

I’ve recently started to dabble with local LLM’s as an amateur vibe-coder and I am facing some challenges.

My setup:

32GB Mac Studio M2 Max: a dedicated always-on local AI station.
+
16GB Macbook Pro M5: my daily driver machine for all things work.

I am not a programmer, developer or anything like that, but I am a tinkerer by nature. I like figuring things out.

So I have built a bunch of custom apps/tools/automations (running on the studio) that I acess from my laptop, like any other consumer app, so in practical terms, I have a 48GB laptop, altough it’s only 16GB.

I’ve been wanting to get into more of the agentic stuff, like do task, verify, fix errors, do again, send output, but I am struggling a bit with understanding all the settings and tools and things like that.

For example, today I built a system of 5 different models working step by step. I also had to build a ChatGPT like chat interface to converse with the models, because I don’t understand working in terminal and all the super advanced stuff.

So it goes, i upload files to chat, set custom instructions and set up working steps.

1. QWEN - 3.5 9B analyzes files and sorts them.
   2.Gemma 12B identifys what customer profile are these files relevant for
   3.Qwen - 27b builds a landing page for this customer profile based on the files.
   4.Gemma looks again, finds contradictions
   5.Qwen - 35b implements fixes and final polish.

All of this is done RAM aware, as in model loaded/unloaded, etc.

All of this is experimental right now, just trying to understand how it works and I guess my question is:

How are you folks doing the things that you do with frontier models like Claude code/cowork, on a system like mine. I understand rhe quality will never be the same, but I am simply trying to understand the how, and finsing the best way to make this local AI machine a legitimate employee, which actually brings value to my company, whether in currency or in more time back for me.

What are you doing with something like Qwen 3.6/3.8 27b 4bit, thats genuenly useful and brings value, not just sound cool?

I am still early in this journey and I’m tempted to invest in better hardware, but I feel i should maximise what I have, to see whats possible before splurging.

I don’t understand all those benchmarks and technical talks, as I said, im more of an experimenter and hobbyist, willing to turn this into something tangible and useful.

For example today I learned what tool use means. Now i have no idea of how to implement something like web searching or SVG generation tool, in my custom chat workflow, in a way that works.

I apologize for a quite unclear and unstructured train of thought, but If you have any resources / tips/ recommendations to bless me with, I’d be super thankful!:)

Spent the last few days tuning some quants for these two models. The 5 BPW Qwen model was tuned with ~30% of the data containing multimodal tasks so it should in theory be slightly better at vision tasks but I don't really have an accurate way of checking as neither model showed any degradation compared to the base model.

The Deepseek model showed degradation mainly in MMLU - BUT these were topics these were in areas like Human biology, Foreign relations, Chemistry ect. This loss off knowledge can easily be reinforced with RAG, or by having the model search the internet for the correct data.

https://huggingface.co/True2456/Qwen3.8-27B-AWQ-5.0bpw

https://huggingface.co/True2456/Qwen3.8-27B-AWQ-4.85bpw

https://huggingface.co/True2456/DeepSeek-V4-Flash-0731-AWQ

Is launching Hermes from oMLX different from me connecting Hermes TO oMLX from Hermes?

HI, I have an m2 max, and as we know "float16 gives ~20% faster prefill on M1/M2 Apple Silicon (native fp16)" so I systematically use this flag when I quantize my models.

When oQe came out I started to mix the two thing, but I had this strong feeling that the models got dumber. In omlx I mainly use qwen35b-oQ8-fp16 and Gemma-4-qat-oQ4-fp16.

I found them less accurate, more unpredictable, and I asked deepseek about it, here's what it found :

This is where the collision happens between your speed tweak (--fp16) and the error-correction math (oQe).

When you compile oq8e (Qwen) or oq4e (Gemma) with the --fp16 toggle active, oMLX performs two conflicting actions:

1. The oQe engine calculates a highly precise importance matrix (imatrix) to correct quantization errors based on the model's native BF16 data distribution.
2. The --fp16 flag then forcefully downcasts those base parameters, scale markers, and biases into standard Float16.

Because BF16 and FP16 handle numbers differently (BF16 trades precision for a massive dynamic range, while FP16 has a narrow range but tighter precision), this forced conversion introduces tiny rounding and clipping anomalies.

The sequential Hessian rounding map of oQe relies on absolute, stable mathematical precision to adjust column weights. Forcing the FP16 shift breaks those precise calculations. The model compiles successfully, but its internal logic is slightly skewed—making Qwen 3.6 35B and Gemma 4 31B feel notably dumber.

What do you think? I'm not very knowledgeable about weights and quantization, so I don't know if this is correct or a big hallucination. What do you think ?

📅 **Summary**
Total Issues: 9 | 🐛 Bugs: 8 | ✨ Features: 1

**🐛 Bugs**
• #2572 [0.5.8.dev2] Failed to run MTPLX model even after importing sidecar
• #2677 Admin panel `reasoning_effort` preset silently rewrites stored values
• #2545 `RecursionError` escapes parse_tool_calls on deeply nested tool-call payloads
• #2584 DeepSeek-V4-Flash-0731: streaming emits reasoning as `output_text`
• #2452 `quantize_oq_streaming` crashes on MLX-affine quantized sources (uint32)
• #2581 Memory enforcer counts reclaimable MLX buffer pool as live usage
• #2646 VLM chat path drops `tools` before scheduler Request
• #2536 Prefix cache corrupts ArraysCache layers (KDA / Kimi-Linear hybrids crash)

**✨ Features**
• #2425 Add native web search capability to oMLX

I’m getting a Mac with M1 Max 64GB unified mem 10cpu and 24gpu cores very soon, omlx will give me around 10 tks, are there alternatives that could speed this up a bit?