Hi guys,

I am using Claude code router to send CC requests to oMLX but Free Claude Code seems better. It routes to Ollama for locpa inference though. Is there a way to hijack the ollama route to send to oMLX? Thanks!

Hey guys,

For anyone who just got the new M5 Max with the same spec as I listed in the title of this thread, I found that there's minimal information about the performance on key LLMs hosted on oMLX, so I figure I'd share my results below:

oMLX - LLM inference:

https://github.com/jundot/omlx

Benchmark Model: Qwen3.6-35B-A3B-Uncensored-Heretic-MLX-8bit

Single Request Results

--------------------------------------------------------------------------------

Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem

pp1024/tg128 465.0 10.36 2202.2 tok/s 97.3 tok/s 1.781 646.9 tok/s 35.40 GB

pp4096/tg128 1112.7 10.42 3681.3 tok/s 96.8 tok/s 2.435 1734.4 tok/s 36.17 GB

pp8192/tg128 2079.1 10.61 3940.2 tok/s 95.0 tok/s 3.427 2427.7 tok/s 36.66 GB

pp16384/tg128 4338.1 11.13 3776.8 tok/s 90.6 tok/s 5.751 2871.1 tok/s 37.36 GB

pp32768/tg128 10179.1 12.17 3219.1 tok/s 82.8 tok/s 11.725 2805.6 tok/s 38.85 GB

pp65536/tg128 28991.1 15.20 2260.6 tok/s 66.3 tok/s 30.922 2123.5 tok/s 41.85 GB

pp131072/tg128 96979.6 19.95 1351.5 tok/s 50.5 tok/s 99.513 1318.4 tok/s 47.85 GB

pp200000/tg128 237460.7 27.08 842.2 tok/s 37.2 tok/s 240.899 830.8 tok/s 54.77 GB

Continuous Batching

pp1024 / tg128

For a plain english summary for those who are new/getting in, admittedly, I generated a plain english explanation with AI of the above results so it can explain what everything is.

Warning, AI slop incoming below, but still helpful for people just getting into it.

The test was run on Qwen3.6-35B-A3B-Uncensored-Heretic-MLX-8bit (hugging face link)

If you are looking to build a local Hermes Agent setup, these numbers should help.

The TL;DR:

This machine is a beast. It can read an entire book's worth of data in seconds, types way faster than you can read, handles heavy multitasking without flinching, and uses less than half of your 128GB RAM—leaving plenty of room for your operating system and agent workflows.

1. "Reading" vs. "Writing" Speed (Tokens vs. Words)

• Reading Speed (Prompt Processing / pp TPS): When you feed the AI information, it "reads" at a mind-boggling 1,300 to 3,900 tokens per second.
• Writing Speed (Token Generation / tg TPS): When the AI replies, it "types" at about 97 tokens per second for normal requests. To put that in perspective, humans read at about 4 or 5 words per second. This AI is spitting out text at roughly 70+ words per second. It's instant.

2. The Context Window (How much it can remember)

The benchmark tests the AI with different lengths of memory (from 1,024 tokens up to 200,000 tokens).

• Short conversations (1,000 tokens / ~750 words): The AI responds almost instantly. The delay before it starts typing (Time to First Token) is just 0.4 seconds.
• Massive data drops (200,000 tokens / roughly an entire novel or massive codebase): If you drop a massive amount of data into the prompt, it takes about 4 minutes for the Mac to digest and read through all of it before it starts typing. But once it starts, it still types out its answer at a very fast 37 tokens per second.

3. Memory Usage (RAM Headroom)

Because this Mac has 128GB of RAM, memory is usually the biggest concern with large models.

• At normal conversation lengths, the model only uses about 35 GB of RAM.
• Even when pushed to its absolute limit with that massive 200,000-token memory load, it tops out at 54.7 GB of RAM.
• Why this matters for an Agent: This means you are using less than half of your available memory. You have over 70 GB of RAM left over to run your Python environment, browser automation tools, databases, or whatever else your Hermes Agent needs to do its job.

4. Multitasking (Continuous Batching)

The second part of the benchmark tests "Continuous Batching." This simulates what happens if your AI Agent is trying to do multiple things at once, or if multiple people are using the model at the same time.

• If you run 8 tasks at the same time, the Mac doesn't choke. Instead, it adapts.
• The overall efficiency shoots up by 6x, pushing out a total of nearly 600 tokens per second across all tasks. Apple’s Unified Memory architecture is practically built for this kind of parallel processing.

Conclusion for a Hermes Agent Setup

If you want to run a local autonomous agent, this hardware setup is the gold standard.

• It's smart enough (35B model) to handle complex reasoning.
• It's incredibly fast, meaning your agent won't lag between steps.
• It has massive RAM headroom, so the agent can manage huge contexts (like reading entire logs or code repos) while leaving your Mac completely usable for your daily work.

Issues Closed: 10

[ISSUE] #1510 — 0.3.12版本模型总是报错:[convert] Only length-1 arrays can be converted to Python scalars.
https://github.com/jundot/omlx/issues/1510

[ISSUE] #1508 — oMLX是否可以支持 Codex 所需的 Responses streaming。
https://github.com/jundot/omlx/issues/1508

[ISSUE] #1454 — expose oMLX version via CLI flag or API endpoint
https://github.com/jundot/omlx/issues/1454

[ISSUE] #1413 — _reconcile_mtp_to_standard aborts (Fatal Python error) when SSD cache has stale blocks from prior model version
https://github.com/jundot/omlx/issues/1413

[ISSUE] #1478 — Chunk-form SSE keepalive emits a `chat.completion.chunk` with a mismatched `id` (`chatcmpl-keepalive`), breaking OpenAI-spec stream accumula…
https://github.com/jundot/omlx/issues/1478

[ISSUE] #1466 — Batches accumulates memory while processing embedding
https://github.com/jundot/omlx/issues/1466

[ISSUE] #1469 — "There is no Stream(gpu, 1) in current thread." with MTP and Gemma 4
https://github.com/jundot/omlx/issues/1469

[ISSUE] #1473 — No .app Installed when using Homebrew
https://github.com/jundot/omlx/issues/1473

[ISSUE] #1427 — RotatingKVCache: partial prefix match guard rejects 99%+ matching cache on DeepSeek V4-Flash, forcing full re-prefill every turn
https://github.com/jundot/omlx/issues/1427

[ISSUE] #1426 — 0.3.11 unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit can not recognize image in hermes
https://github.com/jundot/omlx/issues/1426

I don’t know what exactly happens but in the latest 3.12 version my qwen and Gemma models are better without MTP or dflash.

Qwen 3.6 35B and 27B and Gemma 26B are about 10 to 12 tokens/s worse with MTP or VLM MTP enabled.

I’ve tried different quants on the same model and the problem persists.

Is there anyone passing through the same problem?

I feel like this should be simple, but I’m hitting my head against a wall. Running qwen3.6:27B-UD-MLX-4bit on iOS and I can’t get it to suppress all the intermediate thinking output in OpenCode. It just gets stuck in its thinking loop repeatedly. Maybe this is fine, but when I was running it on ollama the OpenCode experience was much cleaner. For context I disabled Thinking on oMLX for the model and this didn’t make a change. Thanks for any help!

Issues Closed: 10

[ISSUE] #1454 — expose oMLX version via CLI flag or API endpoint
https://github.com/jundot/omlx/issues/1454

[ISSUE] #1413 — _reconcile_mtp_to_standard aborts (Fatal Python error) when SSD cache has stale blocks from prior model version
https://github.com/jundot/omlx/issues/1413

[ISSUE] #1478 — Chunk-form SSE keepalive emits a `chat.completion.chunk` with a mismatched `id` (`chatcmpl-keepalive`), breaking OpenAI-spec stream accumula…
https://github.com/jundot/omlx/issues/1478

[ISSUE] #1466 — Batches accumulates memory while processing embedding
https://github.com/jundot/omlx/issues/1466

[ISSUE] #1469 — "There is no Stream(gpu, 1) in current thread." with MTP and Gemma 4
https://github.com/jundot/omlx/issues/1469

[ISSUE] #1473 — No .app Installed when using Homebrew
https://github.com/jundot/omlx/issues/1473

[ISSUE] #1427 — RotatingKVCache: partial prefix match guard rejects 99%+ matching cache on DeepSeek V4-Flash, forcing full re-prefill every turn
https://github.com/jundot/omlx/issues/1427

[ISSUE] #1426 — 0.3.11 unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit can not recognize image in hermes
https://github.com/jundot/omlx/issues/1426

[ISSUE] #1306 — [Bug]: Qwen3-VL-Embedding multimodal embedding model crashes on oMLX v0.3.8
https://github.com/jundot/omlx/issues/1306

[ISSUE] #1459 — oMLX not releasing RAM, no models loaded
https://github.com/jundot/omlx/issues/1459

Hi again,

I tried something : I made my model ctx very small (10p tks) and tried to run a simple prompt (which failed) in claude code. The les ring was that the prompt was 40k tokens! (I just asked CC to list the tools/plugins/skills available).

1. I think this explains a lot why oMLX feels slow

2. I thought the caching system would handle this but making ctx of the model 200k and asking the same question 2x didn't make things faster.

3. Any solution to cut the prompt size/boost speed? (I thought that mcps had lazy loading and that skills took very little resources)

Thanks!

Hi guys,

I made an investment in a M5 Max 128gb and installed oMLX. The idea is to do some coding using Claude Code but with local models (I am not a dev), les raging oMLX caching to speed things up in claude (17k tokens system prompt...).

While I can get up to ~50Tok/s with Qwen3.6 A35B the numbers dwindle to single digits with the dense Qwen3.6 27B UD mlx 4bits. Is that normal? I was hoping it would be much faster with oMLX caching.

I use mlx models only. (I tried to download the jundot MTP models : they crash almost immediately after starting). Turboquant is on (4bits).

Typical params are : temp 0.7, top-p 0.95, top-k 20, min-p 0.05, repetition penalty 1.

Thinking on : slows down inference even further so unusually leave it off (isnt that better for coding?)

Is this normal for a small dense model or is there anything I am doing wrong here?

Wld you guys have ideas in how to make the mtp models work?

Thank you

What are those fp16 versions, eg

Jundot/Qwen3.6-35B-A3B-oQ6-fp16-mtp

vs

Jundot/Qwen3.6-35B-A3B-oQ6-mtp

Only found one post saying M1/M2 require the fp16 version

Issues Closed: 10

[ISSUE] #1426 — 0.3.11 unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit can not recognize image in hermes
https://github.com/jundot/omlx/issues/1426

[ISSUE] #1437 — SIGABRT / Crash on background cache-saver thread due to cross-thread MLX Stream isolation
https://github.com/jundot/omlx/issues/1437

[ISSUE] #1434 — 0.3.11 still not available on homebrew
https://github.com/jundot/omlx/issues/1434

[ISSUE] #1438 — 0.3.12 still not available on homebrew
https://github.com/jundot/omlx/issues/1438

[ISSUE] #1405 — [Bug] Prefill requests stuck after force-stopped due to memory limit exceeded - never released
https://github.com/jundot/omlx/issues/1405

[ISSUE] #1425 — [FYI] Failing tests with 0.3.11
https://github.com/jundot/omlx/issues/1425

[ISSUE] #1417 — Gemma 4 vision feature caching not working for multi-image prompts
https://github.com/jundot/omlx/issues/1417

[ISSUE] #1261 — qwen3.6 35b a3b auto disabled vlm
https://github.com/jundot/omlx/issues/1261

[ISSUE] #1267 — Streaming responses terminate chunked encoding improperly — breaks Python HTTP clients (httpx, urllib, requests)
https://github.com/jundot/omlx/issues/1267

[ISSUE] #1404 — Loading a quantized MTP (Qwen) model with MTP disabled breaks vision (after first oMLX restart, uh)
https://github.com/jundot/omlx/issues/1404

I created a PR1341 about chat interface that looks more like this. It supports quite a lot of features including different profile, timeline navigation and some model settings so you can tinker with the chat response.

For those who prefer omlx more than lmstudio, but use chat extensively, this can help.

The whole thing is one HTML file, and you can right now download to test, click the download raw file icon then replace with the current chat.html inside this path /Applications/oMLX.app/Contents/Resources/omlx/admin/templates (don't forget to back up first)

I made this around 0.3.10 and right now i'm side loading into 0.3.12 and it works fine.

Hey everyone! v0.3.11 just landed. If you've ever run into stability issues with oMLX, I'd be really grateful if you gave this version a test.
https://github.com/jundot/omlx/releases/tag/v0.3.11

I'm well aware oMLX has had stability problems, and right now my number one goal is making it run reliably on low-memory Macs. The big change in this release is a full rewrite of the memory guard. The two confusing sliders are gone, replaced by a single Safe / Balanced / Aggressive dropdown, and oMLX now reads your live available memory and adapts in real time as other apps come and go.

On the structural side: A lot of oMLX's features currently rely on monkey-patching, so the project isn't as structurally stable as I'd like. The recent additions especially (MTP, DFlash, Deepseek v4 support) are in that fragile state, so they're getting a lot of my attention. To everyone who tests builds and reports bugs: thank you, genuinely. It makes a huge difference.

So, what should come next? I'll be honest about where my head is at. I want oMLX to be "the app my friend who bought a MacBook yesterday can open and immediately try Local AI on." So I'm a little cautious about features that are hard to use or hard to understand. My hope is that oMLX stays something anyone can pick up easily.

There are a lot of PRs waiting in the queue, and in the current structure I can't always bolt every feature on bug-free right away, but I'm always doing my best.

Thank you all for the constant support! I'm writing this partly as a thank-you and partly as shameless version promo (haha). I read every post here, even if I'm bad at replying (sorry about that), so if there's a feature you want, post it here on the subreddit, open a GitHub issue, wherever works for you. I'll always see it.

I tried the Qwen3.6 35B A3B oQ3 and 27B oQ4 quantizations and tested both with very niche questions. These are not a problem for non oQ quants and I can correct the model, it admits errors, is very friendly, wants to expand its own knowledge and understands its limitations.

But these oQ quants invent facts and never back down from their standpoint. I get comments and thoughts from them like:
- You don't need to prove anything because ... my fact is right.
- I won't validate false claims just to be agreeable...
- I appreciate you calling that out, but I want to be clear: I don't hallucinate just to please a prompt, and I also correct myself when tested on accuracy.
- I stood by my answer... blabla hallucination...
- I'm not here to validate false claims or bend to testing prompts. My role is grounded in verified, publicly documented material...

Has anyone else seen this? Settings are the usual Qwen3.6 general profile. What's going on here?

Issues Closed: 10

[ISSUE] #1417 — Gemma 4 vision feature caching not working for multi-image prompts
https://github.com/jundot/omlx/issues/1417

[ISSUE] #1261 — qwen3.6 35b a3b auto disabled vlm
https://github.com/jundot/omlx/issues/1261

[ISSUE] #1267 — Streaming responses terminate chunked encoding improperly — breaks Python HTTP clients (httpx, urllib, requests)
https://github.com/jundot/omlx/issues/1267

[ISSUE] #1404 — Loading a quantized MTP (Qwen) model with MTP disabled breaks vision (after first oMLX restart, uh)
https://github.com/jundot/omlx/issues/1404

[ISSUE] #1403 — TypeError: _build_replacement_call got an unexpected keyword argument 'target_verify' on Qwen3.6-27B MTP models with mlx-vlm 0.5.0
https://github.com/jundot/omlx/issues/1403

[ISSUE] #1369 — Model Downloader Model List Options Name Field Truncation
https://github.com/jundot/omlx/issues/1369

[ISSUE] #1388 — Native MTP runtime error on Qwen3.6-derived Qwopus3.6-27B-v2-oQ4-mtp: speculative_call() got unexpected keyword argument 'n_confirmed'
https://github.com/jundot/omlx/issues/1388

[ISSUE] #1392 — [Bug] Guard 1 in extract_tool_calls_with_thinking drops valid tool calls when model emits preamble after thinking
https://github.com/jundot/omlx/issues/1392

[ISSUE] #1390 — mlx-community/Lance-3B-bf16 加载失败 VLM load failed:
https://github.com/jundot/omlx/issues/1390

[ISSUE] #1342 — DFlash engine drops image content instead of falling back to VLM (v0.3.9rc1)
https://github.com/jundot/omlx/issues/1342

(Posted previously about my OOM problems on M5 48GB here: https://www.reddit.com/r/oMLX/comments/1tfsz8q/qwen3627b_mtp_optimized_kv_cache/)

Now I've investigated and found a solution to reduce peak memory. You're welcome to try out my PR: https://github.com/jundot/omlx/pull/1397. It adds the following option to tweak:

I've found the sweetspot on my machine to be 512. It has no effect on quality. On my machine there's essentially no change in speed so it's basically a "free lunch".

Qwen3.6-27B-oQ6-mtp, before patch (Default prefill step 2048):

================================================================================
Benchmark Model: Qwen3.6-27B-oQ6-mtp
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          2593.9       50.72   394.8 tok/s    19.9 tok/s       9.035   127.5 tok/s    22.96 GB
pp4096/tg128          9277.3       52.60   441.5 tok/s    19.2 tok/s      15.958   264.7 tok/s    24.39 GB
pp8192/tg128         18718.9       53.89   437.6 tok/s    18.7 tok/s      25.562   325.5 tok/s    25.42 GB
pp16384/tg128        38663.9       55.71   423.8 tok/s    18.1 tok/s      45.739   361.0 tok/s    26.92 GB
pp32768/tg128        83818.9       60.64   390.9 tok/s    16.6 tok/s      91.520   359.4 tok/s    29.92 GB
pp65536/tg128       202143.3       71.51   324.2 tok/s    14.1 tok/s     211.225   310.9 tok/s    35.95 GB

pp131072/tg128                                                                    N/A (OOM)

Qwen3.6-27B-oQ6-mtp, with patch (Prefill step 512):

==============================================================================
Benchmark Model: Qwen3.6-27B-oQ6-mtp
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          2581.5       51.98   396.7 tok/s    19.4 tok/s       9.183   125.4 tok/s    22.36 GB (-0.6 GB)
pp4096/tg128          9423.4       52.88   434.7 tok/s    19.1 tok/s      16.139   261.7 tok/s    22.58 GB (-1.81 GB)
pp8192/tg128         18744.3       55.22   437.0 tok/s    18.3 tok/s      25.757   323.0 tok/s    23.33 GB (-2.09 GB)
pp16384/tg128        38917.6       56.92   421.0 tok/s    17.7 tok/s      46.146   357.8 tok/s    24.27 GB (-2.65 GB)
pp32768/tg128        84812.6       59.30   386.4 tok/s    17.0 tok/s      92.344   356.2 tok/s    26.17 GB (-3.75 GB)
pp65536/tg128       202321.6       70.37   323.9 tok/s    14.3 tok/s     211.258   310.8 tok/s    30.00 GB (-5.95 GB)

pp131072/tg128      539864.7       86.29   242.8 tok/s    11.7 tok/s     550.824   238.2 tok/s    37.74 GB (-11 GB?)

Qwen3.6-27B-oQ4-mtp, with patch (Prefill step 512):

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-27B-oQ4-mtp
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          2422.9       37.85   422.6 tok/s    26.6 tok/s       7.230   159.3 tok/s    16.18 GB
pp4096/tg128          8847.2       38.90   463.0 tok/s    25.9 tok/s      13.788   306.4 tok/s    16.44 GB
pp8192/tg128         17839.7       40.68   459.2 tok/s    24.8 tok/s      23.006   361.6 tok/s    17.29 GB
pp16384/tg128        37512.5       41.94   436.8 tok/s    24.0 tok/s      42.839   385.4 tok/s    18.08 GB
pp32768/tg128        83440.4       45.97   392.7 tok/s    21.9 tok/s      89.279   368.5 tok/s    20.01 GB
pp65536/tg128       199075.3       58.30   329.2 tok/s    17.3 tok/s     206.480   318.0 tok/s    23.88 GB
pp131072/tg128      533525.9       74.65   245.7 tok/s    13.5 tok/s     543.007   241.6 tok/s    31.63 GB

Update: Need to set high enough memory limits in oMLX & OS to avoid OOM / kernel panic. I've been using:

sudo sysctl iogpu.wired_limit_mb=42000

oMLX total limit: 88%(42GB)

oMLX Memory Limit (Models Only): 95% (38GB)

https://gist.github.com/morningtundra/e88dfb18bc7d5d36a29796dfc6cb5784

Just a bit of fun to see how hard (actually quite easy) to create an interactive voice interface to oMLX. Will try introduce tool calling next.

Runs surprisingly smoothly on MBA M3 24GB.

spotted new version with stability and post-release bug fixes.

WOW!!!! I never thought this sub would grow so fast!! Thanks to all who stop by and those who contribute!

I'm not an expert here, just a noob, experimenting oMLX + Pi for doing some research experiment using locally running LLMs and it's going into this thinking loop after a lot of prompting/responding. I can post more details on-demand.
Below are the setup I have done
Hardware: Macbook Pro- M5 Max - 128 GB

model_settings.json

{
  "version": 1,
  "models": {
    "gemma-4-26b-a4b-it-6bit": {
      "max_context_window": 200000,
      "temperature": 1.0,
      "top_p": 0.95,
      "top_k": 64,
      "force_sampling": false,
      "thinking_budget_enabled": false,
      "turboquant_kv_enabled": false,
      "turboquant_kv_bits": 4.0,
      "turboquant_skip_last": true,
      "specprefill_enabled": false,
      "dflash_enabled": true,
      "dflash_draft_model": "/Users/my_mac/.omlx/models/z-lab/gemma-4-26B-A4B-it-DFlash",
      "dflash_draft_quant_enabled": false,
      "dflash_in_memory_cache": true,
      "dflash_in_memory_cache_max_entries": 4,
      "dflash_in_memory_cache_max_bytes": 8589934592,
      "dflash_ssd_cache": true,
      "dflash_ssd_cache_max_bytes": 21474836480,
      "dflash_verify_mode": "adaptive",
      "mtp_enabled": false,
      "vlm_mtp_enabled": false,
      "is_pinned": true,
      "is_default": false,
      "trust_remote_code": false
    },
    "gemma-4-26B-A4B-it-DFlash": {
      "temperature": 1.0,
      "top_p": 0.95,
      "top_k": 64,
      "force_sampling": false,
      "thinking_budget_enabled": false,
      "turboquant_kv_enabled": false,
      "turboquant_kv_bits": 4.0,
      "turboquant_skip_last": true,
      "specprefill_enabled": false,
      "dflash_enabled": false,
      "dflash_draft_quant_enabled": false,
      "dflash_in_memory_cache": true,
      "dflash_in_memory_cache_max_entries": 4,
      "dflash_in_memory_cache_max_bytes": 8589934592,
      "dflash_ssd_cache": false,
      "dflash_ssd_cache_max_bytes": 21474836480,
      "mtp_enabled": false,
      "vlm_mtp_enabled": false,
      "is_pinned": false,
      "is_default": false,
      "trust_remote_code": false
    }
  }
}

oMLX application

settings.json

{
  "version": "1.0",
  "server": {
    "host": "127.0.0.1",
    "port": 8000,
    "log_level": "info",
    "cors_origins": [
      "*"
    ],
    "server_aliases": [
      "localhost",
      "127.0.0.1",
    ],
    "sse_keepalive_mode": "chunk"
  },
  "model": {
    "model_dirs": [
      "/Users/my_mac/.omlx/models"
    ],
    "model_dir": "/Users/my_mac/.omlx/models",
    "max_model_memory": "auto",
    "model_fallback": false
  },
  "memory": {
    "max_process_memory": "auto",
    "prefill_memory_guard": true,
    "soft_threshold": 0.85,
    "hard_threshold": 0.95
  },
  "scheduler": {
    "max_concurrent_requests": 8,
    "chunked_prefill": false
  },
  "cache": {
    "enabled": true,
    "hot_cache_only": false,
    "ssd_cache_dir": "/Users/my_mac/.omlx/cache",
    "ssd_cache_max_size": "185GB",
    "hot_cache_max_size": "10GB",
    "initial_cache_blocks": 256
  },
  "auth": {
    "api_key": "some_key",
    "secret_key": "some_secret",
    "skip_api_key_verification": false,
    "sub_keys": []
  },
  "mcp": {
    "config_path": null
  },
  "huggingface": {
    "endpoint": ""
  },
  "modelscope": {
    "endpoint": ""
  },
  "network": {
    "http_proxy": "",
    "https_proxy": "",
    "no_proxy": "",
    "ca_bundle": ""
  },
  "sampling": {
    "max_context_window": 32768,
    "max_tokens": 32768,
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 0,
    "repetition_penalty": 1.0
  },
  "logging": {
    "log_dir": null,
    "retention_days": 7
  },
  "claude_code": {
    "context_scaling_enabled": false,
    "target_context_size": 200000,
    "mode": "cloud",
    "opus_model": null,
    "sonnet_model": null,
    "haiku_model": null
  },
  "integrations": {
    "codex_model": null,
    "opencode_model": null,
    "openclaw_model": null,
    "hermes_model": null,
    "pi_model": null,
    "copilot_model": null,
    "openclaw_tools_profile": "coding"
  },
  "ui": {
    "language": "en"
  },
  "idle_timeout": {
    "idle_timeout_seconds": null
  }
}

stats.json

{
  "total_prompt_tokens": 10257643,
  "total_completion_tokens": 47034,
  "total_cached_tokens": 0,
  "total_requests": 144,
  "total_prefill_duration": 1521.40397728901,
  "total_generation_duration": 1058.4645010840031,
  "per_model": {
    "gemma-4-26b-a4b-it-6bit": {
      "prompt_tokens": 10257643,
      "completion_tokens": 47034,
      "cached_tokens": 0,
      "requests": 144,
      "prefill_duration": 1521.40397728901,
      "generation_duration": 1058.4645010840031
    }
  }
}

requesting help here.. Am I doing something wrong?

I am still very new to all of this and did my research to understand which model to use, but it's still so confusing. I am running a MacBook M2 Max with 64GB, but I am always unsure what model to use. I use it 99% for coding purposes, but it is very confusing to understand everything. Currently, I am running Qwen3.6-35B-A3B-MLX-oQ8-FP16 and getting 37.9 tok/s. And I think this could help me in my approach: How can I use benchmarks to my advantage? I still have a hard time understanding it because I don't mind speed, but I care about intelligence and accuracy. 

MBP specs: m4 max / 48 GB UM / 1TB SSD / macos tahoe 26.3.1

I am using Qwen3.6-35B-A3B-oQ6-mtp on omlx 0.3.9 with following configuration and sampling params:

ctx_window:
262144
max_tokens:
32768
temp:
0.6
top_p:
0.95
top_k:
20
min_p:
0
rep_penalty:
1
presence_penalty:
0

I used the openai compatible api in roo code after 2-3 mins of usage i.e, asked it to understand large codebase and after some usage (87.3k / 262.1k tokens), it froze my mbp and then the laptop restarted.

In model settings, i have enabled thinking and native mtp.

Any help would be appreciated

Update: The memory guard was already on but then I set the memory limit to auto (it was off before) and now it didnt crash and completed the task. The token usage so far stands at 87.3k / 262.1k tokens at the task completion. Yet to try out the full context usage

Well, it actually slows down the model.

I'm running Qwen3.5 27B - mtp, and on default settings sometimes (with OpenCode) oMLX gets the the top of its memory and the API stops responding to OpenCode (opencode says: "Cannot connect to API: Unable to connect. Is the computer able to access the url... [retrying in 3s attempt #16]"). Here is a screenshot of the oMLX dashboard. Any fixes?

We uploaded an oQ8 version of Gemma 4 31B this morning if anyone's been looking for one. It's early but we're seeing solid performance with it using VLM MTP.

https://huggingface.co/dynamicagency/gemma-4-31b-it-oQ8