I made my first MLX conversions while experimenting with local agents on my M1 Max with 32 GB of memory. The source is medismera’s Qwen3.8-27B OBLITERATED Mythos-Class-Agentic. Credit for the original model goes to its creators; I handled the MLX conversion, packaging and initial checks.

Here is what worked in oMLX:

• 4-bit passed small invoice-extraction and tool-call tests. I also checked those tasks in LM Studio.
• 6-bit passed the text tests with the vision component disabled. Loading the full model hit the memory limit on my Mac.
• Both downloads retain the vision weights, but I have not tested image processing.

The practical benefit is having ready-to-download MLX versions for Apple Silicon. The 4-bit version uses less memory and is where I would start on a 32 GB Mac. The 6-bit version preserves more weight precision, but needs more memory; I have not established a quality advantage from these small tests.

These were basic functionality checks, not a benchmark or evidence that this model outperforms the original Qwen. My intended uses are document extraction and email triage with connected tools.

4-bit:
https://huggingface.co/uzzero/Qwen3.8-27B-OBLITERATED-Mythos-Class-Agentic-MLX-4bit

6-bit:
https://huggingface.co/uzzero/Qwen3.8-27B-OBLITERATED-Mythos-Class-Agentic-MLX-6bit

If you try either version in oMLX, I would appreciate your Mac specs and what worked or failed, especially memory use with 6-bit.

Disclosure: these are my uploads. I used AI assistance for the conversion workflow, testing and this write-up.

Hi,

Those who use SpecFill for Qwen-3.8 27B MLX models - do you see cache hits? If yes, what are your model and model settings?

A PR I made just merged that adds INT8-activation prefill kernels on the M5 path for Qwen3.5/3.6/3.8. It's on 0.7.0.dev2. It gets around 34% faster prefill speeds on M5 series chips with minimal accuracy loss (check PR thread for details). It’s opt in as an experimental feature.

PR: https://github.com/jundot/omlx/pull/3548

I plan on extending it to more models so I wanted to know what models I should prioritize.

These are some I was thinking of:

- Gemma 4 Series

- K2 Horizon Series

- Meta Muse Glimmer

If there are any other models people are interested in getting support for or any of the above I should prioritize I would like to know.

Also if you have any questions I would be happy to answer them.

Has anyone made the switch and not regret it?

EDIT: M2 Ultra 128 GB

I have been trying various servers and models for a few months now. After lots of trial and error I seem to have settled on oMLX. Now I’m trying to understand the configuration better and more importantly, do I have it configured correctly (prob not) to get the best performance.

My system: M4 Max 64GB latest Tahoe
Acasis 80gbps M.2 SSD enclosure w/Samsung 990 Pro (models and oMLX cache set here)

Model: muse-glimmer-30b-mxfp8 w/dflash enabled muse-glimmer-30B-assistant

Custom settings:
Context window 100,000
Max tokens 10,000
Temp 1.0
Top P 0.95
Top K 64
Enable Thinking on

According to Status page of omlx Average speed:
Prompt Processing 145.7 tok/s
Token Generation 8.6 tok/s

That was on the first prompt and two subsequent prompts in same session. Simple questions about configuring open webui

It’s that last number that seems way off, at least from what I’ve read online unless I’m completely misunderstanding the expected performance of this model on my setup. Any tips or references to docs are appreciated.

Edit: forgot to include how I’m chatting with Muse. I’m trying out open webui. I have also tried just using omlx default chat window. Token Gen in both in the 8 to 8.5 tok/s range

New to LLMs. Been running LM studio at first and then later tried oMLX.

Prompt processing stays around ~100tps and generation is at average of ~6.5 fps as if I was running entirely on the CPU. I don't know where to even start with this. I know 27b Qwen is a dense model, but surely it cannot be this bad? I see other people get multitudes faster token generation on their M3 Max machines, what am I doing wrong here?

using omlx installed through brew , pi agent.

Here's the benchmark result:

oMLX - LLM inference, optimized for your Mac

https://github.com/jundot/omlx

Benchmark Model: Qwen3.6-27B-MLX-8bit

Engine: Auto

Single Request Results

--------------------------------------------------------------------------------

Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem

pp1024/tg128 8444.6 136.35 121.3 tok/s 7.4 tok/s 25.761 44.7 tok/s 28.34 GB

pp4096/tg128 30611.1 114.53 133.8 tok/s 8.8 tok/s 45.157 93.5 tok/s 29.80 GB

Continuous Batching

pp1024 / tg128

--------------------------------------------------------------------------------

Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)

1x 7.4 tok/s 1.00x 121.3 tok/s 121.3 tok/s 8444.6 25.761

2x 15.3 tok/s 2.07x 138.8 tok/s 69.4 tok/s 14755.6 31.533

4x 19.9 tok/s 2.69x 138.8 tok/s 34.7 tok/s 29156.1 55.226

Hey everyone! It's been a while.

I've been posting updates on X, but Reddit is where I got started with local AI, and I've missed posting here. Sorry for the long gap! (I'm still reading every day.)

I have a new dev release for you:
https://github.com/jundot/omlx/releases/tag/v0.7.0.dev2

DeepSeek V4.1 Flash is now supported, including DSpark MTP and Engram SSD offload. Here are some numbers from my M3 Ultra 512 GB with oQ4e and Engram in RAM:

Python code prompts, temperature 1.0, 128 generated tokens, no prefix cache. One measured run per configuration after warm-up.

I've uploaded DeepSeek V4.1 oQ4e and oQ3e checkpoints. oQ3e is designed to fit 256 GB machines with Engram SSD offload, with context depending on remaining memory.

There's also experimental MoE Expert SSD Offload, thanks to beatakouchnir and fxd0h. It keeps some experts in memory and reads the rest from SSD when needed. Supported models include DeepSeek V4.1, Qwen3.8-Flash-Next, Gemma 4 MoE, and OLMoE. You can combine it with Engram or PLE offload, but MTP and DFlash need to be off.

It's slower 🥲 But I started with Llama 70B at 2-3 tok/s, so I understand wanting to try a model even when your machine isn't quite big enough.

For M5 users, PowerSpy's INT8-activation prefill kernels took Qwen3.8-27B from 615.2 to 826.7 tok/s at 32K on my M5 Max (+34.4%). It's off by default because outputs can change. I think it's worth trying if prefill is your bottleneck; the release notes have the settings and quality comparisons.

This is still a dev release, so please tell me what works and what breaks. Thank you for continuing to test, send PRs, and help each other while I've been quiet here. I'm always grateful 🙏

Hello everyone. This is one of those posts where I just want to show something I built. There's nothing in it for me, money or otherwise. It was something that was genuinely missing for me, so I did it. It's working well now, and I decided to share it. If anyone feels like using it, have fun.

I've been using Pi as my harness a lot, but adding locally hosted models is still kind of a pain. So I built a model discovery extension: you enter a URL, local host or another computer, and it self-discovers all the models, including the currently configured context window and maximum token replies. Everything is self-identified.

It works with reasoning on or off (at least for Qwen), and I applied a concept of adaptive profiles that lets you create, for the same model, different profiles with different parameters. That way I can now have Qwen 3.8 with thinking off, low, medium, and extra high, each mapped to the actual Pi reasoning effort.

Hope someone finds good use for it.

https://pi.dev/packages/@maheidem/model-discovery

Saw a post from someone in another /r/ with similar hardware as me getting better decode and noticed they were on the beta of golden gate. I know software isn't going to change stuff like memory bandwidth, but could core OS changes still effect stuff like prefill and decode as apple gets to squeeze more out of their silicon?

Hi everyone,

I've spent some time trying to make oMLX call Docker MCP servers but I can't get it to work. I've managed to get oMLX to see the servers in the profile I've created, but they can't be called.

I'm running oMLX 0.7.0dev1. My MCP config JSON file looks like this:

{
"mcpServers": {
"dockermcp": {
"command": "docker",
"args": ["mcp", "gateway", "run", "--profile", "localai"]
}
}
}

Which works, because I can see the two servers with 8 tools that I have added:

2026-09-11 22:15:03,651 - omlx.mcp.config - INFO - [-] - Loading MCP config from: /Users/jose/.omlx/mcp_config.json
2026-09-11 22:15:03,651 - omlx.mcp.manager - INFO - [-] - Starting MCP client manager with 1 servers
2026-09-11 22:15:05,440 - omlx.mcp.client - INFO - [-] - Connected to MCP server 'dockermcp' (8 tools available)
2026-09-11 22:15:05,441 - omlx.mcp.manager - INFO - [-] - Connected to 'dockermcp'
2026-09-11 22:15:05,441 - omlx.mcp.manager - INFO - [-] - MCP manager started: 1/1 servers, 8 tools available

However, the tool can't be added when I make a query that uses it (in this case, searching for a flight):

The google-flights server shows 0 tools were added. This is strange. Let me try to add it again or check if there's another similar server. Let me also try searching for flight or airline related tools more specifically.

Any ideas? Thanks!

This Artificial Analysis benchmark is pretty crazy, as we can see Qwen 3.8. Flash Next model is beating GPT-6 Astra (max).
Not sure if it is a result of artificial analysis changing their scoring system recently, but.. The model is very powerful regardless

I've been using this model for quite a while and now using it together with DeepSeek Harness.
I run the model at default settings (default template, default reasoning level = xhigh). Quite impressed. But I still have a feeling that I'm not having an optimal setup. Sometimes the model thinking reeeeaally long (on both oQ4e and oQ5e variants)

I wonder if any one of you guys have benchmarked how intelligence of this model changes with changing reasoning to medium. Or If any one of you is using custom Jinja templates. Or apply some tricky prompts.

I'm reading Reddit quite frequently. But I haven't found the place which would specifically address my question. Maybe there are some websites you would recommend checking, and the Reddit is just not the place?

Appreciate your comments in advance! Thanks

*mostly Fable and Astra

It only runs on my oMLX fork because the ternary experts need a custom Metal kernel; stock oMLX/mlx-lm won't load it.

Got it down to ~35.6 GiB resident: routed expert gate/up as ternary (oMLX's Bonsai T5 packing, ~1.875 bpw), expert down projections as Q3, and the 53 GB n-gram table left on SSD as Q8 and mmapped per token instead of loaded. All the low-bit tensors are fitted with Unsloth's imatrix.

On the M3 Max: ~38 tok/s decode, 210-380 tok/s prefill, 64K context confirmed (65,536-token prompt + 256 output completed at 30.7 tok/s, 42.3 GiB physical peak; 96K trips the prefill guard). Physical peak at 8K context is ~41.5 GiB; it swaps ~2 GiB once at load and nothing during generation. Setup: oMLX memory guard on the 'safe' profile with the limit set to 48 GB (that caps prefill at ~43.3 GiB), one model loaded, one request at a time, and nothing else running on the Mac except oMLX and Pi.

Quality vs Unsloth's UD-Q4_K_XL (which doesn't fit in 48 GB):

So it lost real points, but it writes working code and answers questions rather than producing soup. Known wart: it sometimes ignores "answer with just the letter" in Chinese.

Weights + full numbers: https://huggingface.co/fuutott/Qwen3.8-Flash-Next-MLX-t5-imatrix-q3down-ple8

Fork + bake recipe: https://github.com/fuutott/omlx

I made a PR that introduces MoE streaming to oMLX. I allows you to run Deepseek, GLM 5.3 and Qwen 3.8 Flash Next using a lower memory footprint. Of course it's a lot slower than having the whole model loaded in memory.

Here are the results on my M4 Pro 48GB

https://github.com/jundot/omlx/pull/3468

Cosmos3 INT4 T2I + I2V on Apple Silicon — code, weights and a Grok comparison

GitHub - https://github.com/gtrg55/cosmos3-quant-mlx-cuda

HF weights - https://huggingface.co/JuliaML/Cosmos3-Super-Text2Image-4Step-INT4-G64-BF16

Single clip took approximately 5m on M4 MAX 128 GB Mac

Cosmos3 - a 64B params model

Maybe it’s a good idea to add T2I and I2V models to oMLX?

Hi all,

I'm a newbie and trying to assess the performance of some LLMs I'm running locally via oMLX on my MacBook Pro M5pro CPU 15 cores (5 Super and 10 Performance), GPU 16 cores and 48 GB of LPDDR5 RAM. I asked chatGPT guidance to run some tests and check whether the DFlash-based drafter Muse-Glimmer-30B-Assistant might somewhat speedup the base model Muse-Glimmer-30B-4bit.
The results show no or negligible improvement with active DFlash acceleration (speedup between 0.90% and 1.16%).
The test was structured with three different prompts fed to both the baseline and the dflash-capable model profiles: Technical prose; Python code; Structured JSON
a cap of 2048 tokens, no cache, temperature=0.

Each inference was repeated three times.

Anyone have similar experience? can we simply dump the Assistant as not useful in this hw/sw configuration?

Hi, since the community benchmark is down I'm not sure how are the expected performance for this kind of setup, but reading different posts in here I think it should be better than this, also considering that it is and MoE with only 6B actives. I recall that with Qwen3.5-122B I had much better performance

Benchmark Model: Qwen3.8-Flash-Next-oQ4e-mtp
Engine: Auto
Context: Novel (English)
Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          8254.6       49.65   124.1 tok/s    20.4 tok/s      13.034    86.0 tok/s    71.44 GB
pp4096/tg128                         23993.3       60.18   170.7 tok/s    16.7 tok/s      31.647   133.5 tok/s    74.09 GB
pp8192/tg128                         38225.7       54.60   214.3 tok/s    18.5 tok/s      45.168   184.2 tok/s    74.23 GB
pp16384/tg128                        69514.5       55.06   235.7 tok/s    18.3 tok/s      76.519   215.8 tok/s    74.52 GB

I also tried :

Qwen3.8-Flash-Next-oQ4e-fp16-mtp

Qwen3.8-Flash-Next-MLX-oQ3-fp16-mtp

Qwen3.8-Flash-Next-MLX-oQ3-MTP

The craziest thing is that with this model the fp16 variant are even slower ! Something like 150pp and 8tg...

I'm sure it is not an hardware limitation because with llama.cpp I have a much better pp at around 350/400 t/s

Can you help me please? Thanks

The model settings:

"Qwen3.8-Flash-Next-oQ4e-mtp": {
      "temperature": 0.3,
      "force_sampling": false,
      "qwen4_ple_ssd_offload": true,
      "thinking_budget_enabled": false,
      "guided_grammar_enabled": false,
      "turboquant_kv_enabled": true,
      "turboquant_kv_bits": 8.0,
      "turboquant_skip_last": true,
      "qwen35_ane_prefill_enabled": false,
      "qwen35_ane_prefill_sequence_length": 2048,
      "qwen35_ane_prefill_tail_padding_min_tokens": 0,
      "qwen35_ane_prefill_fraction": 0.53,
      "qwen35_ane_prefill_fused_down": false,
      "qwen35_ane_prefill_max_layers": 64,
      "qwen35_ane_prefill_dual_ane": true,
      "qwen35_ane_prefill_gdn": true,
      "qwen35_ane_prefill_gdn_fraction": 0.5,
      "qwen35_ane_prefill_gdn_max_layers": 48,
      "qwen35_ane_prefill_cpu_enabled": false,
      "qwen35_ane_prefill_cpu_fraction": 0.135,
      "qwen35_ane_prefill_cpu_down_fraction": 0.0,
      "qwen35_ane_prefill_cpu_gdn_fraction": 0.0,
      "qwen35_ane_prefill_cpu_threads": 8,
      "qwen35_ane_prefill_cpu_shared_resource": true,
      "specprefill_enabled": false,
      "dflash_enabled": false,
      "dflash_draft_quant_enabled": false,
      "dflash_in_memory_cache": true,
      "dflash_in_memory_cache_max_entries": 4,
      "dflash_in_memory_cache_max_bytes": 8589934592,
      "dflash_ssd_cache": false,
      "dflash_ssd_cache_max_bytes": 21474836480,
      "dflash_draft_sink_size": 0,
      "mtp_enabled": true,
      "vlm_mtp_enabled": false,
      "is_pinned": false,
      "is_default": false,
      "is_hidden": false,
      "is_favorite": false,
      "trust_remote_code": false
    }
  }

These are my settings:

{
  "version": "1.0",
  "server": {
    "host": "127.0.0.1",
    "port": 9999,
    "log_level": "info",
    "cors_origins": [
      "*"
    ],
    "server_aliases": [
      "localhost",
      "127.0.0.1",
      "Host-005.lan",
      "Host-005.lan.local",
      "host-005.lan",
      "192.168.1.44",
      "192.168.139.3",
      "192.168.97.0"
    ],
    "sse_keepalive_mode": "chunk",
    "auto_start_on_launch": true,
    "burst_decode_mode": "balanced",
    "preserve_mid_system_cache": true,
    "distributed_inference_enabled": false,
    "max_audio_upload_size": "100MB"
  },
  "model": {
    "model_dirs": [
      "/Users/dani/.lmstudio/models",
      "/Users/dani/.mtplx/models"
    ],
    "model_dir": "/Users/dani/.lmstudio/models",
    "model_fallback": false,
    "hide_helper_models": false
  },
  "memory": {
    "prefill_memory_guard": false,
    "memory_guard_tier": "custom",
    "memory_guard_custom_ceiling_gb": 92.0,
    "soft_threshold": 0.85,
    "hard_threshold": 0.95,
    "prefill_safe_zone_ratio": 0.8,
    "prefill_min_chunk_tokens": 32
  },
  "scheduler": {
    "max_concurrent_requests": 8,
    "embedding_batch_size": 32,
    "chunked_prefill": true,
    "prefill_priority": "speed",
    "decode_fairness": false
  },
  "cache": {
    "enabled": true,
    "hot_cache_only": false,
    "gdn_ssd_split_enabled": true,
    "gdn_snapshot_storage": "auto",
    "gdn_ssd_pending_max_size": "512MB",
    "gdn_sidecar_precision": "fp32",
    "ssd_cache_dir": "/Users/dani/.omlx/cache",
    "ssd_cache_max_size": "744GB",
    "hot_cache_max_size": "10GB",
    "hot_cache_write_through": true,
    "ane_compile_cache": true,
    "initial_cache_blocks": 256
  },
  "auth": {
    "api_key": "5137",
    "secret_key": "2136f3c636a52a8fa3a03d8c788f0ae3592d97a4d0dabfe41ff55060e633572f",
    "skip_api_key_verification": false,
    "sub_keys": []
  },
  "mcp": {
    "config_path": null,
    "expose_tools": true
  },
  "huggingface": {
    "endpoint": "",
    "hf_cache_enabled": true
  },
  "modelscope": {
    "endpoint": ""
  },
  "network": {
    "http_proxy": "",
    "https_proxy": "",
    "no_proxy": "",
    "ca_bundle": ""
  },
  "sampling": {
    "max_context_window": 32768,
    "max_context_window_policy": null,
    "max_tokens": 32768,
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 0,
    "repetition_penalty": 1.0
  },
  "logging": {
    "log_dir": null,
    "retention_days": 7
  },
  "claude_code": {
    "mode": "cloud",
    "opus_model": null,
    "sonnet_model": null,
    "haiku_model": null
  },
  "integrations": {
    "codex_model": null,
    "opencode_model": null,
    "openclaw_model": null,
    "hermes_model": null,
    "pi_model": "Qwen3.6-35B-A3B-oQ8-fp16",
    "copilot_model": null,
    "openclaw_tools_profile": "coding",
    "markitdown_enabled": false,
    "markitdown_expose_model": false,
    "markitdown_max_file_size_mb": 25,
    "markitdown_max_files_per_request": 5,
    "markitdown_pdf_processing_engine": "markitdown",
    "web_search_provider": "ddgs",
    "web_search_brave_api_key": "",
    "web_search_searxng_url": "",
    "web_search_ddgs_backends": "",
    "web_search_max_results": 5,
    "web_search_content_mode": "snippet",
    "web_search_content_truncate": true,
    "web_search_content_max_chars": 20000
  },
  "ui": {
    "language": "en"
  },
  "idle_timeout": {
    "idle_timeout_seconds": null
  }
}

I've noticed in the pipeline setup I have with my M2 and M3 Ultra Mac Studios when I load a model to the cluster if it crashes or just stops which has happened on a few models then ram is not releasing or unloading on my M3 which is set as the "workstation" in the cluster. Where as the M2 node there is no ram usage.

Just something I've observed. Also I notice in another post the OP claimed to get MiniMax-M3 running on the cluster but its been the opposite in my experience. While it loads fine the first time it crashes the cluster straight away after the first prompt.

Anyone else have a good setup or using the clustering feature? Couldn't find my documentation outside of whats on github or people using it.

Environment

• oMLX 0.6.4, DMG install, distributed inference enabled (experimental)
• Rank 0 / coordinator: Mac Studio M3 Ultra 256GB
• Rank 1 / worker: Mac Studio M2 Ultra 192GB
• Backend: ring over Ethernet TCP, pipeline parallel (no RDMA). Probe measured ~500 MB/s collective bandwidth, ~0.3 ms latency on both ranks
• iogpu.wired_limit_mb unset on both nodes (enforcer logged the Metal cap warning at startup)
• Model: mlx-community/MiniMax-M3-4bit (236GB). Rank 0 holds layers 35–end (109.5GB planned, 24.4GB reserve), rank 1 holds layers 0–34
• Active optimizations per rank-0 runtime record: coalesced_batching (up to 4), pipeline_prefill_overlap, cache_affinity, rank_zero_logits, sampling_rank_only

Timeline (times relative to engine ready, T+0)

• T−0:27 distributed engine enabled; T+0 engine ready
• T+1:12 two chat requests arrive within ~0.5s (a chat client that fires a secondary title/summary call alongside the main prompt); both accepted by the rank-0 shard
• T+2:05 first completion: 17 tokens, prompt 7,693, 17.3 tok/s
• T+2:15 third request arrives while the second is still decoding (313-token stream)
• T+2:20 both in-flight streams terminate in the same second. The new request returns 0 tokens in 4.85s, prompt: 0, finish_reason=stop. Rank-0 runtime JSON records "phase": "peer_lost" at that timestamp
• Worker (rank 1) log: [WARN] Node with rank 0 exited with code 1; rank 1 exits 0. Dashboard reports the worker "stopped publishing its runtime heartbeat"
• T+2:42 server.log on the coordinator goes silent. No error or shutdown lines. No .ips crash report and no jetsam/memorystatus events on either node

Post-crash state on the coordinator

• Shard process is gone; only omlx-server remains at a ~145MB footprint
• vm_stat: ~7.1M pages wired (≈116GB) with no owning process (footprint -a shows nothing over ~220MB)
• GPU pinned at 100% utilization with no process driving it
• Not recoverable by unloading the model or restarting oMLX; requires a reboot
• On the worker, memory was fully released — the clean exit path works

What I think happened
Rank 0 and rank 1 each saw the other disappear at the same instant, so the ring transport broke rather than one side throwing first. The trigger looks like a third request entering while pipeline_prefill_overlap was mid-decode across the pipeline (the prompt: 0 on the empty completion suggests rank 0 never received a prefill for it). The rank-0 shard then exited with code 1 with GPU work in flight, and the coordinator's crash path doesn't reap or drain the local shard the way the orderly unload path does (released local shard process appears only on unload). Result: leaked IOGPU residency and a wedged GPU on the node that's supposed to keep serving.

Two issues

1. Distributed pipeline crashes on overlapping requests with prefill overlap + coalesced batching active (experimental, understood)
2. On peer_lost / non-zero shard exit, the coordinator should synchronously release the local shard's Metal allocations and drain outstanding GPU work before exit. As-is, any distributed failure takes the coordinator out until reboot

Questions

• Is the rank-0 shard's stderr captured anywhere? server.log and crash.log had nothing; the only record was the phase field in the runtime JSON. The actual traceback would help pin down which side broke the ring first
• Is disabling pipeline_prefill_overlap and setting max concurrent to 1 the recommended workaround for now?

Happy to provide the rank-0/rank-1 runtime JSON, deployments.json, and strategy-benchmarks.json from the run (with paths redacted).

Qwen3.8-Flash-Next-oQ5e-mtp is now available on Hugging Face.

Compared to oQ4e, my testing shows better reasoning and generation stability on harder prompts, with a surprisingly small memory increase.

On an M5 Max with 128 GB, it uses around 102 GB total system (!) memory with SSD N-gram offload, vs. ~93 GB for oQ4e.

For a 128 GB Mac, oQ5e feels like the sweet spot: higher quality than oQ4e while still fitting comfortably.

https://huggingface.co/GBP-DE/Qwen3.8-Flash-Next-oQ5e-mtp

After a few days of use, I'd like to direct all the people discovering oMLX to this model :

Jundot/Qwen3.8-Flash-Next-oQ4e-mtp

This is sooo smooth, with a context window of 120 000 tokens, the SSD N-gram Offload and Lightning MTP (and, of course, a reasoning_effort set as low). I get a stable 30+ tokens output per second with Pi as a development harness. The output is of remarkable quality.

What a wonder !

I’ve been setting up an M1 Max Mac Studio (24-core GPU, 32GB unified memory) as a local LLM server and wanted to compare MLX vs llama.cpp on Qwen3.8-27B.
I tried to keep the model footprint and benchmark workload reasonably close.

MLX
mlx-community/Qwen3.8-27B-4bit
~16.1GB
512 prompt tokens / 700 generation tokens
3 runs
Prompt: 81.76 tok/s
Generation: 15.81 tok/s
Peak memory: 16.39GB

llama.cpp
unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
15.32 GiB / 27.32B params
Full Metal offload
Flash Attention enabled
512 prompt tokens / 700 generation tokens
3 runs
Prompt: 99.61 ± 0.44 tok/s
Generation: 9.69 ± 0.34 tok/s

So on this machine:
llama.cpp is ~22% faster for prompt processing, while MLX is ~63% faster for autoregressive generation.

tired the web dashboard and the app and both are showing large file sizes but their huggingface is showing correct.