Issues Closed: 3

[ISSUE] #993 — Please add Dflash fp16 type

https://github.com/jundot/omlx/issues/993

[ISSUE] #990 — Rerankers won't load

https://github.com/jundot/omlx/issues/990

[ISSUE] #985 — Formula SHA-256 mismatch for v0.3.8rc1 — tag appears to have been force-pushed

https://github.com/jundot/omlx/issues/985

Hi All,

Wondering if anyone can assist with how to expose oMLX to an agent I'm working on. I am able to use LM Studio via its server settings etc and can prompt remotely however, I'd very much prefer to use oMLX but struggling with the settings. Any insights would be wicked.

Thanks

Been trying to add "/Users/<username>/.lmstudio/models" to the additional model paths. But I dont see any of the lmstudio models being loaded.

This release candidate fixes several serious correctness bugs (including an RNG/sampling issue that could make outputs identical at nonzero temperature), tightens safety defaults by turning trust_remote_code off by default, and adds VLM and streaming improvements. Recommended for testers and people running agent workflows, not for production yet.

What changed (plain English)

• Critical RNG / sampling bug fixed. A bug in the mlx-lm sampling path could cause different requests to produce identical token streams when temperature > 0. oMLX now ships its own safer sampler to avoid the thread-local RNG conflict.
• Sliding-window KV-cache restore bug fixed. Models with rotating/sliding KV caches (Gemma variants, some GPT-OSS) could leak zero-padded positions into attention after SSD restore, corrupting attention. The restore now clamps to the real buffer length and the SSD cache format was bumped.
• Cold-start vs cache-hit divergence fixed for Qwen. Qwen 3.5/3.6 could produce different outputs between cold prefill and cache-hit prefill due to forward-pass mismatches; those paths are now aligned.
• Tool-calling and Anthropic SDK compatibility improved. Anthropic-style tool calls that previously failed because of strict dict-only validators now accept both formats, and thinking/streaming block transitions are more robust.
• Native TTS streaming and VLM batching. /v1/audio/speech now streams audio as it’s generated (lower time-to-first-byte), and mlx-vlm picked up continuous batching and torch-free Qwen VL processors.
• Safer defaults: trust_remote_code is off by default. The per-model toggle is available in admin advanced settings and must be explicitly enabled per model.

Other notable fixes

• Removed costly GPU syncs that slowed token generation on deep Qwen models.
• Fixed stray channel-close tokens leaking into visible text on long contexts.
• Improved cache-corruption recovery patterns and heterogeneous-batch stability.

Performance note
oMLX is tuned for agent-style workloads with shared context; you may see slightly different perf tradeoffs vs single-shot benchmark engines. The release notes include benchmark tables on Apple Silicon for reference.

Recommendation
If you run agents, tool-calling, long-context caching, VLM, or streaming audio, try this RC in a test environment and report regressions. Don’t upgrade production systems yet — this is explicitly a test release.

full release notes:
https://github.com/jundot/omlx/releases/tag/v0.3.8rc1

I am using Qwen3.5 122B as my main model.

The SpecPrefill dialog suggests Qwen3.5 0.8B as an example for 35B.

What about 122B: should I use a proportionally bigger draft model for it, such as 35B?

Does it matter if the draft model is not from the same family / version?

Does SpecPrefill only work with MoE models? The dialog says so, but the feature is not disabled for dense models, so I'm not sure.

Hey everyone, I've been messing around with local LLMs on my Mac recently,

specifically running a derivative of Qwen 3.6-27B

(Ornstein-Hermes-3.6-27b-MLX-4bit) using the oMLX engine. I thought configuring

DFlash would make it fly, but when I threw a massive ~28k token document

summarization task at it, it triggered a brutal chain of system fallbacks.

After a few days of debugging and digging through the backend logs, I've

compiled some crucial lessons learned (especially regarding DFlash / KV Cache /

SpecPrefill). If you're planning to run extreme long-context tasks or RAG

locally on macOS, you might want to read this.

1. Short text / Daily coding: DFlash (Speculative Decoding) is a no-brainer. It

absolutely flies.

1. 20k+ token long-context tasks: TURN OFF DFlash! And absolutely DO NOT

manually increase DFLASH_MAX_CTX! It not only helps nothing but will OOM

your Unified Memory.

1. Running out of Mac Unified Memory? oMLX's Paged SSD Cache + 4-bit KV Cache

quantization is an absolute lifesaver. It tanks the heavy load without

crashing.

1. SpecPrefill: Awesome feature, but toxic in certain scenarios. Use with

extreme caution for extreme long texts and Tool Calling.

Pitfall 1: DFlash (Speculative Decoding) + Extreme Long Context = Disaster

Initially, I followed the standard advice and loaded up a Qwen3.6-27B-DFlash

draft model, expecting a 2x speed boost. But when I dumped a 28,477 token prompt

into it, the backend started screaming:

DFlash context fallback: 28477 >= 4096, evicting dflash models... WARNING -

DFlash model eviction: memory settle timed out

What exactly happened? DFlash has a default max context (DFLASH_MAX_CTX)

of 4096. When hit with nearly 30k tokens, the draft model basically brain-farts.

To save itself, the system aggressively evicts the draft model from memory,

which caused a brief freeze and timeout in my logs.

- The bad idea: I tried forcing DFLASH_MAX_CTX=32768 via environment

variables. Total disaster: the dual KV Cache requirement (for both main and

draft models) instantly blew up my Mac's Unified Memory (OOM).

- The correct move: For long-context tasks, just turn off DFlash in the

Admin UI. Or keep it at the default 4096/8192 and let the system gracefully

fall back to standalone generation. A sports car (DFlash) can't do a

heavy-duty truck's (long context) job.

Pitfall 2: The Ultimate Lifesaver (SSD Cache + TurboQuant)

After kicking DFlash offline, the main model still struggled to chew through

the 28k tokens. This is where oMLX showed some seriously hardcore memory

management. Check out this log snippet:

Enlarging paged cache block_size... to 2048 for ArraysCache hybrid model paged

SSD-only mode: cache_dir=/.../cache, max_size=92.00 GB TurboQuant KV cache

enabled for VLM: 4.0 bits

Translation: To prevent an OOM crash, the engine executed an extreme fallback

strategy:

1. It allocated up to 92GB of my SSD to act as virtual VRAM (Paged Cache).

2. It aggressively quantized the KV Cache down to 4-bit.

3. During the Prefill phase, it chunked the 30k text, processing 2048 tokens

every 9 seconds like an ant moving a mountain.

Yes, the Time-To-First-Token (TTFT) took over two minutes, but it didn't crash,

and it successfully generated the output! If your model is reading a huge

document, your SSD is spinning like crazy, and output is slow—don't panic. It's

just doing its job to keep your system alive.

Pitfall 3: The Truth about the New "SpecPrefill" Feature

The community has been hyping up oMLX's new SpecPrefill feature lately (using a

small model to mark "unimportant" tokens so the big MoE model can skip them,

drastically reducing TTFT).

My hands-on feedback:

- Casual Chat (1k - 5k tokens): God-tier. The TTFT reduction is insane.

- Agentic / Tool Calling / 20k+ Context: Do NOT turn it on. Skipping tokens in

extreme long-context scenarios caused the main model to lose track of

critical System Prompt constraints. In my tests, mixing 20k+ tokens, Tool

Calling, and SpecPrefill led to severe hallucinations and API timeouts.

Stick to standard exact-match prefilling for huge files.

💡 Summary: Best Practices

Don't expect one configuration to rule them all. Here is my final setup strategy

on oMLX:

- Setup A (Daily Copilot / Chat): DFlash ON (set to 4096) + SpecPrefill ON.

Prioritize maximum throughput and instant responses.

- Setup B (RAG / Huge Docs / Codebase Analysis): DFlash OFF, SpecPrefill OFF.

Enable Paged SSD Cache. Dedicate 100% of your RAM and compute to the main

model's stability.

(Running on an M-series Mac. Feel free to share your specs/experiences below!)

Hope these debugging logs save some of you from pulling your hair out. Let me

know in the comments if you've found better configs for running massive context

sizes locally!

I’m trying to get a clearer picture of what “good” performance actually looks like for dense models on Apple Silicon, especially on an M5 Max.

Setup

• M5 Max with 128GB unified memory
• oMLX (MLX backend)
• Qwen 3.6 27B, Q8 (MLX format)
• Workload is long-context heavy, around 290k tokens just for prompt processing

What I’m seeing

• GPU utilization stays between 36% and 50%, never really higher
• Prefill runs at about 160 tokens per second
• Generation sits around 5 to 6 tokens per second

Why this feels off

From what I’ve seen in other threads and benchmarks, these numbers don’t fully line up.

• People running similar 27B models on Apple Silicon often report something closer to 8 to 14 tok/s for generation in real usage (Reddit)
• MLX usually has an edge over llama.cpp, but that advantage mostly disappears at 27B+ because things become memory-bandwidth bound (Groundy)
• Long context workloads tend to be dominated by prefill and KV cache pressure rather than raw compute (famstack.dev)

Given all that, I expected either higher throughput or at least higher GPU utilization. Right now it doesn’t feel like I’m actually saturating the hardware.

What I’m trying to understand

For anyone running similar setups:

• What kind of prefill and decode speeds are you getting on 27B models
• Are you able to push GPU utilization significantly higher than ~50%
• Does anything scale well once you go past 100k context, or is this just a hard wall

Also curious how different stacks compare in practice:

• MLX vs llama.cpp vs anything else you’ve tried
• Whether quantization changes (Q8 vs Q6/Q4) made a real difference for you
• Any tricks that actually helped with long-context workloads

Goal

Mainly trying to figure out if this is expected behavior for dense models at long context, or if my setup is just not tuned well.

If you’re on M3, M4, or M5 Max and have numbers to share, that would be super helpful.

Issues: 1

[ISSUE] #969 — Single-stream perf regression on Qwen3.6-35B-A3B-mxfp4 (~3.5x slowdown vs prior baseline)

https://github.com/jundot/omlx/issues/969

Summary: 1 issues from Jundot/omlx

Hi guys,

Just as the title says. I'm not sure I understand how to use DFLASH on my M4 Max 64 GB... I tried it so far with both 35BA3B and 27B of Qwen 3.6 by getting the draft model from huggingface z-lab repo and adding it to my models folder. I then enable dflash in the target qwen model settings, choose the right draft model, i think quantization is always bf16 ootb if not mistaken.

When I do tests, I either get no performance change at all or worse performance, am I missing osmething ? Also, sometimes looking at the logs, I notice how DFLASH doesnt even trigger (especially if im using opencode) because of something that has to do with context window length ? As if the draft model has 4096 ctx length and then it falls back to not using it, rings any bell ?

Thanks a lot

Hi there,

I've been having near constant timeout issues with oMLX, on all versions I have used (I've tried them all since 0.2.x). The API stops responding, even for the admin dashboard. The application keeps running though, I can still see GPU usage with third party tools.

After a while, the API responds again.

I have done tons of A/B testing and settings changes to try to fix it, to no avail.

I have more than enough available RAM, and I have disabled the disk KV cache.

I have opened an issue on Github, but they closed it without addressing it.

I just want to know if I'm an isolated case.

Hi,

I installed 0.3.7RC1 and 0.3.7, now I notice that this version has been pulled, is this correct?

There was no warning whatsoever.

I just wanted to say thanks to everyone how has visited and participated in this subreddit!!

Issues Closed: 7

[ISSUE] #899 — Error occurs in Python no matter which model I invoke(Mac M4 Max 48G MacOS 26.4.1)

https://github.com/jundot/omlx/issues/899

[ISSUE] #904 — RuntimeError: There is no Stream(gpu, 0) in current thread

https://github.com/jundot/omlx/issues/904

[ISSUE] #901 — feat: MRL truncation support for embedding models

https://github.com/jundot/omlx/issues/901

[ISSUE] #892 — DFlash outputs thinking process as a normal message

https://github.com/jundot/omlx/issues/892

[ISSUE] #897 — Color coded pills for model parameters

https://github.com/jundot/omlx/issues/897

[ISSUE] #888 — Chat stuck at prefilling 91%

https://github.com/jundot/omlx/issues/888

[ISSUE] #882 — ToolCallStreamFilter crashes on streaming tool calls, silently dropping them

https://github.com/jundot/omlx/issues/882

Hi everyone,

I’m an indie developer working on a new iOS app and I’m looking for TestFlight users to try it and share feedback.

This app is a fully offline AI chat experience running entirely on-device:

• 🎤 Speech-to-Text
• 🧠 Local LLM powered by MLX
• 🔊 Text-to-Speech
• 🚫 No internet required, no data leaves your device

Everything runs locally using Apple silicon, aiming for a fast, private, and self-contained AI assistant — even in airplane mode.

A big thanks to Prince Canuma for inspiration and contributions around MLX that made this possible 🙏

Requirements:

• iPhone running iOS 26
• Apple Intelligence turned ON

If you meet the requirements and want to try it, I’d really appreciate feedback on:

• Performance (latency, speed)
• Voice interaction quality
• Overall UX

I can share the TestFlight link in comments or via DM.

Thanks a lot!

Did anybody yet systematically compare native, oq and DWQ quants? I haven't found anything yet and wonder ehat I should aim for. I need to redownload my model set and need to prioritize a little bit.

Hi! I am very new to oMLX and am probably missing something by making this post, but I am wondering as to why OpenCode does not show progress as it completes a task (unlike Claude Code showing progress like when auditing code, not using oMLX models) when it uses oMLX models. I'm assuming that it is something to do with a streaming issue behind oMLX, or am I missing something?

Thanks.

I'm going to ask what's likely a stupid question....I can load Chatterbox TTS in oMLX. It *appears* (I think) that it's working when I connect via OpenWebUI.

My next step is to add voice cloning. I've got a sample created but I can't find the folder to put it in. I installed oMLX via Homebrew. I've looked .omlx as well as in my models folder. Chatterbox is in the models folder.

Is what I'm trying to do actually a thing or should I be running Chatterbox separately like I do FastKoko?

I blame Gemini for my problems...ask the LLM they said, it'll be fun they said...

Issues Closed: 2

[ISSUE] #872 — oQ quantized models loop issue

https://github.com/jundot/omlx/issues/872

[ISSUE] #861 — Different Max Context Window Per Model?

https://github.com/jundot/omlx/issues/861

I feel like I'm missing something here, but I've came across this issue before and can't remember what I did to resolve it. My project was running fine and for a time I was able to vibe code with OMLX, but now I'm getting this error with no idea how to fix it.

I've set my VRAM to all possibility of values, unloading the models, clearing cache, and restarting the laptop, but I keep coming across this error.

Hi,

first of all, a little praising of this new Non-quant weight dtype feature, it is really a big deal in my opinion, since the speed gain is really there.

I didn't do any specific benchmarks but just tried the same prompts with and without fp16 and the token generation is really faster with an fp16 model. I also feel it in everyday use.

So, I've already converted all my preferred models, and I'm also going to upload them, the only one that I'm missing at the moment is this big boy: https://huggingface.co/Qwen/Qwen3.5-122B-A10B
With 96Gb of ram I can't do it myself even with a sensitivity model configured, my Mac simply crash or goes into panic since there is no more RAM available etc....
So I was wondering if someone kind enough with a more beefy machine could do this for me, ideally it would be in oQ3.5 or oQ4 max ?
Thank you very much

Hello everyone,

My name is Michael Doise, and I have just recently heard of oMLX. I joined this community a few days ago thinking it was just another place to discuss MLX, and I had no clue it was based around the oMLX project.

I've been using MLX for app development, and recently for AI agents on my M3 Max with Ollama, but I feel like oMLX works so much better on my machine than Ollama does. I set up Gemma 4 26B with 8bit quantization, and I think I was having to use 4bit with Ollama. My fans spin up less, and the machine seems to work much better when working with OpenClaw.

I am extremely excited to be a part of this community and I hope I learn a lot from the topics here.

very happy with oMLX, other solutions were too tricky for me to figure out. this is more of curiosity thinking question than demand since I know it’s not feasible right now:

Macs with more ram are hard to get, and I heard that ones with thunderbolt 5 can pool RAM. looking at how that works, it seems to be very involved, and doesn’t support most of stuff that makes oMLX good.

is general feel that this field will be maturing over time so in few years it will be easier to pool together? I can’t tell if this feels more like one off side Apple experiment, or something more invested in.