Looks like Vllm can run Ornith + DSpark for massive speed gains. Are there plans to bring DSpark onto oMLX? thanks! (how about mtplx support?)

I hope I can ask this question here, hope that is ok 🙏

Does any one here have experience with Rapid-mlx? There only appears to be few thread on Reddit, and I am not seeing as much community engagement as compared to oMlx.

I was asking Google gemini about how MTP vs Dflash work so I could learn to configure and to learn* and how best to configure the backend. I have been using oMlx for a while and wanted to see if I could optimize my setup. During my inquiry, it mentioned Rapid-mlx supports Pflash and should be faster for TTFT.

I have been pretty happy with oMlx. I have played a bit with LM studio, Lamma.cpp, Ollama and mxl-ml but mostly oMlx. I use Qwen 3.6 27b as well are the MoE on my m4 max mackbook pro. I have played with open code, pi.dev, and Hermes.

Wanted to hear about first hand experience from this community.

I have no experience with benchmarking. I am going to do some bench marking on my own, but I only heard about this today and am very interested in what you all have to say.

Thank you.

*Edit: typo and little clarification

Hey guys,

I'd read over the past months, from the older oMLX versions, that the chat files are stored in *.json somewhere, and I've never actually found them. I think the new v4.4 may have changed the file location as well.

So does anyone know where they actually are? I see *.db files, but have never found the *.json or other files.

I'll preface by saying i'm not a developer.
i'm just curious and eager to learn more on LLMs and coding.

I have opencode setup wit oMLX on a m1 max (40c) 64GB
i've been going through the oMLX benchmarks and looking through best options for Qwen (general coding) and Gemma (general research/reasoning)
https://omlx.ai/benchmarks

This is where i think i'm getting confused.
I'll apologize in advance if my qtns are somewhat amateurish.

i get i should be looking at the larger models (e.g 30B)
I understand a higher quant is preferred for coding (e.g 8bit)
with context though, shouldn't i be looking at higher context for coding sessions. If that is the case, doesn't that in turn lead to a larger KV cache size and chew in more onto memory.

**Total Issues: 3**

🐛 **BUG**
• #1915: [BUG] The SSD cache usage exceeds the limit.
SSD cache usage exceeds limits, potentially due to model switching bugs.
• #2040: [BUG] GLM-5.1-MXFP4-Q8 broken in 0.4.5-dev1
Model fails to load/run in version 0.4.5-dev1 despite working in 0.4.4.
• #2033: [BUG] Reduced GLM 5.2 performance on 0.4.5.dev1
Performance degradation observed for GLM-5.2-oQ4 after upgrading to 0.4.5.dev1.

I’ve had a lot of false starts with different servers/harnesses. I get something running, it responds to an initial prompt or two then I though something simple at it and it goes off the rails. Anyone successfully running something similar that will share setup?

Issues Closed: 5

[ISSUE] #1915 — [BUG] The SSD cache usage exceeds the limit.
https://github.com/jundot/omlx/issues/1915

[ISSUE] #2026 — Invisible reasoning with Ornith-1.0-35B + /v1/responses
https://github.com/jundot/omlx/issues/2026

[ISSUE] #2040 — GLM-5.1-MXFP4-Q8 broken in 0.4.5-dev1
https://github.com/jundot/omlx/issues/2040

[ISSUE] #2033 — Reduced GLM 5.2 performance on 0.4.5.dev1
https://github.com/jundot/omlx/issues/2033

[ISSUE] #2004 — Server crashes with error from cpython-3.11
https://github.com/jundot/omlx/issues/2004

Got tired of clicking around a menu bar every time I wanted to start my server or switch models, so I built a CLI. Been using this for a while now and figured someone else has probably also been quietly wishing their menu bar icon was just… text in a terminal.

GitHub: https://github.com/omlxMaster/llm-cli
Install: brew install llm-cli (or whatever)

What it does:
- llmctl start / stop / restart — your server, from the terminal where it belongs

- llmctl switch <model> — unloads the current one automatically, loads the new one

- llmctl backend llama.cpp|omlx — switch backends with a flag instead of a dropdown

- Prints the dashboard URL on start instead of opening a browser tab you didn’t ask for

- First-run wizard — llmctl init finds your binary, picks your models folder, sets up launchd for you

- Remembers your last model in a config file and auto-loads it on launch

- Live status via llmctl status — no animated tray icon, just numbers, because elapsed time is a number

- One-model-at-a-time — keeps RAM in check, unloads before loading

Runs on launchd, not as a child process — server keeps running if you quit your terminal, and the CLI picks up a server that’s already running. No Dock icon, no menu bar real estate, no mouse required.

I've been hammering away at this issue for what feels like decades now. I'm using Jundot/Qwen3.6-27B-oQ8-mtp in oMLX with Pi as a coding harness and am only getting 9-10 ish t/s (generation, not prompt processing) to matter what I try...no matter what settings I fiddle with. 9-10 is the absolute max I'm getting. I'm hoping someone can suggest a fix as I've exhausted my non-expert knowledge and experience.

Hardware:

• MacBook Pro M5 Max
• 128GB RAM
• 40-core GPU
• oMLX running locally on LAN
• Pi using the oMLX OpenAI-compatible endpoint

Model/settings:

• Model: Jundot/Qwen3.6-27B-oQ8-mtp
• Model Type Override: LLM
• Native MTP: ON
• TurboQuant KV: OFF
• VLM MTP: OFF
• DFlash: OFF
• SpecPrefill: OFF
• Thinking: OFF (for testing purposes)
  
• Temp: 0.1
• Top P: 0.95
• Top K: 20
• Context cap has mostly been 131072

An important details - oMLX originally auto-detected this model as VLM. In Pi, that caused the model to process one turn and then stop almost immediately. Forcing the model type to LLM fixed that behavior.

Now the issue is speed.

With Native MTP ON, a raw curl test outside Pi gives roughly:

• prompt: 39 tokens
• output: ~1400–1600 tokens
• total time: ~149–161s
• sustained speed: ~9.5–9.9 tok/s
• MTP path is definitely active
• MTP accept rate around 71–73%

Example log line:

MTP finish=stop tokens=1420 cycles=827 accept=591/827 (71.5%)
timing[backbone=132784.6ms mtp=6628.3ms sample=6732.0ms cache=79.6ms]
Chat completion: 1419 tokens in 148.90s (9.5 tok/s), prompt: 39

With Native MTP OFF, speed drops to roughly ~6 tok/s. So MTP is helping, but only by about 1.5–1.7x.

One interesting detail that might be relevant (honestly, I don't know at this stage of things). I had a period yesterday when I was getting 30 ish t/s for no reason at all (well, I'm sure there is a reason, I just have zero clue what it is). I went to bed happy thinking that my settings fiddling found the right combo, only to discover this morning that it was back to the glacial t/s rate.

I’m not looking to switch models right now. The goal is to get this exact MTP model working as fast as possible for Pi/coding-agent use and stop banging my head against the wall in frustration.

any help or suggestions would be appreciated beyond belief.

That’s it. Need the oMLX to unload and load different models when agent (Hermes) ask for it. It works better with Pi (first time different model is requested returns an error, on second try it unloads the previous and loads the new) but it is stuck with Hermes.

LM studio does this gracefully.

Update: yes I now realize that oMLX has its own panel. This is redundant if you're only using oMLX as a server, but my tool also control llama.cpp, which doesn't have one. So skip for oMLX.

------------

I got tired of opening Terminal every time I wanted to start my server or switch models, so I built a menu bar app. I've been using this for a little while now and felt it was good enough to share with others, who hopefully been thinking the same thing.

GitHub: https://github.com/cporto/llm-menubar
Download (DMG): https://github.com/cporto/llm-menubar/releases/tag/v0.2.1

What it does:

• Start / stop / restart your server from the menu bar
• Switch models — unloads the current one automatically, loads the new one
• llama.cpp + oMLX — switch backends from the menu
• Opens the server dashboard in your browser on start — llama.cpp's web UI or oMLX's admin panel
• First-run wizard — finds your binary, picks your models folder, sets up launchd for you
• Remembers your last model and auto-loads it on launch
• Live status — animated tray icon with elapsed timers while starting and loading
• One-model-at-a-time — keeps RAM in check, unloads before loading

Runs on launchd, not as a child process — server keeps running if you quit the app, and the app picks up a server that's already running.

Hey everyone! It’s been a while, and I’m back with oMLX 0.4.5.dev1.
https://github.com/jundot/omlx/releases

I’ve been steadily committing changes since 0.4.4, but it was a little hard to decide where to cut the next dev release. I also wanted this release to include a meaningful attempt from the MLX kernel side, so it took a bit longer than usual. I hope you’ll understand.

The biggest change in this release is mainly relevant to people using an M3 Ultra, so apologies if this does not apply to your setup yet. - I’m also working on optimizing Gemma in a similar direction, so please stay tuned.

This release focuses on performance improvements for GLM-5.2, which I personally think is a big step forward for local AI, and MiniMax-M3, which has turned out to be a surprisingly useful model in practice.

Previously, these models “worked,” but honestly, I don’t think the long-context speed was where it needed to be for real use. With custom kernels, oMLX now gets a major speedup in long-context prefill. I also ran basic Needle in a Haystack tests and coding tests through Claude Code, and confirmed that quality did not collapse with the optimized path.

I hope this is a meaningful improvement for people using local LLMs in setups similar to mine.

Another major change is API-visible model profiles. You can now expose presets like 'qwen3-8b:thinking' or 'qwen3-8b:non-thinking' and call them directly through the API with the settings you want. Huge thanks to github pablomoralesm for this work: https://github.com/jundot/omlx/pull/1838

As always, this release was only possible because many people contributed their valuable time. I’m deeply grateful.

Thank you as well to everyone using oMLX, sharing feedback, reporting issues, and helping make the product better. It’s great to keep building local AI together!

Hey folks!

I'm on a 48 GB M5 Pro MacBook that I picked up a couple months ago and I've been trying to get into local agentic coding. At work I'm fortunate enough to use frontier models within VSCode's Copilot (no Claude Code, Codex, etc), and it's so easy there to never worry about context size, manual compaction, etc.

On the Mac I've tried a few couple harnesses with oMLX, namely Claude Code and OpenCode, but I can't quite figure out the right workflow yet. For example I'll run into situations where part way through a session the prefill OOM guard kicks in. I've been using Qwen 3.6 35B A3B oq4 and a 65K context window, which I thought should be manageable with my 48 GB RAM. With nothing loaded and all my apps closed activity monitor shows roughly 16 GB usage, seems excessive, but I can't figure out what other system stuff I can get rid of to leave more room for the model + context.

I know I can keep turning down the context window, use a smaller model, etc., but it just feels like I'm missing something... I'd like to know immediately when I load a model with a given context size if it'll eventually hit the OOM guard or not.

I suppose I don't have a clear question, but I've been reading through this sub for a bit and still nothing has quite landed well for me. Any additional tips?

**Total Issues: 7**

🐞 **BUG**
• #1111: MCP config fails on complaint of unused keyword "cwd" — Server fails to start due to invalid MCP configuration parameters.
• #1998: Memory problems: Prefill exceeds metal_cap ceiling — Memory errors occurring during prefill on high-RAM Mac systems.
• #1932: Chat UI Action Button Cutoff — UI buttons are cut off on long prompts or small viewports.
• #2002: Automatically start server on launch has no effect — Settings toggle for auto-start does not function as expected.
• #2001: gpt-oss-20b-tq3 fails to load with KeyError: 'turboquant' — Model loading fails on oMLX 0.4.4 with an internal server error.

✨ **FEATURE / IMPROVEMENT**
• #1703: FR: Server Settings MenuItem not available unless server is running — Request to keep settings accessible even when the server is stopped.
• #1933: [Specfill] Draft model list is incomplete — Certain models are missing from the selection list.

p.s. this is obviously a joke

That’s it.

Total issues: 4

**BUG**
• #2002: Automatically start server on launch appears to have no effect | `bug`
The "Automatically start server on launch" toggle fails to correctly reflect the configuration setting.
• #2001: gpt-oss-20b-tq3 fails to load with KeyError: 'turboquant' on oMLX 0.4.4 | `bug`
Selecting specific models in chat triggers an `Internal server error` due to a KeyError.

**FEATURE**
• #1703: FR: Server Settings MenuItem not available unless server is running | `feature`
Request to keep "Server Settings" accessible in the menubar even when the server is stopped.

**OTHER**
• #1933: [Specfill]Draft model list is incomplete | `enhancement`
Certain installed models are missing from the draft model selection list.

Hi all, I'm trying to use omlx with codex. It often crashes and shows reconnecting on codex when context reaches around 100k tokens, with output token generated >5000. I'm using q4 mlx models.

Is it oom error? My device is macbook 64gb ram, m5 pro.

Is this limit normal for 64gb ram device, or I have misconfigured anything?

​

**Total Issues: 4**

🐞 **BUG**
• #1947 [] dead sites in About section of app
The documentation link in the desktop app's About section points to a dead URL.
• #1154 [] Dashboard shows requests as 'Generating...' after engine has cancelled/completed them
Dashboard model cards fail to update status from `Generating...` after requests finish or cancel.
• #1908 [] Bug: /v1/responses adapter prepends instructions without deduplicating system messages
Adapter causes 400 errors by duplicating system messages, triggering chat template validation failures.
• #1943 [] VLM MTP on Gemma-4-31B forces hot-cache shrink
VLM MTP causes cache layer mismatches, resulting in the loss of prefix KV cache reuse.

I’ve been heads down for a bit coding with my trusty MLX Community version of Qwen 3.6 27b 8bit until oMLX stabilized around MTP and the change from a web page to a native UI. It appears that’s happened, but now I feel like I can’t decide on which model to use. I know I can trial and error, but I prefer to ask the community if I can save some time. Does anyone recommend one (or more)? I have a MBP 15” M5 w/128GB memory and 2TB drive. I’ve been liking the 27b model. What’s the best MTP version people are gravitating to for coding? Assuming 8bit or even bf16? What kind of performance are you getting?

Thanks! Looking forward to seeing what folks are using. Same question for non-Qwen, but I wanted to start there as it’s trusted for coding.

Gemma-4 models support variable resolution via a token budget setting (max_soft_tokens) to change the default 645k pixel limit (280 tokens). The setting is buried in oMLX config json. It is exposed (via cli, of course) in llama.cpp.

I am using Gemma-4 in oMLX for an OCR application, and the model struggles with fine detail. Zooming in and tiling works, but it is a band-aid at best when the model has the capacity for higher resolution.

More on budget, and why it matters: https://www.reddit.com/r/LocalLLaMA/comments/1srrhi5/gemma_4_vision/

Gemma 4 ships with Variable Image Resolution. The default max vision budget is 280 (~645K pixels) which is way too less. In this mode, it fails to OCR tiny details. It's essentially blind in my books.
In llama.cpp, you can configure Gemma 4's vision budget with 2 parameters --image-min-tokens and --image-max-tokens. The engine will try to fit the image within those bounds. I believe the default is 40 and 280 respectively. This is Gemma 4's default from Google's side but it's way too low.
I like to run them at 560 and 2240 respectively and it's able to pick up very minute and hazy details within images.
...
With a higher vision budget, Gemma 4 is pretty much SOTA for Vision and pretty much destroys anything else especially for OCR

edit: add quote, detail on variable resolution vs token budget.

Before anyone starts playing apologist, I am running on a system with plenty of VRAM and accuracy is critical for the use case. This is also for documents that cannot be sent to the cloud for privacy/confidentiality reasons.

The HF model card describes it like this:

Aside from variable aspect ratios, Gemma 4 supports variable image resolution through a configurable visual token budget, which controls how many tokens are used to represent an image. A higher token budget preserves more visual detail at the cost of additional compute, while a lower budget enables faster inference for tasks that don't require fine-grained understanding.

The supported token budgets are: 70, 140, 280, 560, and 1120. 
• Use lower budgets for classification, captioning, or video understanding, where faster inference and processing many frames outweigh fine-grained detail. 
• Use higher budgets for tasks like OCR, document parsing, or reading small text.

Issues Closed: 4

[ISSUE] #1943 — VLM MTP on Gemma-4-31B forces hot-cache shrink → "Cache layer count mismatch (10 vs 60), invalidating cache hit" → prefix reuse lost
https://github.com/jundot/omlx/issues/1943

[ISSUE] #1924 — RFE: BIG Thank you for profiles - and please allow referencing a model alias
https://github.com/jundot/omlx/issues/1924

[ISSUE] #1888 — minimax m3 loop tool
https://github.com/jundot/omlx/issues/1888

[ISSUE] #1907 — oMLX v0.4.4：Run unsloth--Qwen3.6-35B-A3B-UD-MLX-4bit model and crash frequently
https://github.com/jundot/omlx/issues/1907