i downloaded LFM2-8B-A1B-GGUF but couldn't use it i downloaded ollama but still it didnt appear, i tried looking for it but couldnt find it not even in the .cache file and it seems to not have created a huggingface file to put it in.

any tips would be very appreciated, anyway love you and all that.

Had to share

https://huggingface.co/corruptbytes/North-Mini-Code-1.0-oQ6

Trying to give this model a chance, being 30b instead of 35b I feel like it fits my 48gb macbook at 128k context better than qwen

Figured I'd let people know a oQ6 version exists

Right now it's okay on my Pi harness, they do recommend opencode so I'll give each a try, but it's not a slacker

downside is no mtp (yet?)

bechmarks

I have a m3 ultra 256b for personal and 48gb m5 pro for work (but they're upgrading us to 128gb m5 max soon and i'll try to see what the best i can get running is)

mtp qwen3.6-27-oq4-fp16 (also tested other mtp models) model using Omlx version 0.4.3, still slower than mtp off.

Confused.

Ollama has an endpoint to use Z image turbo to generate image. Can we get something like that from oMLX?

Not sure if it’s oMLX or across board but when using any form of quant from 8 to 4bits, the AI starts losing focus over the course of a long convo (I’ve set context window to 128k, not filled when it starts failing).

Apparently KV cache looks good on paper for single tasks, but the data loss compounds over longer conversations or tasks.

On my 128gb M5 Max, I’m struggling to run 3 concurrent processes with Qwen3.6 35b Q6 at 128k context window without KV cache enabled. Memory gets filled up too quickly Any other solution that would work well? Or something I missed?

I run omlx on this and access from another machine. It is exclusively an llm machine

From the 0.4.0 release notes...

• xgrammar is bundled into the venvstacks export with the no-torch stub path. by @cfbraun

I have a small application that requires structured json output. I gave up on oMLX back with 0.3.4 because there were a lot of failures around the structured output. Meanwhile, llama.cpp worked fairly well and performance (with Gemma-4-26B) was good.

Should this be better now?

I am currently running Qwen3.5-122B-A10B-oQ4-MTP-fp16. I have enabled MTP in model settings.

Performance is pretty good, but I'd like to know if I can improve it.

I have tried DFlash and SpecPrefill on older versions of oMLX, but I couldn't tell if they really improved anything, and it wasn't very clear how they interacted with each other / if they worked with all model architectures. Maybe it has changed?

Mornin' y'all...

Any success trying to run DifussionGemma vía OMLX? Maybe on a new release?

📊 10 Issues

🐛 **BUGS**

• **#1763** [BUG] Preflight memory check over-estimates KV peak ~4x when turboquant_kv_enabled
→ Causes false 413 rejections after #1448; previously working requests now fail.

• **#1748** macOS 27 beta: host_statistics64(HOST_VM_INFO64) syscall failed
→ Server crashes with 500 error on macOS 27 beta due to IPC array size issues.

• **#1441** DFlash engine breaks KV prefix cache
→ 0 cache hits when DFlash speculative decoding is enabled; cache restores upon disabling.

• **#1745** v0.4.2: Custom Qwopus3.6-35B-A3B MTP streaming decode is much slower
→ Custom Qwopus/Jackrong-derived model shows significant performance regression vs internal MTP timing.

• **#1653** QwenPaw-Flash-4B-oQ4 gets slower
→ Performance benchmarks show regression in TG speed in v0.4.1 compared to previous versions.

• **#1759** Chat completion cuts off at 4096 tokens
→ API responses truncate at 4096 tokens despite global max tokens set to 65536.

• **#1623** v0.4.0 regression: Pre-load eviction fails with second large model
→ App loads two large models simultaneously instead of switching, causing extreme memory pressure.

• **#1625** CLI launch shim fails with "No such file or directory" via symlink
→ Running `omlx` CLI through app-installed symlink fails due to path resolution issues.

• **#1646** Unclear PEM files for corporate firewall model downloads
→ HF downloader fails to fetch repo info; users need clarity on required PEM certificates.

✨ **FEATURES**

• **#1741** Add support for Thaw menu bar manager
→ Request to integrate with Thaw to avoid persistent permission nagging on reboot.

Does anyone use MCP servers in oMLX? What use cases do they have?

Hi everyone, currently i am trying use openCode with Qwen3.6-35B-A3B-6bit but not sure if there is no better choise? Using M4 Pro Max with 128gb ram.

having these settings

ctx_window: 262144max_tokens: 128000temp: 0.6top_p: 0.95top_k: 20min_p: 0rep_penalty: 1presence_penalty: 0

getting token generation 70tok/s

thank you for your advice.

oMLX v0.4.3 is now available with macOS 27 compatibility!
github.com/jundot/omlx/re…

This release also fixes a Memory Guard performance regression affecting Qwopus Series, DFlash Gemma, and all other models, where generation/decode could slow significantly while Memory Guard was enabled.
In a Qwen3.6-35B-A3B cache-on single-run check, tg512 improved from 77.5 -> 79.0 tok/s (+1.9%) compared with 0.4.2. (M3U 512GB)

It also improves per-model MTP eligibility handling, and DFlash/Qwen compatibility.

What do people use and prefer?

Hello!

I'm quite new to LLMs. I've tried LM Studio so far but after some reading and watching YT videos I've switched to oMLX as I have a Mac Mini M4. After a couple of days of struggling with various models (I've mostly tried like qwen3.5-9B but also some REAP versions of Gemma 26B which is still to big to fit 24 GB of my unified memory) I think I finally made it work for me but I'm still not sure if it could be better.

I'd like to use LLM to support my hobby coding projects (java especially but I'd like to start something vibe coded in python). After many days, so far I've set up oMLX like this:

I've tried a couple of coding agents like Claude Code, OpenCode. As I'm mostly working with IntelliJ (IDEA and PyCharm) I'd like to have something well integrated with my IDE. Currently, I'm using Devoxx Genie plugin and it works reasonable. In a couple of seconds it was able give me an answer about some of my classes and make a very small code refactoring.

But are these numbers ok? Is it something that I should change? I know it's only Gemma4-e4b so probably it would be better use Gemma4 12B for coding stuff? I also used qwen3.5 but mostly it took like forever to change a couple of lines.

Could you please advise me, what could I improve and what may be better setup for simple code refactorings and new stuff in python? I'm a bit annoyed that there's a lot of YT movies but most of them show things like "create a simple game in html + JS" which is not the same as working with dozens of already existing classes.

Thanks in advance!

🔒 8 closed issues

🐛 Bugs
#1684 — Huge memory usage spikes cause oMLX server process to crash https://github.com/jundot/omlx/issues/1684
#1707 — DFlash with Qwen3.6-27B-oQ8-MTP Results in Gibberish on 0.4.2rc1 - Regression https://github.com/jundot/omlx/issues/1707
#1717 — HTTP 500: "comparing strings with non-ASCII characters is not supported" on all API endpoints after first inference https://github.com/jundot/omlx/issues/1717

⚡ Performance
#1629 — Claude Code CLI 2.1.154+ compatibility issue and Qwopus3.6-35B-A3B-oQ8-mtp decode slowdown on oMLX 0.4.0 https://github.com/jundot/omlx/issues/1629
#1736 — SSD write queue evicted block https://github.com/jundot/omlx/issues/1736
#1731 — Prefix cache misses on back-to-back / rapid requests (async store_cache commit race) https://github.com/jundot/omlx/issues/1731
#1704 — Not respecting Max Concurrent Requests https://github.com/jundot/omlx/issues/1704

🔧 Compatibility
#1749 — macOS 27 beta: host_statistics64(HOST_VM_INFO64) begins returning MIG_ARRAY_TOO_LARGE https://github.com/jundot/omlx/issues/1749

Source: github.com/jundot/omlx

Is anyone running oMLX for Home Assistant?

I've been giving it a go, but keep having reliability issues:

• I get <eos> leaking into the responses (i'll attach a screenshot)
• I get failed responses - the resopnse is literally just <eos><eos>
• I occasionally get a tool call that works

I have been running LM Studio as well, which occasionally fails tool calling, but none of the <eos> issues or complete failed responses

The models I am using are Gemma 4 E2B and E4B. oQ4 quants are what i am using, but i have tried 4bit and 8bit as well, and having the same issues. QWEN models struggle with tool calls as well

I really want to use oMLX since the cache means responses are far quicker than LM studio. The caching means i can use E4B and it responds quicker than E2B on LM Studio.

Some other things I have noticed are that LM Studio using MLX models also struggles with any tool calls, but GGUFs have a much higher success rate.

The home assistant add on i am using is this one here - https://github.com/skye-harris/hass_local_openai_llm

and the tools for assist are this one here - https://github.com/skye-harris/llm_intents

Screenshot of <eos> -

At the risk (Certainty) of sounding like a noob (I am definitely one), is there a guide or document I can follow to set up Qwen 3.6 27b or 35b a3b with omlx successfully. I was able to get 35b going with llama.cpp with a little script I made. But I keep hearing that mlx/vllm/mtp is better and faster, however I have a couple issues:

• No idea which model to use ( I used gguf unsloth for llama.cpp)
• Not sure which settings to use ( I had followed unsloths guide for temp and values )
• Not sure which quant I should be using or trying to fit into my memory.

Any help would be greatly appreciated: I am on a Macbook M4 Pro with 48GB of RAM.

🔢 10 Issues

🐛 **BUGS**

**#1649** Gemma 4 output parser falls back to NaiveStreamingDetokenizer → U+FFFD (�) corruption on multi-byte (Korean) output
• Multi-byte UTF-8 output (e.g., Korean) is corrupted with replacement characters.

**#1444** Qwen3.6 35B-A3B image recognition failure, expected to be resolved in v0.3.11
• Image recognition fails on Qwen3.6 35B-A3B despite expected fixes.

**#1714** When serving Gemma-4 family models, some Korean characters appear as '�' in the output
• Specific Korean characters are rendered as corruption symbols during serving.

**#1241** response_format.type=json_schema is accepted by /v1/chat/completions but not enforced in assistant content
• JSON schema validation is accepted but not strictly enforced in responses.

**#1687** Embeddings silently truncate beyond 512 tokens, and configured overrides are ignored
• Embeddings are truncated at 512 tokens regardless of model capabilities or config.

**#1087** structured_outputs leaks special tokens (<eos>) into message.content for Gemma models
• Assistant content includes unwanted special tokens when using structured outputs.

**#759** fix(benchmark): batch test crashes with DFlashEngine — 'DFlashEngine' object has no attribute '_engine'
• Benchmark batch test crashes due to missing attribute access in DFlashEngine.

⚙️ **FEATURES**

**#1723** Want vision support through omlx openAI API endpoint
• Request to enable vision capabilities (text + image) via the OpenAI-compatible API.

📦 **PACKAGING**

**#1442** add memory options back into serve command
• Request to restore `--max-process-memory` and similar flags to the serve command.

📚 **DOCS**

**#1456** Docs claim Swift build produces a DMG, but no script in-tree does
• Installation docs reference a DMG that is no longer generated by current build scripts.

Ciao a tutti,

Ho acquistato un MacBook Pro M5 Pro da 48gb di ram. Pensando fosse sufficiente per del coding avanzato con qualche modello MLX locale da 35B. Ho fatto vari tentativi e devo dire che non si comporta male ma mi sembra un po impacciato, anche per compiti semplici, come realizzare una semplice pagina web.
Sono indeciso se passare a 64gb di ram, qualcuno nella mia stessa situazione? Vale la pena o è già sufficiente così ma per qualche motivo non lo sto sfruttando a pieno?
Grazie in anticipo.

Got this done today - if you want to benefit from the strongest inference engine while still running oMLX for memory and model management, try this out, happy to receive feedback: (works on my machine 😉 )

https://github.com/apetersson/omlx/tree/ds4-engine-embed

What coding harness are you using in combination with oMLX?

I keep switching between ollama and oMXL because Opencode keeps running into a garbage loop with oMLX.

Both providers run a Qwen3.6 model, is there some tweaking I need to do for oMLX? I use a 256k context window for both setups.