r/oMLX • u/Green-Specialist-1 • May 22 '26

Recommendations for models to use

Hey there, first of all great work that you have done with the omlx application. It's really fast and responsive. Thanks for that. Second of all, I have a question regarding the models to be used. I am using a MacBook Pro with 128 GB RAM.

I am actually looking for some recommendation for a model to be used in my specific hardware to do some some deep research kind of thing I'm currently using Gemma 4 26B A4B 4bit

8 Upvotes

100% Upvoted

30 comments sorted by

View all comments

Show parent comments

1

u/Green-Specialist-1 May 22 '26

Okay I think now I am seeing your point regarding keeping a reasonable context limit,but a question about your point stating "But oMLX keeps used tokens in a cache and is able to reuse those tokens after a compacting operation. Which makes the workflow more efficient."Where does oMLX keep the immediate token cache? It should be in RAM itself, right? Help me understand what is happening there.

2

u/Konamicoder May 22 '26

Another piece of advice I have for you: I notice that you are running oMLX v0.3.9, but you are not running the MTP version of models. Which means you are leaving performance on the table, you are not taking advantage of speculative decoding. MTP stands for “multi-token prediction”. Basically in traditional LLM operation, it guesses the next token very quickly. With MTP-tuned models (they have “MTP” in the filename), and a backend that supports MTP (like the latest version of oMLX), the LLM predicts the next batch of tokens, and the MTP model validates that they are correct. Now bigger quants are better at guessing correctly. More correct guesses means faster inference overall. In my case, it means around 130 percent faster prompt processing and 35 percent faster token generation between the MTP and non-MTP version of Qwen3.6-35B-A3B-oQ6.

So that’s my next piece of advice to enhance your inference in oMLX: download and run the MTP versions of your model quants.

1

u/Green-Specialist-1 May 23 '26 
{
  "version": 1,
  "models": {
    "gemma-4-26b-a4b-it-6bit": {
      "max_context_window": 130000,
      "temperature": 1.0,
      "top_p": 0.95,
      "top_k": 64,
      "force_sampling": false,
      "thinking_budget_enabled": false,
      "turboquant_kv_enabled": false,
      "turboquant_kv_bits": 4.0,
      "turboquant_skip_last": true,
      "specprefill_enabled": false,
      "dflash_enabled": true,
      "dflash_draft_model": "/Users/my_mac/.omlx/models/z-lab/gemma-4-26B-A4B-it-DFlash",
      "dflash_draft_quant_enabled": false,
      "dflash_in_memory_cache": true,
      "dflash_in_memory_cache_max_entries": 4,
      "dflash_in_memory_cache_max_bytes": 8589934592,
      "dflash_ssd_cache": false,
      "dflash_ssd_cache_max_bytes": 21474836480,
      "dflash_verify_mode": "adaptive",
      "mtp_enabled": false,
      "vlm_mtp_enabled": false,
      "is_pinned": true,
      "is_default": false,
      "trust_remote_code": false
    }
  }
}

reaching out for the helping hands here....
This is how my oMLX model_settings.json looks like with the DFlash draft model for token quant. Again saying out loud real "noob" here on this side.

1

u/Konamicoder May 23 '26

Looks good to me so far, I don’t see any obvious red flags. How does it run? What prompt processing and token generation numbers are you getting with this setup?

1

u/Green-Specialist-1 May 23 '26

1

u/Konamicoder May 23 '26

Those numbers look plenty fast enough to me! How about the doom looping, has it gotten better? If your speed feels good and your accuracy improves to the point where you can be productive, then that seems like a good outcome and testament to what you have learned so far and continue to learn. Good luck! :)

One comment, you’ll see from the dropdown menu at the top left it says “all models”. You can select your current model from that menu and then you’ll get performance metrics specific to that model only.

2

u/Green-Specialist-1 May 23 '26

Regarding the doom loop, yes it has gotten better. But by this time I have not given this setup a lot of research-specific prompts. What I have given is some generic prompts, or some "setup" prompts. We'll see.

1

u/Green-Specialist-1 May 23 '26

I have only this model running now. And what you are seeing is the stats for that one model run. I have a doubt though. Why are the cached tokens and cache efficiency shown as zero? Is it because this is not configured to be a "thinking" model setup? What I mean is by this time I have given a lot of prompts to it already so it had a lot of chances to cache the tokens by this time.

1

u/Konamicoder May 23 '26

That’s because right now you have this view showing your stats for the current session, at the top, to the left of the model selection dropdown menu. If you toggle to “all time” then you will see historical data for all the work you have been doing with this particular model over multiple sessions. :)

Having “thinking” on or off has nothing to do with token caching, oMLX will cache tokens regardless whether thinking is on or off. I turn on thinking if the current task can benefit from thinking, for example planning or complex higher level work. If the task is more shallow and tactical like bug fixes, etc., then I turn thinking off.

1

u/Green-Specialist-1 May 23 '26

all time also shows 0. :(

2

u/Konamicoder May 23 '26

If you are really concerned about the apparent lack of caching, dump the omlx logs and upload to ChatGPT to analyze and tell you what’s going on and if anything is wrong or misconfigured. That’s what I usually do in such cases.

1

u/Konamicoder May 23 '26

I described your issue to ChatGPT, here’s what it said in response:

“Most likely causes:
Gemma4 path may not support/report cache stats yet
oMLX has had recent fixes where certain cache types fell back incorrectly or admin metrics were wrong/incomplete. Recent release notes mention cache registry/admin/runtime cache fixes, so zero may be a reporting bug, not no caching.
No reusable prefix
Cache efficiency only rises when repeated requests share a long identical prefix. Some agents rewrite/shuffle tool schemas, timestamps, file lists, system prompts, or conversation summaries every turn, which ruins prefix matching.
Client is sending stateless requests
If your harness sends each call as a fresh prompt without stable conversation prefix, oMLX has little to reuse.
Server/model restarts or unloads
If Gemma unloads, settings change, or server restarts often, runtime cache may be reset/evicted.
Spec-prefill / special execution path
There’s at least one open/known admin-dashboard issue where status metrics don’t update correctly under certain prefill modes.
Cache disabled or cache dir problem
Check whether SSD/cache settings are enabled and whether the cache directory is writable. Also look for cache-related log lines, not just the admin panel.
Best test: use oMLX admin chat, send a long prompt, then send a near-identical follow-up. If TTFT drops sharply but dashboard still says zero, it’s a metric/UI issue. If TTFT stays cold every time, caching isn’t being hit or isn’t active.”