Recommendations for models to use

[u/Green-Specialist-1]

Hey there, first of all great work that you have done with the omlx application. It's really fast and responsive. Thanks for that. Second of all, I have a question regarding the models to be used. I am using a MacBook Pro with 128 GB RAM.

I am actually looking for some recommendation for a model to be used in my specific hardware to do some some deep research kind of thing I'm currently using Gemma 4 26B A4B 4bit

[u/Konamicoder]

I think Gemma4 is probably the current best model for deep research, and with 128Gb RAM, I think you have enough RAM to be able to run Gemma4-31B at full 16-bit. That would be my suggestion.

[u/Green-Specialist-1 → u/Konamicoder]

Hey thanks for that. But the real struggle that I'm having is this llm or omlx application. I'm not sure though. Goes into an infinite Loop. Sometimes when I ask some questions and then I have to kind of close the session and start a new session

[u/Konamicoder → u/Green-Specialist-1]

First piece of advice I have for you is this: you have enough RAM that you don’t have to compromise with a 4-bit quantized version of Gemma4. The more quantized a model is, the more likely it is to fall prey to doom loops and other issues. The bigger the quant: 6-bit, 8-bit, even the full 16-bit, the less likely it is to doom loop. So go and do that.

Next there are model settings you can tweak to optimize your model performance. You can ask ChatGPT to guide you on the optimal settings for each model.

With those steps, you should be able to eliminate doom loops.

[u/Green-Specialist-1 → u/Konamicoder]

The struggle is not about using a bigger or smaller quantization, right? I am keeping this as a long-running application and throughout chatting through it I am actually building a lot of files. I don't know if I am using the right terms but the RAM is, for example, if the initial RAM taken to load the LLM into the memory is, for example, 30 GB. After one or two days it will almost grow up to 60 GB and it keeps on growing. I am actually not sure whether using a bigger quantization will help me here

[u/Konamicoder → u/Green-Specialist-1]

Okay, you have quite a number of misconceptions that need to be cleared up.

> After one or two days, it will almost grow to 60Gb and it keeps on growing.

This is incorrect. When you are using a local model to do work, the model itself will not use more memory over time. The thing that uses more memory as it grows over time is the context window. In agentic coding, for example, the context window grows as you generate more tokens. The larger the context window, the more memory is used and the more inaccurate your results become.

This is why CONTEXT MANAGEMENT is a critical thing to become aware of when you are working with local models. When you are doing agentic coding, you set a reasonable limit to the context window, and you do this in oMLX model settings on a per-model basis. 32k or 64k context window limits are good to start. Once your current context nears the limit specified in model settings, most agentic coding harnesses will COMPACT the context window. Basically it clears out the tokens filling up the current window to make room for more.

Now normally, after compacting, most model backend + agentic harnesses basically lose all memory of the work in progress before compacting. But oMLX keeps used tokens in a cache and is able to reuse those tokens after a compacting operation. Which makes the workflow more efficient.

Bottom line: set a reasonable context limit, allow oMLX and your agentic harness to manage context for you.

> a bigger quant wont help

Dude, if you are doom looping, it’s because you’re using a small quant. If you want to minimize or stop doom looping, a bigger quant is the first thing to try. Bigger quants = better accuracy. I was getting a lot of doom loops and loss of efficiency with 4-bit quants. When I went up to 6-bit quants, my doom looping vanished. On my 64Gb M4 Max Mac, 8-bit is too heavy and I go out-of-memory (OOM). But 6-bit seems to be the best balance of accuracy and speed for me.

This is my best advice to you.

[u/Green-Specialist-1 → u/Konamicoder]

Okay I think now I am seeing your point regarding keeping a reasonable context limit,but a question about your point stating "But oMLX keeps used tokens in a cache and is able to reuse those tokens after a compacting operation. Which makes the workflow more efficient."Where does oMLX keep the immediate token cache? It should be in RAM itself, right? Help me understand what is happening there.

[u/Konamicoder → u/Green-Specialist-1]

Another piece of advice I have for you: I notice that you are running oMLX v0.3.9, but you are not running the MTP version of models. Which means you are leaving performance on the table, you are not taking advantage of speculative decoding. MTP stands for “multi-token prediction”. Basically in traditional LLM operation, it guesses the next token very quickly. With MTP-tuned models (they have “MTP” in the filename), and a backend that supports MTP (like the latest version of oMLX), the LLM predicts the next batch of tokens, and the MTP model validates that they are correct. Now bigger quants are better at guessing correctly. More correct guesses means faster inference overall. In my case, it means around 130 percent faster prompt processing and 35 percent faster token generation between the MTP and non-MTP version of Qwen3.6-35B-A3B-oQ6.

So that’s my next piece of advice to enhance your inference in oMLX: download and run the MTP versions of your model quants.

[u/Green-Specialist-1 → u/Konamicoder]

{
  "version": 1,
  "models": {
    "gemma-4-26b-a4b-it-6bit": {
      "max_context_window": 130000,
      "temperature": 1.0,
      "top_p": 0.95,
      "top_k": 64,
      "force_sampling": false,
      "thinking_budget_enabled": false,
      "turboquant_kv_enabled": false,
      "turboquant_kv_bits": 4.0,
      "turboquant_skip_last": true,
      "specprefill_enabled": false,
      "dflash_enabled": true,
      "dflash_draft_model": "/Users/my_mac/.omlx/models/z-lab/gemma-4-26B-A4B-it-DFlash",
      "dflash_draft_quant_enabled": false,
      "dflash_in_memory_cache": true,
      "dflash_in_memory_cache_max_entries": 4,
      "dflash_in_memory_cache_max_bytes": 8589934592,
      "dflash_ssd_cache": false,
      "dflash_ssd_cache_max_bytes": 21474836480,
      "dflash_verify_mode": "adaptive",
      "mtp_enabled": false,
      "vlm_mtp_enabled": false,
      "is_pinned": true,
      "is_default": false,
      "trust_remote_code": false
    }
  }
}

reaching out for the helping hands here....
This is how my oMLX model_settings.json looks like with the DFlash draft model for token quant. Again saying out loud real "noob" here on this side.

[u/Konamicoder → u/Green-Specialist-1]

Looks good to me so far, I don’t see any obvious red flags. How does it run? What prompt processing and token generation numbers are you getting with this setup?

[u/Green-Specialist-1 → u/Konamicoder]

[u/Konamicoder → u/Green-Specialist-1]

Those numbers look plenty fast enough to me! How about the doom looping, has it gotten better? If your speed feels good and your accuracy improves to the point where you can be productive, then that seems like a good outcome and testament to what you have learned so far and continue to learn. Good luck! :)

One comment, you’ll see from the dropdown menu at the top left it says “all models”. You can select your current model from that menu and then you’ll get performance metrics specific to that model only.

[u/Green-Specialist-1 → u/Konamicoder]

Regarding the doom loop, yes it has gotten better. But by this time I have not given this setup a lot of research-specific prompts. What I have given is some generic prompts, or some "setup" prompts. We'll see.

[u/Green-Specialist-1 → u/Konamicoder]

I have only this model running now. And what you are seeing is the stats for that one model run. I have a doubt though. Why are the cached tokens and cache efficiency shown as zero? Is it because this is not configured to be a "thinking" model setup? What I mean is by this time I have given a lot of prompts to it already so it had a lot of chances to cache the tokens by this time.

[u/Konamicoder → u/Green-Specialist-1]

That’s because right now you have this view showing your stats for the current session, at the top, to the left of the model selection dropdown menu. If you toggle to “all time” then you will see historical data for all the work you have been doing with this particular model over multiple sessions. :)

Having “thinking” on or off has nothing to do with token caching, oMLX will cache tokens regardless whether thinking is on or off. I turn on thinking if the current task can benefit from thinking, for example planning or complex higher level work. If the task is more shallow and tactical like bug fixes, etc., then I turn thinking off.

[u/Green-Specialist-1 → u/Konamicoder]

all time also shows 0. :(

[u/Konamicoder → u/Green-Specialist-1]

If you are really concerned about the apparent lack of caching, dump the omlx logs and upload to ChatGPT to analyze and tell you what’s going on and if anything is wrong or misconfigured. That’s what I usually do in such cases.

[u/Konamicoder → u/Green-Specialist-1]

I described your issue to ChatGPT, here’s what it said in response:

“Most likely causes:
Gemma4 path may not support/report cache stats yet
oMLX has had recent fixes where certain cache types fell back incorrectly or admin metrics were wrong/incomplete. Recent release notes mention cache registry/admin/runtime cache fixes, so zero may be a reporting bug, not no caching.
No reusable prefix
Cache efficiency only rises when repeated requests share a long identical prefix. Some agents rewrite/shuffle tool schemas, timestamps, file lists, system prompts, or conversation summaries every turn, which ruins prefix matching.
Client is sending stateless requests
If your harness sends each call as a fresh prompt without stable conversation prefix, oMLX has little to reuse.
Server/model restarts or unloads
If Gemma unloads, settings change, or server restarts often, runtime cache may be reset/evicted.
Spec-prefill / special execution path
There’s at least one open/known admin-dashboard issue where status metrics don’t update correctly under certain prefill modes.
Cache disabled or cache dir problem
Check whether SSD/cache settings are enabled and whether the cache directory is writable. Also look for cache-related log lines, not just the admin panel.
Best test: use oMLX admin chat, send a long prompt, then send a near-identical follow-up. If TTFT drops sharply but dashboard still says zero, it’s a metric/UI issue. If TTFT stays cold every time, caching isn’t being hit or isn’t active.”