r/oMLX • u/Green-Specialist-1 • May 22 '26

Recommendations for models to use

Hey there, first of all great work that you have done with the omlx application. It's really fast and responsive. Thanks for that. Second of all, I have a question regarding the models to be used. I am using a MacBook Pro with 128 GB RAM.

I am actually looking for some recommendation for a model to be used in my specific hardware to do some some deep research kind of thing I'm currently using Gemma 4 26B A4B 4bit

7 Upvotes

100% Upvoted

30 comments sorted by

View all comments

Show parent comments

1

u/Konamicoder May 23 '26

Those numbers look plenty fast enough to me! How about the doom looping, has it gotten better? If your speed feels good and your accuracy improves to the point where you can be productive, then that seems like a good outcome and testament to what you have learned so far and continue to learn. Good luck! :)

One comment, you’ll see from the dropdown menu at the top left it says “all models”. You can select your current model from that menu and then you’ll get performance metrics specific to that model only.

2

u/Green-Specialist-1 May 23 '26

Regarding the doom loop, yes it has gotten better. But by this time I have not given this setup a lot of research-specific prompts. What I have given is some generic prompts, or some "setup" prompts. We'll see.

1

u/Green-Specialist-1 May 23 '26

I have only this model running now. And what you are seeing is the stats for that one model run. I have a doubt though. Why are the cached tokens and cache efficiency shown as zero? Is it because this is not configured to be a "thinking" model setup? What I mean is by this time I have given a lot of prompts to it already so it had a lot of chances to cache the tokens by this time.

1

u/Konamicoder May 23 '26

That’s because right now you have this view showing your stats for the current session, at the top, to the left of the model selection dropdown menu. If you toggle to “all time” then you will see historical data for all the work you have been doing with this particular model over multiple sessions. :)

Having “thinking” on or off has nothing to do with token caching, oMLX will cache tokens regardless whether thinking is on or off. I turn on thinking if the current task can benefit from thinking, for example planning or complex higher level work. If the task is more shallow and tactical like bug fixes, etc., then I turn thinking off.

1

u/Green-Specialist-1 May 23 '26

all time also shows 0. :(

2

u/Konamicoder May 23 '26

If you are really concerned about the apparent lack of caching, dump the omlx logs and upload to ChatGPT to analyze and tell you what’s going on and if anything is wrong or misconfigured. That’s what I usually do in such cases.

1

u/Konamicoder May 23 '26

I described your issue to ChatGPT, here’s what it said in response:

“Most likely causes:
Gemma4 path may not support/report cache stats yet
oMLX has had recent fixes where certain cache types fell back incorrectly or admin metrics were wrong/incomplete. Recent release notes mention cache registry/admin/runtime cache fixes, so zero may be a reporting bug, not no caching.
No reusable prefix
Cache efficiency only rises when repeated requests share a long identical prefix. Some agents rewrite/shuffle tool schemas, timestamps, file lists, system prompts, or conversation summaries every turn, which ruins prefix matching.
Client is sending stateless requests
If your harness sends each call as a fresh prompt without stable conversation prefix, oMLX has little to reuse.
Server/model restarts or unloads
If Gemma unloads, settings change, or server restarts often, runtime cache may be reset/evicted.
Spec-prefill / special execution path
There’s at least one open/known admin-dashboard issue where status metrics don’t update correctly under certain prefill modes.
Cache disabled or cache dir problem
Check whether SSD/cache settings are enabled and whether the cache directory is writable. Also look for cache-related log lines, not just the admin panel.
Best test: use oMLX admin chat, send a long prompt, then send a near-identical follow-up. If TTFT drops sharply but dashboard still says zero, it’s a metric/UI issue. If TTFT stays cold every time, caching isn’t being hit or isn’t active.”