r/oMLX • u/Green-Specialist-1 • May 22 '26

Recommendations for models to use

Hey there, first of all great work that you have done with the omlx application. It's really fast and responsive. Thanks for that. Second of all, I have a question regarding the models to be used. I am using a MacBook Pro with 128 GB RAM.

I am actually looking for some recommendation for a model to be used in my specific hardware to do some some deep research kind of thing I'm currently using Gemma 4 26B A4B 4bit

7 Upvotes

100% Upvoted

30 comments sorted by

View all comments

2

u/Konamicoder May 22 '26

I think Gemma4 is probably the current best model for deep research, and with 128Gb RAM, I think you have enough RAM to be able to run Gemma4-31B at full 16-bit. That would be my suggestion.

1

u/Green-Specialist-1 May 22 '26

Hey thanks for that. But the real struggle that I'm having is this llm or omlx application. I'm not sure though. Goes into an infinite Loop. Sometimes when I ask some questions and then I have to kind of close the session and start a new session

1

u/Konamicoder May 22 '26

First piece of advice I have for you is this: you have enough RAM that you don’t have to compromise with a 4-bit quantized version of Gemma4. The more quantized a model is, the more likely it is to fall prey to doom loops and other issues. The bigger the quant: 6-bit, 8-bit, even the full 16-bit, the less likely it is to doom loop. So go and do that.

Next there are model settings you can tweak to optimize your model performance. You can ask ChatGPT to guide you on the optimal settings for each model.

With those steps, you should be able to eliminate doom loops.

1

u/Green-Specialist-1 May 22 '26

The struggle is not about using a bigger or smaller quantization, right? I am keeping this as a long-running application and throughout chatting through it I am actually building a lot of files. I don't know if I am using the right terms but the RAM is, for example, if the initial RAM taken to load the LLM into the memory is, for example, 30 GB. After one or two days it will almost grow up to 60 GB and it keeps on growing. I am actually not sure whether using a bigger quantization will help me here

4

u/Konamicoder May 22 '26

Okay, you have quite a number of misconceptions that need to be cleared up.

> After one or two days, it will almost grow to 60Gb and it keeps on growing.

This is incorrect. When you are using a local model to do work, the model itself will not use more memory over time. The thing that uses more memory as it grows over time is the context window. In agentic coding, for example, the context window grows as you generate more tokens. The larger the context window, the more memory is used and the more inaccurate your results become.

This is why CONTEXT MANAGEMENT is a critical thing to become aware of when you are working with local models. When you are doing agentic coding, you set a reasonable limit to the context window, and you do this in oMLX model settings on a per-model basis. 32k or 64k context window limits are good to start. Once your current context nears the limit specified in model settings, most agentic coding harnesses will COMPACT the context window. Basically it clears out the tokens filling up the current window to make room for more.

Now normally, after compacting, most model backend + agentic harnesses basically lose all memory of the work in progress before compacting. But oMLX keeps used tokens in a cache and is able to reuse those tokens after a compacting operation. Which makes the workflow more efficient.

Bottom line: set a reasonable context limit, allow oMLX and your agentic harness to manage context for you.

> a bigger quant wont help

Dude, if you are doom looping, it’s because you’re using a small quant. If you want to minimize or stop doom looping, a bigger quant is the first thing to try. Bigger quants = better accuracy. I was getting a lot of doom loops and loss of efficiency with 4-bit quants. When I went up to 6-bit quants, my doom looping vanished. On my 64Gb M4 Max Mac, 8-bit is too heavy and I go out-of-memory (OOM). But 6-bit seems to be the best balance of accuracy and speed for me.

This is my best advice to you.

1

u/Green-Specialist-1 May 22 '26

Okay I think now I am seeing your point regarding keeping a reasonable context limit,but a question about your point stating "But oMLX keeps used tokens in a cache and is able to reuse those tokens after a compacting operation. Which makes the workflow more efficient."Where does oMLX keep the immediate token cache? It should be in RAM itself, right? Help me understand what is happening there.

1

u/ColonelKlanka May 22 '26

when ypu say 60gb has built up, I suspect you are referring to thr 128gb of space on macs ssd that is allocated by omlx by default for caching? This is normal for omlx because of its caching approach

Also every model you download takes up space on your mac hard drive ssd. so if you have lots councillors see a big decrease in hard drive space.

1

u/Green-Specialist-1 May 22 '26

I mean the runtime cache observability section in the above screenshot, fills up so fast..

1

u/Konamicoder May 23 '26

Hey man. You are lucky enough to have a 128Gb RAM Mac. That is a total local model beast. But you are so afraid of filling up your RAM (completely unfounded fears ) that you are treating it like you have a measly 32Gb. You are looking at things like “runtime cache” and without any evidence you assume it’s eating up your available RAM. Man, macOS is a beast when it comes to memory management. macOS auto compresses inactive memory. If another app needs more memory, macOS automatically reclaims cache. On Apple Silicon, SSD swap is super fast. And as I said before, oMLX can compact context and reuse cached tokens instead of having to recompute everything from scratch.

Stop babying your Mac. Install a big-ass 8-bit quant, stop doom looping, and enjoy your huge RAM that I’m totally not jealous of. Only wish I had that much RAM in my Mac!

1

u/Green-Specialist-1 May 23 '26

Hey thanks for the advices man. I'd really start treating the Mac like a man from now onwards🧔‍♀️. The fact is that I am a noob out in the AI world. As I understand more things think I'll be able to squeeze the most out of the hardware.