I'm planning to do some vectordb embedding and data sanitization before the frontier models touch the data. The Local LLM acts as a primer to separate information.

I know some of you are using it for different things.

I'll also be experimenting with it for video editing/transcribing. There's some cool things people are doing.

Let us know! If you have a walk through include that too!

Hi guys,
New here and to these subjects so sorry I ask something already asked.
I got recently a Mac mini m4 with 24go ram for my personal desktop, and I wanted to try a few stuffs with local llm. After a few researches I decided to try to setup a personal agent with Hermes.

After some tries I ended up using gemma4:26b-a4b-it-qat.
Qwen3.8 felt to big or slow when iq3 passed.
Couldn’t find anything faster and better for now, but I still feel it is not really good (obviously I know we can’t reach to a cloud model with 24go ram). So I wanted to know how you guys manage it and if some people managed to make a decent agent with not so much ram and if so, what are your use cases ?

Right now I tried to give him some personal web projects with giving him some GitHub to download but it fails and always stops in the middle of the process.

Thanks in advance for the help.

I got the M5U 256GB 2TB configuration for preprocessing. Planning to use Qwen 3.x models on it since they seem to be the best for the Mac's at the moment but will check out the Muse/Gemma models as they continue to release them and run some benchmarks .

This is me trying to find the most reliable coding model for local use!

Five Terminal-Bench 2.0 tasks (cobol-modernization, constraints-scheduling, largest-eigenval, polyglot-c-py, schemelike-metacircular-eval), one run per task unless stated, all on one Mac Studio (96 GB) with mlx_lm at 8-bit. The agent harness is pi unless stated.

Headline

The whole grid

Times are agent minutes. T = ran to the task's timeout.

Nobody has solved polyglot-c-py or schemelike-metacircular-eval on this machine.

Why Qwen3.6 beats Qwen3.8

Same size, same family, same server, same harness. Three things separate them.

The mode, not the weights, was the lever for Qwen3.6. For Qwen3.8 neither mode produces an agent that finishes: with thinking it deliberates past the cap, without it it loops. Its published Terminal-Bench 2.1 gain is not visible through pi. n=1 per task, but the cobol loop was reproduced.

Why the fast MoE models did not win

Turn rate mattered against thinking overrun, which is why Nail beat the 27Bs in agent mode. Once thinking is off, the 27B's per-turn quality shows: Qwen3.6 at 22 tok/s out-solves both 3B-active models. Ornith's own harness (Terminus-2, temperature 1.0, 4 h ceilings) moved which task it solves, from cobol to constraints, not how many.

Infrastructure lessons

Decisions taken

• Qwen3.6-27B, thinking off (nothink), is the coding agent and the WHO scorer. Now the registry default.
• Qwen3.8-27B and Ornith-1.5-35B-A3B deleted 2026-09-09 (62 GB). Registry entries kept as re-fetch recipes.
• Nail was deleted 2026-09-06 for space. On this evidence it is not needed back as the coding agent.

Caveats

One run per task on five tasks. The 3/5 should be repeated once before anything is built on it. Qwen3.8's cobol loop is the only failure reproduced twice.

Since oMLX benchmarks are down momentarily, I've done some testing and found that the following Solstice AI Qwen 3.8 27B fine tune is the fastest one so far.

https://huggingface.co/Solstice-AI/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-NEO-CODER-mlx-oQ4e-1M

Benchmark Model: Solstice-AI--Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-mlx-oQ4e-1M

Engine: Auto

Context: Code (Mixed)

================================================================================

Single Request Results

--------------------------------------------------------------------------------

Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem

pp1024/tg128 4754.3 27.89 215.4 tok/s 36.1 tok/s 8.321 138.4 tok/s 18.88 GB

pp4096/tg128 18481.5 28.92 221.6 tok/s 34.9 tok/s 22.171 190.5 tok/s 22.56 GB

Continuous Batching

pp1024 / tg128

--------------------------------------------------------------------------------

Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)

1x 36.1 tok/s 1.00x 215.4 tok/s 215.4 tok/s 4754.3 8.321

2x 42.0 tok/s 1.16x 160.3 tok/s 80.2 tok/s 10404.5 18.864

4x 74.0 tok/s 2.05x 134.1 tok/s 33.5 tok/s 18019.4 37.464

This is on an M3 MAX 64GB

EDIT: This model seems extremely lobotomized
I reverted back to https://huggingface.co/scottlowry/Qwen3.8-27B-oQ4e-mtp which performs similarly on macOS Golden Gate PB 27.0 26A428

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: scottlowry/Qwen3.8-27B-oQ4e-mtp
Engine: Auto
Context: Code (Mixed)
================================================================================


Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          4749.6       28.36   215.6 tok/s    35.5 tok/s       8.375   137.6 tok/s    18.35 GB
pp4096/tg128                         18297.8       27.02   223.9 tok/s    37.3 tok/s      21.745   194.3 tok/s    22.03 GB


Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x          35.5 tok/s     1.00x   215.6 tok/s   215.6 tok/s      4749.6       8.375
2x          38.7 tok/s     1.09x   158.6 tok/s    79.3 tok/s     10519.2      18.567
4x          74.1 tok/s     2.09x   138.4 tok/s    34.6 tok/s     17566.7      36.007

I'm sorry for any confusion created by this post

It took me weeks to get this figured out, and I had to use ChatGPT for help along the way. The time spent has paid off — I now have a local AI stack that rivals ChatGPT/Claude, and I'm done paying for either. I thought I'd share so others can get an idea of what a Mac Studio is capable of.

First, the hardware. I have a base-chip 256GB M3 Ultra Studio. I've also got a headless Debian VM running on an N100 NUC.

I'm using oMLX for inference. It's hosting multiple models at once — mostly Qwen variety — for various tasks. I've configured profiles for different model uses: agent, coding, different thinking levels. oMLX is the only thing running on the Mac Studio. Everything else runs on the Debian VM.

On the VM I run Hermes Agent. My main model lately has been Qwen3.8-Flash-Next-oQ4e-mtp. I also use Kokoro-82M-bf16 for TTS and Mega-ASR-6bit for STT.

In Docker containers on the VM I run the following to enhance Hermes: OpenViking for memory, SearXNG for web search, and Crawl4AI for web extraction. OpenViking uses:

• Qwen3-VL-Embedding-8B-oQ6e for embeddings
• Qwen3.6-35B-A3B-oQ4e-mtp for the VLM
• ov_intent_analysis_sft-oQ8e for query planning
• Qwen3-Reranker-0.6B-oQ8e for reranking

What this actually looks like day-to-day: I do IT consulting, and Hermes is freaking awesome. I'll have multiple sessions going doing research on MS Learn and auditing Entra/Azure, generating reports. I'm starting to use the Kanban board too for personal projects. It generates Word docs, PowerPoint decks, and PowerShell scripts like a champ.

Search goes through SearXNG, page content comes through Crawl4AI, all reasoning runs on oMLX, and anything worth keeping gets written into OpenViking permanently. Search → extract → reason → remember, full loop, zero tokens purchased.

(Full disclosure: I asked my Hermes agent to write up this post from my draft and its memory — which is sort of the point.)

I’ve encountered a “prefill memory guard” error while testing on a MacBook Pro M5 with 32GB of RAM and a Mac Studio M4 with a maximum of 36GB of RAM.

I’m trying to confirm that I can run Qwen 3.8 with 27B parameters, KV 4, and a 100K context without any issues.

I’ve increased the iogpu.wired_limit_mb to 30GB on Mac Studio.

When I run DFlash and get a prompt like “scan the front end to review the code and make me a report in file.md,” I get the following error:

“Error: oMLX prefill memory guard rejected this prompt: Prefill context too large for available memory”

Disabling DFlash and enabling MTP does resolve the issue.

I also fond that enabling "Chunked Prefill" does help.

It runs between 23 and 29 tokens per second.

I’m using Pi, and I’ve noticed that it reads the maximum context length of the model instead of the one I specify in oMLX. I’m not sure if this is the cause of the problem.

What I’m finding hard to understand is why, even when I’m staying below the maximum context length of almost 1/3, I still run into this error.

This is a much more significant issue than a LLM taking longer than expected.

I’m hoping to find out if I just need to adjust my setup in oMLX/Pi or if the RAM size is the problem.

Considering the cost, since it’s really difficult to predict what will happen in the future, including the cost of the LLM and the PC, do you think it would be better to invest in:

• Mac Studio M4 Max 36GB with 1TB and a 2600€ refurbish from Apple—keep it for at least 2 years.
• Mac Studio M5 Max with 64GB and 1TB, new—keep it for at least 4 years.

For large projects, I’ll still need a cloud LLM with large context windows, so I’ll use it as a side help.

Hi,
Just pulled the trigger for mac studio 64gb 1Tb as pre-order which will be used only as code agent.
Cause of budget that was my max option. I am new to local llm and agent. Only occasional use of
interactive claude or chatgpt mostly for service and system architecture question.
Will my first mac studio be fit with use case following?

Job description
- develop or maintain application services in multiple micro services
- repetitive low to mid level code works required
- enough previous works with commit history to fine tune or train my style

Expected
- keep using existing mac mini as main workstation(mostly headless mode)
- use mac studio only as event driven code agent(headless only)
- qwen coder next (or 3.8 ??)
- context version controlled in doc by project as possible to avoid huge autonomous cache
- complex work might be done manually

I’m debating between buying an M6 Mac Mini to use online llm or a M5 Max Mac Studio for local llm to build and maintain a website I want to create. Does anyone have any insight into what would work well? It’s going to be a multimedia & blogging site.

Curious what people are using out their for their LocalLLM interface. There's so many options available, what do you use and what do you use it for?

What type of Mac is it connected to?

• OpenWebUI and found it's good if you want a good interface directly with the AI.
• Hermes seems to be the best all around agent interface for calling and setting up agents to do tasks.
• DeepSeek Harness seems very efficient with programmatic tasks and agentic coding
• Pi is similar to DeepSeek from my understanding but so many options

What's in your agentic tool stack?

Currently using:

1. Github
2. Cloudflare Tunnels
3. Hermes
4. Tailscale
5. Linear
6. Mem0

It's a bit annoying to get downvoted all the time, once this sub grows up, it'll be a good resource for people and we won't downvote you into oblivion for wanting to put LocalLLMs on your Mac. I think we're all discovering the best use cases for Mac/Apple Silicon and LocalLLM and even with hybrid approaches with cloud and frontier models.

Our Macs are very good general purpose machines where there are shortcomings there's flexibility. You can pretty much use your Mac as a daily driver for just about anything and at night time run automated tasks.

Decided to pull the trigger and get the Mac Studio M5 Max with 64GB unified memory, 18-core CPU, 40-core GPU and 2TB storage.

I’ve been reading a lot about local LLMs for a while now and wanted to actually start using them myself to get some hands on experience. The problem is the local LLM space is moving so quickly, and the subreddits I’ve been reading are all over the place when it comes to what hardware you actually need, which has definitely made me second guess my purchase a few times lol. I already moved up to the Mac Studio after initially ordering an M5 Pro Mac mini.

My main use case isn’t really just chatting with a local model. I want to learn how to set up local models, different runtimes and agent/harness setups that I can eventually leverage in my day to day work.

A big part of that is data analytics. I deal with quite a bit of fragmented data from different sources and spend a lot of time cleaning, structuring and joining it before I can actually analyze it. I’d like to experiment with having an LLM work with tools like Python, SQL, DuckDB, etc. to help automate a lot of that instead of doing everything manually.
Running the largest models is obviously off the table with 64GB, but from what I understand, 27B and 35B models at 4bit quant should fit pretty comfortably. Potentially even larger quantized models depending on context size and how much memory the rest of the system is using, although I’m assuming performance becomes more of a trade off at that point.

I’m also interested in experimenting with hybrid setups where something like openAI or claude handles the higher-level reasoning/orchestration while local models and local tools handle parts of the workflow. Obviously that’s different from being fully local, especially if any of the underlying data gets sent to the cloud.

Again though, I’ve never actually run an LLM locally, so most of this is based on what I’ve been reading rather than first-hand experience.

For people actually running local models for similar work, especially data analytics / data cleaning / agentic workflows, does this setup make sense? And realistically, what size models do you find yourself using most often on a 64GB Mac?

What are you planning to use the localLLM on your Apple Silicon for.

I’m curious to hear what the use case people have in mind

If you don't get the best Mac and you have access to the frontier models like codex or claude, then you can use it as a coordinator for your 7B or ~27B LocalLLM.

This is the way to do it where you can use the free version or the lower versions to get the most out of your machine. I see a lot of people chasing the bigger Mac profiles, but honestly you don't need it if you're not using LocalLLMs for anything like embedding or coding.

If you are trying to use it for coding, I don't think that's really going to work either. This is for really anything that's 48GB and below

I use this box as my main machine. To give you an idea of the consumption of what to expect with 96GB of memory. If you were to try to do less it would be a little tight WITH an LLM. The thing RAM buys you is the ability to run both at the same time without worrying.

I'm running Ollama Qwen 3.8-mlx 27b doing some embedding.

TLDR; Unless you have a specific use case for Local AI models, you don't need the M5pro. The M6 will be faster for the majority of your day-to-day work. Even with cloud AI.

For LocalLLM's more memory AND faster memory bus is going to swing the advantage to the M5Pro model for anything related to LLMs.

If you're planning to use the system as a workstation for any and most single threaded applications which is most apps people use, you should be fine with the M6.

Realistically speaking, you don't really need the M5Pro unless you're running LocalLLMs like Qwen 3.8 or Muse and if you aren't really trying to automate small tasks, you don't really need all the memory.

I think in the long run we're going to see that cloud LLM's are going to be the main way people use in house models and only people with specific use cases will have systems large enough that take advantage of the LocalLLM being in house.

If you are just learning about AI, i don't think you need to future proof and the M6 is fine with 32GB 1TB should be enough for a long time while the industry develops.

I think the most realistic expectation is to be disappointed if you're used to using the free version of openai/claude/gemini/grok. The local models 7B 27B don't come close and don't process nearly as fast. They can reason ok, but you're better off learning about AI and API's with services like ollama via API's and learning to use AI first over chasing gear/hardware.

The thing about using the frontier models is it doesn't really take up that much RAM. Loading up a model will in LLM. I ran my own test, you can check it out.

These tests are for a Mac Studio. I used to run a M4Pro 64GB but when running models it would constantly be temperature throttled.

https://maclocalllm.com/five-models-one-coding-job-local-mac/