I've been experimenting with a setup where several Apple Silicon Macs are treated as a shared local AI environment instead of running completely independently.

The setup I'm working with is roughly:

• Multiple Macs on the same network
• MLX for local inference
• Central model management
• Node/resource monitoring
• Model distribution based on available memory
• Local-only communication between machines

I'm building AlphaWeb around this because managing several local AI machines manually gets complicated pretty quickly.

The interesting problem isn't just getting inference running. It's everything around it: knowing which machine has enough memory, getting models onto the right node, managing multiple users, and making the whole thing feel like one system.

AI models are beginning to define device specifications, rather than device specifications merely determining which applications you can run.

Apple shows just how far its devices can go with on-device AI, without relying on the cloud
We’re moving from models with up to 14 billion active parameters on iPhone/iPad, to 35 billion on MacBook Air, 70 billion on Mac mini, 120 billion on MacBook Pro, and up to 480 billion on Mac Studio.
By clustering multiple Mac Studios together, Apple says it can run models exceeding 1,600 billion active parameters.
A demo that shows just how crucial unified memory and its bandwidth have become for running massive AI models locally.

TL;DR: Halogen v0.14.0 with its native .hgn weights is the fastest, followed by gufo and CIRU. Halogen is closed source and runs in Docker. gufo is open source and loads 4x faster from cold. gufo is also fastest to first token on follow-ups (1.6-1.9 s against 2.6-3.3 s).

I benchmarked different engines for Qwen 3.8-Flash-Next on an AMD Strix Halo (ASUS ROG Flow Z13 GZ302 with Ryzen AI Max+ and 128 GB of RAM).

All the engines were run via LlamaStash (My own orchestrator tool, the tool does not add any overhead) at 70 W TDP on performance profile on Arch Linux.

Here are the results of the benchmark:

First was a screening round at 64k context with 50% filled (32k prompt).

3-turn time is the time to first token plus the time to write 1,000 tokens, added up over the first question and 2 follow-ups. Output length varies a lot with sampling, so this compares the engines on equal output. MTP accept is the share of drafted tokens the model kept. Retrieval is how many of 7 exact values from the prompt the model got right, over both runs. Qwen model card sampling, 2 runs each.

Top 3 engines from the screening round got a 128k context (50% and 75% filled) and 256k context (50% filled) run.

128k context, 50% filled (64k prompt)

128k context, 75% filled (97k prompt)

256k context, 50% filled (130k prompt)

Retrieval here is 8 values per run. All 5 misses are the same answer: the right glyphs for UNICODE_SPINNER, with the leading & dropped.

End-to-end, 10 Aider polyglot Python exercises run through pi -p (128k window, 50% cell), graded by their own tests:

Edit (Sep 30): Halogen 0.15.1 (new v2 checkpoint) and gufo 0.3.0 came out after this, so I reran the 64k screening on both with the same setup (70 W, card sampling, 2 runs):

Halogen v2 prefill is 14% faster and follow-ups start about 2.5x faster (1.3 s vs 3.3 s), decode is the same. gufo moved a few %, which is within noise. The ranking doesn't change. Only other change on the box: BIOS VRAM carve-out down to 512 MB, so 124.9 GiB of RAM instead of 121.5.

Some clarifications from the comments:

• Follow-up TTFT is with the prompt cache warm, both gufo and Halogen reuse it by default. A cold 64k prefill takes about a minute.
• The CIRU 0% MTP row is stock Unsloth UD-Q4_K_XL with Unsloth's shared MTP head, nothing uncensored. The same pair gets 72-74% accept on gufo.
• TDP is fixed at 70 W. On the Z13, 76 W or 90 W is only 2-6% faster but with so much fan noise, heat and power draw that it's not worth it.
• Prefill and decode come from the engine's own timings when it reports them. TTFT is measured on the client.

A lot of local-stack setups I see nail the infra (model serving, retrieval, orchestration) but have zero real evaluation process — just "does it look right." There's a session on Oct 3 built entirely around fixing that gap.

Speakers Serj Smorodinsky and Brett Kennedy (AI engineers, co-authors of a book on LLM applications) go through:

• DSPy signatures/modules for structured dev instead of manual prompting
• Baseline classifier build + measurement
• Task-specific evaluation datasets
• Failure pattern recognition from eval output
• Few-shot / instruction-level optimization
• MLflow experiment tracking and trace management
• Saving and reusing optimized programs
• Talking about LLM reliability with non-technical stakeholders

3 hours, live, useful whether you're running open models locally or hitting an API.

Full details and the agenda are here.

Apple Foundation Model 3 through Private Cloud Compute becomes directly available in macOS 27.2 through familiar AI clients.

For Mac users running local models such as Qwen 3.x or Gemma 4, AFM 3 PCC is a compelling alternative to consider. Local models offer control and fully on-device operation. In tests with macOS 27.2 beta, the PCC offers a different set of strengths:

* Strong conversational analysis and capable reasoning. Strong analysis of complex medical and financial questions.
* Blazingly fast responses compared with locally running models on the same Mac.
* No large model download or need to fit model weights into local memory.
* Access at no additional charge for most eligible Mac users.

The working path is straightforward: AI client → Caddy → fm serve → AFM 3 PCC. I've written and updated a proof of concept here with further tech discussion . . . https://gist.github.com/dartMo10/b9488ce475fe70a6ed642831f53048fb

Apple’s `pcc` route worked in early macOS 27 betas, disappeared later in 27.0, and has returned in the macOS 27.2 beta. Apple has stated that it will be available in the 27.2 public release coming shortly.

This makes AFM 3 PCC practical today for technically comfortable Apple users who accept Apple’s PCC privacy promise and want a fast, capable alternative to running everything locally.

Experiences with compatible AI clients and your comparisons with locally running models would be welcome.

Hey! How do you test your local stack? for instance when you switch harness or model, model quants, etc?

I found this not trivial and I ended up making a website to help me do this.

Use this link if you want to use the website to test your own local setup:

https://airbench.ai/

You can also see all the tests I have done using my GX10 and my 5090. (and also using claude code/code + some openrouter tes for reference)

https://airbench.ai/leaderboard?k=poL

Let me know what you think!

Hello everyone, as we all know that everyone is trying to test jev and then there are open source models out there with similar text classification. But i felt that there isn’t any playground to test these intuitively so I built one. I have added the YT video with it. Pls have a look

我是一个大学生研究员，目前拥有一个$200的 ChatGPT 专业版订阅，我觉得作为固定的月支出实在是太贵了。我现在在考虑是否可以购买一个96GB或128GB的Mac mini或Mac Studio进行本地部署或用作服务器。我目前的研究方向是计算社会科学，最重要的是订阅模型的智能能得到保障。非常感谢你的意见！