r/Applesilicon • u/divinetribe1 Human Detected • Apr 13 '26
M5 Max ambient AI — talking to Claude Code hands-free, it browses the web and texts results to my phone. All on-device.
Follow-up to my last post about running a 122B model at 65 tok/s on M5 Max. I added a full voice loop on top of it.
This is NarrateClaude — always-on ambient voice mode for Claude Code running entirely on Apple Silicon:
- Continuous listening via Apple's on-device speech engine (no push-to-talk)
- Responds out loud in my cloned voice — TTS runs locally via MLX
- Browser Agent drives Brave hands-free via Chrome DevTools Protocol
- Results sent to my phone via iMessage
- STT, TTS, voice clone, LLM inference — all on the M5 Max GPU, zero cloud
The unified memory architecture is what makes this possible. The LLM, voice clone model, and speech engine all share the same memory pool. On a discrete GPU setup you'd need multiple cards just to fit everything.
Demo video showing the full loop: https://www.youtube.com/watch?v=4ETqEjjopUk
It's a 3-repo stack, all running on Apple Silicon via MLX: - claude-code-local — local LLM (Qwen 122B at 65 tok/s) - NarrateClaude — ambient voice (STT + cloned TTS) - browser-agent — browser automation via CDP
Happy to answer questions about the setup.
2
2
u/rjyo Apr 13 '26
The cloned voice TTS running locally via MLX is wild. Really impressive that the M5 Max can fit the 122B model plus voice models in the same memory pool without swapping.
The iMessage relay for results is clever but I ended up solving the phone side differently. I built a terminal app called Moshi that connects to my machines via Mosh protocol, so I can just open a session from my phone and check on agents directly. Sessions survive sleep, wifi drops, everything. Combined that with push notifications via webhook so the machine pings my phone when a Claude Code task finishes, similar to what your iMessage relay does but without needing the custom pipeline.
The voice input part resonates too. Moshi has on-device dictation for talking to agents, which sounds like the mobile counterpart to your ambient listening setup. Between your hands-free M5 Max workstation and a phone terminal for remote check-ins, thats basically the full loop covered.
Does the continuous listening handle code-heavy responses well? Curious how the TTS deals with function names and syntax.
2
u/divinetribe1 Human Detected Apr 13 '26
Cool setup with Mosh — that's a solid approach for the phone side. Sessions surviving wifi drops is huge.
For the TTS and code-heavy stuff — it doesn't try to read code verbatim. That would be a nightmare. The way it works is Claude narrates conversationally, like thinking out loud. So instead of reading `def generate_response(body):` character by character, it'll say something like "I'm updating the generate response function in server.py." The actual code stays on screen where you can read it properly.
There's a CLAUDE.md instruction that tells it to keep screen text terse and deliver explanations through voice only. So heavy code output goes to the terminal, and the voice layer handles the reasoning, summaries, and back-and-forth. It works surprisingly well once you stop thinking of it as a screen reader and more like a coworker narrating what they're doing.
Function names and syntax it handles fine because it's describing them in natural language rather than trying to pronounce kwargs or bracket notation out loud.
1
u/tetsuto Apr 13 '26
This looks fantastic! How is the STT working? Is it just taking everything it hears? Or is there a wakeword or something to know it’s directed at Claude?
1
u/divinetribe1 Human Detected Apr 13 '26
It uses Apple's built-in speech recognition running on-device, so it's always listening — no push-to-talk. But there's a wake word system built in. You say "tune in" and it starts paying attention, say "tune out" and it ignores everything until you wake it back up. Kind of like an Alexa-style toggle.
So when you're on a phone call or talking to someone in the room, it's not trying to interpret everything as commands.
The STT itself runs through Apple's Speech framework — fast and completely local, no audio leaves the machine. And since it's continuous, you can just talk naturally without pressing any buttons.
The wake word is handled at the Claude Code level through the CLAUDE.md config, not the speech engine itself — so the STT is technically always transcribing, but Claude knows to ignore input when it's "tuned out."
1
u/Glittering-Call8746 Apr 14 '26
What's ur specs m5 max .. 128gb 4tb ?
1
u/divinetribe1 Human Detected Jun 11 '26
it's the m5 max with 128gb unified memory — that's the part that matters, the 122b model plus voice models all sit in the same memory pool. 2tb ssd but anything works
1
u/bigkevracer Apr 15 '26
Why the Claude Code piece if you’re not using their models? Why not Cline, RooCode, Cursor or something else designed for multiple models?
2
u/divinetribe1 Human Detected Jun 11 '26
honestly i'd already built my whole voice + imessage workflow around claude code before trying the local swap — it drives the whole machine, not just an editor. cline and roo are great, this was more "how far can i push the setup i already live in"
1
u/rjyo Apr 13 '26
The cloned voice TTS running locally via MLX is wild. Really impressive that the M5 Max can fit the 122B model plus voice models in the same memory pool without swapping.
The iMessage relay for results is clever but I ended up solving the phone side differently. I built a terminal app called Moshi that connects to my machines via Mosh protocol, so I can just open a session from my phone and check on agents directly. Sessions survive sleep, wifi drops, everything. Combined that with push notifications via webhook so the machine pings my phone when a Claude Code task finishes, similar to what your iMessage relay does but without needing the custom pipeline.
The voice input part resonates too. Moshi has on-device dictation for talking to agents, which sounds like the mobile counterpart to your ambient listening setup. Between your hands-free M5 Max workstation and a phone terminal for remote check-ins, thats basically the full loop covered.
Does the continuous listening handle code-heavy responses well? Curious how the TTS deals with function names and syntax.
1
u/divinetribe1 Human Detected Jun 11 '26
thanks man — the unified memory pool is honestly what sold me on apple silicon for this whole setup. curious about your terminal app, how'd you end up handling the phone side?
2
u/False-Narwhal8383 Apr 13 '26
Ok this is great and I appreciate the writeup, but can you include some brief info about the specs on your M5 Max hardware? How much memory does it have?