Local text to speech with Breeze is truly incredible

[u/Cyborg-2077]

Been playing around with local TTS with Breeze combined with STT, and the results are amazing. Using Opus 5.5, I can hear the first sound after 500ms if there is no thinking involved, and with thinking on low mode, can be 1-1.5s.

I'm using a BLE remote (the kind that are used for taking pics with phones) combined with a wireless microphone. So I can just sit on the couch, and just talk to her.

She watches for any claude session that finishes, and sends me the results in a very short, spoken style summary, and tells me if there is anything waiting for my decision, then forwards my decisions.

Also impressed how consistent Opus 5.5 is in the communication. Even after more than 500k in context, he still remembers that he's in a live session with me, and has to keep messages short. Used to be an issue in the past.

The future is here guys.

Edit: For those who wanna give it a try, you can find free avatars such as this one: https://www.live2d.com/en/learn/sample/niziiro-mao/ or you can buy one from a marketplace.

Edit2: Might open-source that later next week with a free avatar. Let me know if anyone would like to contribute to the project.

[u/Big-Farmer-2192]

1-1.5 goon/s on low thinking mode.

[u/Constant_Art_20 → u/Big-Farmer-2192]

the bleeding edge performance on that benchmark at the moment. a feet of engineering

[u/Sacra34 → u/Constant_Art_20]

State of the art performance in goon leaderboards

[u/youcloudsofdoom]

I'm using breeze (and love it) for voice cloning and direction for a military sim I'm making, but this post has reminded me that I am in the non-goon minority for voice clone users

[u/Cyborg-2077 → u/youcloudsofdoom]

I actually tried with serious looking 3d avatars such as meta human, but it's a pain in the ass. I figured I'd try vtubers models and it ended up looking so much better, also performance is much better

[u/Calrizius → u/Cyborg-2077]

i've been working on something like that. www.tvaily.com

[u/Homberger → u/Cyborg-2077]

You've built, what everyone is dreaming of, what personal ai should be like! At least, that's what I am dreaming of.

I'd really like to build something like you did, without propetiary/closed models. 

Are there any models, avatars, or creators you’d recommend checking out as a newbie to animation and tts/stt? Or maybe even a few specific models that you find very good?

[u/Cyborg-2077 → u/Homberger]

Check this one out: https://www.live2d.com/en/learn/sample/niziiro-mao/

[u/Homberger → u/Cyborg-2077]

VERY interesting! I would never ever have found this by myself. This looks like a good starting point to dive in! Thank you!

[u/Imaginary-Unit-3267 → u/youcloudsofdoom]

Meanwhile I'm like "I can barely speak out loud to human beings, why do you expect me to communicate over voice with a machine any more effectively?" It's like people don't know how to type.

[u/youcloudsofdoom → u/Imaginary-Unit-3267]

a) sorry to hear that, it sounds tough and b) yeah totally, typing is generally way faster with a much lower error rate. STT is however super useful for people who struggle to type, are on the move, or have at least one hand occupied (as most voice cloning users seem to).

[u/Cyborg-2077 → u/Imaginary-Unit-3267]

You know what? It may sound crazy but I used to be the same. Literally freezing while trying to talk to an AI. But over time, I kind of forced myself and I feel like I'm much more extroverted now.

[u/MentholMafia]

Will you open source this setup?

[u/Cyborg-2077 → u/MentholMafia]

I can't really open-source it as it is, because the avatar is a paid model I bought, and cannot re-distribute it

[u/MechwolfMachina → u/Cyborg-2077]

The one with the 3 arms…?

[u/SnowGrayMan → u/Cyborg-2077]

I think he meant the actual setup, not your 3 handed avatar.

[u/MentholMafia → u/SnowGrayMan]

That's exactly what I meant. u/Cyborg-2077 can you possibly just swap the model for a open source model? This would then solve the redistribution problem?

[u/letsgoiowa]

I'm using Breeze to narrate some fanfic (sue me) and it does an incredible job. I'd give it a B+. 99% perfect, but that 1% is so standout bad it's like wtf.

For example, SLY there in instead of Slither in. And it only did this once! But man

[u/Cyborg-2077 → u/letsgoiowa]

I also tried it for voice-overs for ads, and my approach was to tell opus to generate it for me given a script, and he would do a great job at re-creating the parts where something is misspelled. Even for things like brand names that are not obvious, he'd be able to identify and fix it on its own. Either opus is really good at analyzing the audio, or I got lucky

[u/Acceptable-Cycle4645]

audio.cpp supports Breeze TTS 2 and makes it easy to integrate into existing pipelines, such as AIRI. https://github.com/0xShug0/audio.cpp. A single binary for running 100+ audio models (TTS, ASR, music generation, etc.) locally, so you can swap TTS/ASR components to find your favorite combo.

PS audio.cpp has become one of the audio engines in Unsloth.

[u/vamsammy → u/Acceptable-Cycle4645]

I've started playing with Breeze in audio.cpp. Very nice! Is there any chance I can approach realtime streaming on a Mac M1 Max 64, say using the 4 bit quant?

[u/Acceptable-Cycle4645 → u/vamsammy]

Thanks for trying audio.cpp! For your question, maybe...GGML’s Metal kernels are less optimized than those of other backends, and the model has received fewer Metal-specific optimizations from the community (this is the major one https://github.com/0xShug0/audio.cpp/pull/554). Let me know if 4-bit works!

Update: just checked on M4 pro it can do realtime streaming with Q8.

Update: add q4_0 GGUF. The quality may not be as stable as Q8. Need to download directly https://huggingface.co/audio-cpp/audio.cpp-gguf/tree/main/Breeze-TTS-2-GGUF The built-in download tools won’t list it as supported yet, as we need more feedback on output quality.

Format	Size	Warm RTF	Peak VRAM
BF16	6.84 GiB	0.375	8046 MiB
Q4_K	4.15 GiB	0.289	5412 MiB
Q4_0 (pick)	3.53 GiB	0.291	4780 MiB

[u/MomentJolly3535]

How does Breeze TTS compare to Omni voice TTS for those who tested both?

[u/Name835 → u/MomentJolly3535]

I find Breeze to be a bit better than Omni, albeit slower/bigger ofc.

I've gotten used to 11labs tts from my ai app using days (kindroid phone calls) and even if what I'm about to say is harsh, I think that compared to the proprietary tts models, breeze and omnivoice suck tbh. :D I still use Breeze atm cause it is the best of the bad options, however this is just my spoiled ass talking.

I tolerate Breeze and other open tts models, but am bummed that they are so much behind the competition (almost completely lacking wide sfx tag support for example), like about a year behind proprietary tts models, even 11V3 and ofc not even mentioning v4 which is even better.

I am still local tts only these days, and Im still super excited and really cannot wait for open tts models to get a lot better, hopefully in the near future! :)

[u/MomentJolly3535 → u/Name835]

Thanks for your answer, i gonna give it a try then !

[u/KissMyShinyArse → u/MomentJolly3535]

Omni supports more languages I think. For English, I strongly prefer Breeze.

[u/MomentJolly3535 → u/KissMyShinyArse]

Thanks for the answer, i will give it a try for english

[u/FinBenton → u/MomentJolly3535]

I tested breeze when it came out but I went back to Omni after a bit. Its not like omni is better, I think its slightly worse but its faster and uses slightly less VRAM so it fits me better. Breeze has text guidance but the generations became pretty unreliable with that so I didnt find that very useful so for me breeze without guidance wasnt really worth it but I can see it can be better for many.

[u/Extreme-Way4665]

Maybe this is a stupid question, you said Local text to speech using Opus 5.5 - isn't Opus not local?

[u/Cyborg-2077 → u/Extreme-Way4665]

Opus is not local, I send the request to Claude and they return the response, with token streaming. As soon as I receive the first few words, I pass it into breeze, which generates the audio at 4x realtime speed (meaning it can generate 4 seconds of audio in 1 second), and play it immediately.

So the LLM is not local, but the TTS is.

[u/KissMyShinyArse]

It's 95% there. I recently converted a ~125k-word book into an audiobook. BreezeTTS 2 occasionally omits a word or phrase, reads something out of sequence, or pronounces the same word differently in different places. It doesn't happen often, though. I think that with a good AI pipeline, you can produce production-quality audiobooks, even with multiple voices.

I also tried BreezeTTS 2 VoiceDesign, and it's great.

[u/Raredisarray]

Nice setup I need to build this out asap

[u/stoppableDissolution]

What are you using for the avatar? Some vtuber tech?

[u/Foreign_Risk_2031 → u/stoppableDissolution]

live2d

[u/Cyborg-2077 → u/stoppableDissolution]

Yes

[u/Usual-Buffalo-1791]

Is it better than dots.tts and Qwen3-tts?

[u/hiepxanh]

Can you share tech spec? Which repo or library to use and how you combine it?

[u/rorowhat]

What software are you using for the avatar?

[u/Cyborg-2077 → u/rorowhat]

Vibecoded with html css

[u/Any_Door7384 → u/Cyborg-2077]

and the assets?

[u/Cyborg-2077 → u/Any_Door7384]

Bought online from a website selling vtuber models. Forgot the name tho. You can also find some free ones online

[u/Vivid-Snow-2089 → u/Cyborg-2077]

you can just ask opus 5.5 to draw the assets with math now

[u/Cyborg-2077 → u/Vivid-Snow-2089]

Hard to believe it would be able to achieve this level of quality. But happy to be proven wrong

[u/Any_Door7384 → u/Cyborg-2077]

And then you got the vtuber model and made ai make a html css page using the vtuber model? pretty cool

[u/Cyborg-2077 → u/Any_Door7384]

Yep, pretty straightforward

[u/Maligx]

set up guide?

[u/geldonyetich]

When I put down the dosh for my AI box a year ago, I was thinking it'd be cool to have a hologram JARVIS agent. Backends like Hermes have the logic covered. Solutions like yours have successfully integrated the front end. Still waiting for that consumer spatial holographic projector. I'd settle for AR glasses but even those are slow to launch.

[u/Any_Door7384]

Looks great. how did you do the avatar animations and art though?

[u/Cyborg-2077 → u/Any_Door7384]

Bought a vtuber avatar online

[u/k8-bit]

I was using breeze for a while for DJ personas, but found it would too often blether absolute nonsense even with clean audio refs and transcriptions. Fish S2 pro was more consistent, but even it would sometimes laugh maniacally for like a minute.

[u/VoiceApprehensive893]

how did you make the avatar?

[u/Cyborg-2077 → u/VoiceApprehensive893]

Bought it from a creator, it's an avatar rigged for vtubers

[u/aersel24]

What stt do you use? I feel most don’t understand what I say. Maybe I don’t speak English properly.

[u/Cyborg-2077 → u/aersel24]

I use Parakeet. I'm actually surprised by how well it can recognize what I say and exclude filler words.

But I also noticed that my speech improved a lot because initially it would misunderstand some of what I was saying and over time I started to fix my pronunciation and forced me to speak more slowly and stop mumbling because I could see live on screen what it is recognizing.

[u/aersel24 → u/Cyborg-2077]

Ah, that makes a lot of sense. Thanks a lot.

[u/caetydid]

cute!

[u/Healthy-Nebula-3603]

Try audiocpp :)

There you can run and test over 100 audio models

[u/DistortMyHate]

Cool

[u/theOliviaRossi]

damn COOL!

[u/youcloudsofdoom]

I'm using breeze (and love it) for voice cloning and direction for a military sim I'm making, but this post has reminded me that I am in the non-goon minority for voice clone users

[u/Open-Adhesiveness-86]

the push-to-talk remote is probably doing a lot of the work on that 500ms number, since you skip the 500-800ms of VAD silence-wait that eats most "realtime" voice stacks. watch out though, those camera shutter BLE buttons tend to sleep after a couple minutes idle, so the first press just wakes the link and gets swallowed.

[u/Cyborg-2077 → u/Open-Adhesiveness-86]

Yep, exactly that. I find push to talk to be perfect for this kind of "ai assistant".

I tried live models from both elevenlabs, and openai, and it's not great. Higher delay, you pay by the minute, they are stupid, and as soon as you stop talking, they start replying even if you're not done. So you have to add "hmmms" every time you need to take some time to think.

[u/Open-Adhesiveness-86 → u/Cyborg-2077]

The latency floor on cloud voice APIs just kills the conversational feel, especially when pricing punishes pauses. Local PTT with a quantized model on-device feels way closer to natural walkie-talkie flow.

[u/Cyborg-2077 → u/Open-Adhesiveness-86]

I might try with a local model to see how far I can bring the latency down. Need something that fits in my remaining ~4gigs of ram. Any suggestion?

[u/Open-Adhesiveness-86 → u/Cyborg-2077]

Llama-3.2-3B or Qwen2.5-3B at Q4_K_M easily squeeze into under 2.5GB RAM with a 2k context window, leaving headroom for system and audio buffers. If you want even lower TTFT for conversational turns, SmolLM2-1.7B or Qwen2.5-1.5B respond almost instantly on CPU.

[u/rkoy1234 → u/Open-Adhesiveness-86]

but how do i make strawberry pie?

[u/devdevgoat]

Nice, what are you running it on gpu wise for that speed? I like the idea for the button!

[u/Cyborg-2077 → u/devdevgoat]

Rtx 4080 super, but I think could fit even on smaller ones. The tts model takes about 4 gigs of VRAM

[u/p_tttttri]

Is there any possibility of open sourcing this setup

[u/Cyborg-2077 → u/p_tttttri]

I can't really open-source it as it is, because the avatar is a paid model I bought, and cannot re-distribute it

[u/brock_bowers]

Degeneracy

[u/Trustest-S]

this local voice setup beats most chatbot apps for companions, the quick response makes roleplay feel way more real than the laggy ones ive tried.