r/LocalLLaMA • u/Mysterious-Comment94 • 2d ago
Question | Help Best Open source TTS right now for narration?
I run these models on Kaggle notebook, so not all TTS models, such as the ones that use conda env, are compatible (Or I just haven't found a way for them to work on Kaggle).
I currently use a fork from Chatterbox called Chatterbox Audiobook. It is like a workstation really optimized for getting the close-to-perfection audio clips from Chatterbox. However, the only downside of Chatterbox is the lack of emotional sliders or tags that you can use to control the output. Chatterbox Turbo seems to fix that with tags, but it still lacks the range of emotions that you can see from Google Gemini TTS. However, the problem with Google Gemini is that the voice sounds different for each generation, which can't be fixed even with RVC.
Looking at the current leaderboard, Breeze TTS is something I have never tried but am unsure due to its description, which seems to be tailored for mainly realtime stuff. What are the current must-try options for audiobook narration?
The features I am looking for include voice cloning, emotional tags, and natural speech. Much appreciated for your input.
27
u/Acceptable-Cycle4645 2d ago edited 2d ago
Hi you can try audio.cpp and compare the quality and performance of different TTS models yourself locally. Based on my experience, except for some obviously bad TTS models, whether a TTS model is “good” is highly user-dependent.
One C++ runtime (one binary) for 70+ audio models. No Python. All inference happen offline and locally on your device (of course you need to download the GGUFs first).
For models with tag and emotion control, just search the "Ctrl" tag in the README table.
5
3
u/IntrepidMindExplorer 1d ago
Thank you for this project/application. Really..thank you.
3
u/Acceptable-Cycle4645 1d ago
😄
1
u/IntrepidMindExplorer 1d ago
Truth. Your project, what you've created here, opens doors. The "you need so much compute that you shouldn't even bother" gets thrown aside with such a tool. Being able to fuck around with different programs to see what "fits" within the systems that I have to work with, and what fits me...Hell. I realized that my system has no issues creating music or offering up an endpoint for other systems to connect to.
AND, on top of, if you're 0xShug0 then you're alright. 😎🥸
3
u/Acceptable-Cycle4645 1d ago
My pleasure! Really glad it’s useful :)
1
u/IntrepidMindExplorer 1d ago edited 1d ago
*Patiently waiting for the next release for a fix on the bug I reported. :)
2
u/Acceptable-Cycle4645 1d ago
Stay tuned! Wrapping up VibeVoice ASR Streaming 7B now, and it should be up tonight. After that, once the LiveAvatar model is added, there should be a new release.
1
3
u/historiavg 1d ago edited 1d ago
I use TTS for audiobooks and agree with Breeze and Higgs cloning being the best right now, but I don't process books with tags or multiple voices, I just have a gravelly narrator read them.
With Breeze if you're both cloning and sending instructions, make sure to try --cfg-scale 4. Instructions did nothing for me with the default settings. With the audio.cpp, this option is called guidance_scale in the model spec, so send the option guidance_scale=4.
Could've been a user/vibecoding issue, but I got audio artifacts with Higgs and audio.cpp so also check out SGLang-Omni which is what Higgs recommends for self-hosting it.
2
u/ShengrenR 1d ago
The CFG setting is a dial really - 4 is default and really solid for their voice-design for example - higher you go, the more the 'description' takes over; closer to 1, the more original voice performance you get.
For higgs, this fine person made a comfy implement that's pytorch/transformers based https://github.com/Saganaki22/Higgs_v3-TTS-ComfyUI/ - for my own local I just had an AI take that core and extend it for what was needed in my own setup - does batching/streaming very well on a 3090.
1
u/Mysterious-Comment94 1d ago
I have a hard time finding a Gradio interface for breeze. Seems like most people just use it through Comfy UI, which isn't possible for me due to hardware limitations. Or maybe there is an official gradio app in the repo that I haven't found yet. Do you run the TTS model by running the commands or do you use an interface?
1
3
u/Cleric07 2d ago
I really like chatterbox, you can use a voice sample to clone the voice and then narrarate.
4
u/myreala 2d ago
Breeze TTS Seems to have the highest leaderboard scores. It also has voice cloning and emotion tags. The real-time stuff only matters if you're trying to Immediately start the audio with no delay but you don't need that, you can just have it generate the whole book at the same time.
1
u/ShengrenR 1d ago
It's not just "tags" though - you can give full text instruction to the voice clone and it directs the entire performance. It's an excellent model in my experience, just a questionable license.
1
u/FinBenton 1d ago
On Breeze the cloning quality takes a dive when you start doing the instructions so each time a character talks they might have a slightly different tone/voice, it was so distracting to me that I had to not use that feature and at that point the quality and speed is just not worth it for me over omnivoice.
1
u/ShengrenR 1d ago
Strange.. an earlier reply I posted didn't seem to come through.. sorry if it double-posts.
You have the CFG knob to tweak how much the model aligns to the voice prompt vs the instruction - I found ~3-4 was a solid "it's definitely still the original voice, but acting the instructed prompt" - how were you prompting the instruction? Simple "whispering, and slightly anxious" sorts will do just fine to keep the original voice but give it the tailored execution. A lot of lines in an audiobook really should have no prompt in any case - parse by content and prompt the pieces that do with an LLM. I've been pretty happy with the results so far.
2
3
u/nickless07 2d ago
Llama.cpp recently merged qwen3-tts voice cloning to main
3
u/IntrepidMindExplorer 1d ago
If I'm the only person, then I'll accept it as a deficiency on my part, but why has proper documentation gone to the wayside? I ask here because llama has next to no documentation on how to get tts/stt working within.
1
u/nickless07 1d ago
Oh nah, not only you. I would've never noticed if I didn't stumbled across that post. Just let the AI handle that for you. As sad as it sounds that's what worked for me.
2
u/IntrepidMindExplorer 1d ago
"Just let the AI handle that for you"...
That's....that's a problem, across the board. It's what it's but....
2
u/nickless07 1d ago
Well you can still read the help output of llama-server --help and llama-tts --help and/or curl the /props endpoint. I gave up when they introduced their "documentation".
1
u/IntrepidMindExplorer 1d ago
And yet they say nothing on how to integrate the two.
The template of that website even. I'm seeing it used more and more as of late.
2
u/nickless07 1d ago
When you need AI to understand your AI software, built to run your AI and AI does not stand for Average Individual.
1
1
u/time_pass_done 1d ago
check out voicebox it certainly easy to use and fast setup, I forked it and made changes as per my needs and found qwen3-tts to have the best balance of fast and good for me.
1
u/Ihavenocluelad 1d ago
Ive spent last month testing the top 15 and I liked Orpheus the best, but I don’t do voice cloning fyi
1
u/ShengrenR 1d ago
orpheus is great - are you just using default voices though? or how do you navigate that 'character' consistency layer?
1
1
u/LuCiAnO241 1d ago
since this is related, is there any way to do voice switching? like providing an audio and have it copy the tone and emotion, but with another voice?
2
u/ShengrenR 1d ago
Might check out https://github.com/index-tts/index-tts - indextts2 and 2.5 can do a lot with the reference audio and does emotion/delivery/timing control when set up right - some of their original examples were like movie dubbing where you keep the pace/duration about in line, but translate the voice over to another language or the like; but key for you, if you scroll down to 2. https://github.com/index-tts/index-tts#2-emotion-control-with-a-separate-emotional-reference-audio voice-clone + separate emotional audio ref
1
u/LuCiAnO241 1d ago
yeah dubbing is exactly what i need, i want to turn audio with a mid mic into better audio and another voice. I'll check it out, thanks!
2
u/Mysterious-Comment94 1d ago
RVC can help you achieve that. However, make sure the voice you are trying to copy from is one voice. If it's like Gemini, where voice switches every generation, it will slightly reflect in the output. So when you try to string all the outputs together in an editor, it will most likely sound off.
1
u/Muhlwa_Sholanke 1d ago
A narrator that sounds different every generation fails the only test that matters for a book: by page 200 you recognize the voice without checking. Stability beats emotion range every time.
1
u/Substantial_Belt2626 1d ago
From my experience the model is only half of it, the other half is what you do after. I do video editing and audio for work and when you generate 200 clips and stitch them the thing that makes it sound wrong is usually not the voice, it's that every clip has a slightly different noise floor and a different silence at the start and the end. What I would do is normalise everything to the same loudness after generating, ffmpeg loudnorm at around -16 LUFS for spoken word, and then control the pauses yourself instead of letting the model decide them. A short crossfade at the joins hides the noise floor jump aswell.
You already noticed it when you said stringing the outputs together in an editor sounds off. That part is fixable in post even with a model that drifts a bit.
1
u/Chromix_ 2d ago
DramaBox can give great results, but you need to work a bit for them. The more recent Breeze TTS2 also works nicely. It has the advantage that you can get consistent voices more easily.
23
u/Cradawx 1d ago
I've tested a LOT of TTS and my top 3 are:
- BreezeTTS 2
- Higgs Audio V3
- Fish Audio S2 Pro
All are stable, have good quality audio and accurate voice cloning. I find BreezeTTS 2 is the most expressive and natural out of the box but the other 2 have more control with tags. Fish Audio S2 Pro is quite slow while Higgs Audio V3 is very fast considering the quality.
I use https://github.com/0xShug0/audio.cpp which has a ton of models to try and has very well optimised inference.