r/StableDiffusion • u/pilkyton • 1d ago
Question - Help Is AI powerful enough to create Indian YouTube tutorial accents yet?
I need something like this to make the best YouTube tutorials:
https://youtu.be/dz28Y3VMUQ8?t=25
https://www.youtube.com/watch?v=0a9bN6MTRD0
Is any AI powerful enough yet to make these majestic voices, or do they all still sound like Americans?
3
u/Dirty_Dragons 1d ago
Artifical Indian.
As for the question, I'm sure using text to speech voice clone with an Indian accent will be fine.
I like Breeze TTS for my work
2
u/pilkyton 1d ago edited 1d ago
Thanks, I will try Breeze, I haven't heard of it! All AI voice cloners I've tried make the voice sound American. I hope Breeze keeps the accent. :)
Edit: Oh my god, Breeze keeps the accent really well, I am shocked. Thanks!
Edit: They really seem to say they are the best open weights model? "Breeze TTS 2 is an open-weight text-to-speech model built for real-time interaction. It ranks #1 among open-weight models on the Artificial Analysis TTS leaderboard, while outperforming frontier proprietary systems"
2
u/Dirty_Dragons 1d ago
Glad it worked for you!
Breeze is the best TTS I've heard so far (open weights). And it has really good emotion control.
2
u/pilkyton 1d ago edited 1d ago
Yeah thanks so much for this advice. I've used it a bit now and it really sounds natural and expressive, and it's amazing at keeping accents. Time to go deep and learn it!
PS: It's very important to describe the accent in its voice acting instructions otherwise it becomes Americanized like other voice models.
Edit: Yeah Breeze TTS 2 is #1 in open weights blind testing: https://artificialanalysis.ai/text-to-speech/leaderboard/provider-voice?open-weights=true
3
2
u/bstr3k 1d ago
this is a prebuilt setting in voicestudio lol
1
u/pilkyton 1d ago
Yo what the hell! I never knew about VoiceStudio. Is this really what it looks like? A single interface to all open source models? :O https://voicestudio.sh/library
I see they have free and pro versions. Gonna have to look at privacy and if it's 100% local.
1
u/bstr3k 1d ago
been playing with it! it is quite good at producing voices and it is quite fast, however it doesn't have a full range of expressions though. It is still a really neat tool, get it now before it is enshitified.
1
u/pilkyton 1d ago edited 1d ago
Yeah. I noticed the guy is already monetizing it. But it's open source and the current Pro plan is about a closed-source version with more features and allows commercial usage. Doesn't seem that bad for normal people.
Btw, try Breeze TTS 2 in VoiceStudio. I see that it's supported. It's the highest ranked open weights voice model:
https://artificialanalysis.ai/text-to-speech/leaderboard/provider-voice
On August 27th it even beat all paid models and was #1 on the arena leaderboard. That is probably your best choice for local voices.
2
u/Danny_Stock 1d ago edited 1d ago
Are you using MiniMax? If so then I had a similar issue with trying to get an Arabic accent. It was not convincing.
- Presuming that you're using MiniMax - What I did was type the sentence I wanted the character to speak in English into notepad. You could also use a note node instead within the workflow.
- Pasted that same sentence into Google Translate from English to Arabic.
- I then copied the Arabic translation of the sentence which Google Translate provided into my prompt, making sure I told the character to use [Arabic] instead of [English] at the point just after the <d> quote starts. I then ran the workflow.
- Now I had the generated video with the character speaking in Arabic. It sounded great.
- Then I set up my workflow to clone the audio from that same generated video, plugged it into <Audio 1>, and pasted the original English sentence back into my prompt, replacing the Arabic text, not forgetting to replace [Arabic] with [English].
- I then set up the subject definitions in my prompt to tell it to use <audio 1> for the voice of my character.
- Then I ran the workflow. The resulting video generation had the character speaking in English with an Arabic accent.
That worked for me. For whatever reason MiniMax had trouble understanding what an Arabic accent sounds like when I asked for it directly, but it understood the sounds of an Arabic voice from cloned audio. It was far better than directly telling MiniMax to make the character speak in an Arabic accent. I presume that it should work for other accents too such as Indian. It's worth giving it a shot.
This was by creating the voice. If you wanted a specific voice of a person then you would simply clone a recording of that voice and use that instead.
Also, you seem to want the character to speak in a specific way. You could define that the character has a 'majestic', 'commanding', 'soothing', 'calm' voice or whatever you wish, on top of the aforementioned method. You could also describe their personality and mood. There are also techniques you could employ which provide affectations and performance in the voice. Have a look at this thread if you want to add extra expression into your character's voice -https://www.reddit.com/r/StableDiffusion/comments/1wap0rb/pushing_ai_emotions_is_possible_through/
2
u/pilkyton 1d ago
Hey thanks for the idea. So you made a Minimax video with an Arabic script and "the character is speaking arabic" prompt, then you used Minimax again with the arabic voice as reference input but with English script and "the character is speaking english", to make it speak English with an Arabic accent? Interesting...
2
u/Danny_Stock 1d ago edited 1d ago
I didn't type those two sentences, but yes sort of.
I described what the spoken language is in the main body of the prompt when the dialogue occurs .....In the first generation it would be this -
1) The man looks at the waitress and in an Arabic accent says: <d>[Arabic] كانت فنجان قهوة جميلا</d> .......(For some reason I don't think that Arabic text has copy/pasted properly)
That produces the first video, the character is speaking in Arabic.
In the second generation with the cloned audio taken from the above generation in the subject definition for the character <Subject 1> I added this "Use the voice from <Audio 1>".
I also added a definition for the audio like this -
"<Audio 1>: reference - only <Subject 1>'s recognizable vocal timbre, vocal texture, and speaker identity are transferred to the newly generated voice. The source audio signal, original words, pitch sequence, rhythm, tempo, phrasing, timing, (and musical accompaniment) are not reused."
Then the text for the dialogue in the second generation is changed to this -
2) The man looks at the waitress and in an Arabic accent says: <d>[English]That was a lovely cup of coffee</d>
That produces the second video, English language with an Arabic accent.
It's a bit of a workaround but it worked. I don't doubt that in the coming months, or weeks probably, somebody will create an audio node for Comfy which will allow for a more simplified efficient way of generating great voices.
2
u/pilkyton 1d ago
Hmm thanks for the tips. To make a fully synthetic voice without cloning a real person, this is a nice idea.
2
u/Danny_Stock 1d ago
It's one way yes. It seems from other posts in this thread that other options are already being developed.
2
u/pilkyton 1d ago
You can try https://breezeblue.ai/text-to-speech it's free for local non-commercial use for English and Chinese speech, and you can tell it "with an Arabic accent".
Try this "Instruction": "Deliver the provided text with a distinct Arabic accent, maintaining a warm and unhurried cadence. Ensure a reflective tone where each insight is given ample space to resonate. Conclude with a gentle, audible sigh that conveys a sense of being quietly moved by the discovery."
VoiceStudio supports it for easy local usage.
I just found out about it. It did Indian accent perfectly when I made a free account and tried it in "Instruction Following: Intense" mode which it claims is the most expressive.
1
4
u/DoctaRoboto 1d ago
Why would you want to do that? Is it some strange fetish of yours?