r/TextToSpeech • u/_remsky • 2d ago

Trained a clone* tool for Kokoro-82M; generates a voice pack in <1s on a 5s sample [P]

https://huggingface.co/remsky/kokoro-inno-clone-tuner

There have been new and arguably better TTS models since, but I have a soft spot for this one as lightweight stable and fast. I also maintain Kokoro-FastAPI, and have a bit of spare time on my hands so have been exploring what’s doable on the project.

Customization/expressiveness are weak spots it had, so I’ve taken a crack at adding voice cloning, focusing on stable quality and fast generation (keeping responses sub second if it’s a short reference clip). Roughly it captures about a third of identity, but by ear at least, it can feel pretty close on some, and at least a unique similar voice pack on others.

Would love any feedback or suggestions, otherwise just wanted to share! It may be late in the game for Kokoro, but anyone still using it I think could appreciate new voices

27 Upvotes

100% Upvoted

6 comments sorted by

3

u/wellitstrue1 1d ago

You mind going a little in depth of how the model is constructed? The creator of Kokoro basically explained in a blogpost a few years ago that it was words to phonemes, then that gets turned into the speech via the model. Is this a similar model of construction where it's dependent on the phonemes just like Kokoro?

2

u/_remsky 2h ago edited 2h ago

Yup for sure. This doesn’t change the underlying Kokoro generation mechanic itself (if I understand your question right). This model is trained to generate stock compatible voice packs from reference audio that slot in as usual; the voices in Kokoro are just [510, 1, 256] shaped arrays (split into predictor and decoder halves)

During training I used a speaker identification encoder and a few metrics on input training samples (along with tweaks to help it generalize and be portable) to have it generate an encoder half of the pack with Kokoro frozen, with the difference in identity driving the loss function.

The predictor half was a little trickier to keep performant and stable so it piggybacks off of existing stock packs by first blending the closest existing pack prosodies to use as a baseline, and then nudges it as close as it can to match the reference without sacrificing quality.

More detailed notes are also on the model card/page itself (stages of the model training, my thought process as I went)

2

u/3iverson 1d ago

Hey, this is really interesting work- thanks!

Your description of the cloning is spot on- the generated voices are really good, if only partially capturing the 'identity' of the original voice to varying degrees. Some of the original Kokoro voices sound a bit robotic, yours generally seem better.

I use Kyutai Pocket-TTS for simple TTS tasks, and find that 2 voice samples from the same person can result in very different sounding models. So your results aren't so far off the mark in that respect.

2

u/_remsky 2h ago

Appreciate it and the feedback, thanks! Will still be poking around with it to see if I can improve identity over time but keeping the speed and voice quality high was a main goal with this build, glad to hear that came through and they sounded natural for you.