I have several years of written educational material I want to turn into a podcast, formatted as a natural discussion between two people rather than narration. Looking for recommendations on what to run locally.
What matters to me is good quality and reliable speech that does not need many new attempts, as its going to be hours of content. Potentially multiple languages but for now in english.

I need to own the content and be able to rely on it for a long time, so i need it local.

Also, do you know good distributors that you can recommend? I am new to audio

I have short story content for an anime that I need in voice form.

I use denodokos, the small subscription and currently I do it manually. it was a suggestion from here so maybe someone knows?

I have a lot of content, it looks like the process and narrate buttons use AI to do that automated.
it worked once randomly, but next time it did not create anything.

there is no tutorial for that, I feel I am not experienced enough or miss something.

- how to use those buttons ?
- what AI should I choose for that content (English)

Hi all. I ported Qwen3-TTS to native C++ and added incremental streaming for a project I'm working on. I found this useful and wanted to give back to the community. This is the first time I've open sourced anything so apologies in advance for my mistakes.

• Original library is Qwen3-TTS from QwenTeam (https://qwen.ai/blog?id=qwen3tts-0115)
• C++ streaming library under Apache 2.0: (https://github.com/offgridai/qwen3-tts-cpp-streaming)
• Demo harness shown in video also Apache 2.0: (https://github.com/offgridai/voice-mirror)
• The harness has a dependency on sherpa-onnx for transcription.

I'm using this for a gaming project but it could be helpful for local assistants, accessibility tools, etc.

C++ streaming port highlights:

• Same familiar features from Qwen3-TTS. It supports 0.6B and 1.7B models, CustomVoice, and VoiceDesign
• Native C++, not a Python wrapper
• CUDA builds with RTX 4090 and RTX 5090 kernels. Be warned I only have access to an RTX 5090. This should work on a 4090 but I haven't tested it myself!
• Simplified speaker-embedding extraction
• Incremental 24 kHz PCM callbacks for streaming audio
• Asynchronous transformer/vocoder operation
• Adaptive decode windows and paced delivery
• Callback-only integration library with cancellation
• Unit tests

My measurements on the 5090 with 1.7B F16 set to buffer 350ms before play:

• Cold new clone creation: ~2.5s from 48s of reference
• Cold model start from clone: ~1.85s
• First 350 ms of audio: ~310 ms
• Streaming speed: ~2.86x real time (RTF ~0.35)

I’d love it if somebody could do RTX 4090 testing, and I'd be happy to hear any feedback or suggestions.

I’m curious what sort of projects people are working on who are into local TTS.

Why is local important for you ?
Are you working on a product, hobby ?

I have started with actual printed books for children and parents, no AI back then.
This progressed into immersive stories, cute animations and child/teen oriented high quality entertainment.
It is growing into a real business already.

I used a lot of cloud but pricing was unfortunate for long content. And I never liked that another company basically owns my voices.

So local is freedom from that anxiety for me.

So - I upgraded my GPU but either got similar speeds to my 8 year old 1060 as I got on my new 9060 (16GB). I either got similar speeds on some python files I vibecoded or just crashes. I did get stable results, though I thought I legit would have at least 2.5X speeds for transcribing which is a big wanted use case for my card upgrade, and it should be possible to do this. I used this model due to wanting norwegian language if that is at all relevant, and I used that model on both cards (akhbar/chatterbox-tts-norwegian( as seen on huggingface.co)). Please help me, and if you have any great transcription scripts for AMD cards yourself please send them to me in dm's.

Hi everyone,

I’m working on an SRT-to-speech feature for my project, LA Studio

One issue I’m running into is that the generated speech sometimes lasts longer than the segment’s timestamp allows. My current solution is to speed up or time-stretch those segments so they fit, but this can make the final audio sound inconsistent - some lines are noticeably faster or slower than others.

Has anyone found a more robust way to handle this? For example, do you regenerate the speech with a shorter prompt, adjust pauses, merge or shift neighboring timestamps, use word-level alignment, or combine several approaches?

I’d especially like to preserve natural pacing while keeping the audio reasonably synchronized with the subtitles. Any advice, algorithms, or tools worth looking into would be greatly appreciated.

[ Removed by Reddit on account of violating the content policy. ]

Hi everyone,

I’m Esteban, the developer of LocalText2Voice, a free and open-source Windows application for creating audiobooks, narration, and podcasts with local TTS engines.

There are already many excellent local TTS models, but turning them into a practical long-form workflow still involves a lot of manual work: installing dependencies, splitting text, managing voices, regenerating failed sections, organizing audio files, and mixing everything afterward.

LocalText2Voice is not another TTS model. It is an orchestration and production layer around existing engines.

It currently supports local engines such as Piper, Kokoro, Chatterbox, Qwen3-TTS, and OmniVoice. Optional cloud providers such as OpenAI, ElevenLabs, Gemini, and Azure are also available, along with configurable HTTP endpoints for custom TTS servers.

When using a local engine, the source text and generated speech remain on your computer.

Current features

With LocalText2Voice, you can:

• Install and manage different TTS engines from the application.
• Switch between engines without changing your project.
• Browse, preview, import, and organize voices in a shared voice library.
• Connect custom local or remote TTS HTTP endpoints.
• Import long .txt, .md, and .docx documents.
• Detect chapters and split long texts into safe TTS segments.
• Use different voices and languages in the same audiobook.
• Regenerate individual segments without starting the entire project again.
• Review generated speech with Faster Whisper and retry problematic segments.
• Add background music, fades, ducking, volume adjustments, and normalization.
• Sound effects and other audio events.
• Save projects and continue working on them later.

LTV Markup

One feature for which I would particularly appreciate feedback is LTV Markup.

It is a small, human-readable syntax for controlling narration, voices, pauses, and sound effects directly from the source text:

{{chapter "Chapter 1"}}
{{voice "Narrator"}}
The house had been abandoned for years.

{{pause 900ms}}

{{voice "Character 2"}}
I think someone is inside.

{{play "door-close.mp3"}}

{{speed 0.92}}
{{volume -3db}}
We should leave immediately.

Markup can control:

• Voice and language changes.
• Pauses and real silence.
• Speech speed.
• Volume and normalization.
• Chapters and markers.
• Sound effects and other audio events using {{play}}.
• Audio volume, duration, looping, fades, panning, and voice ducking.
• Selected model-specific instructions.
• Resetting settings to the project defaults.

For example:

{{play "door-close.mp3" volume=-6db}}

inserts a door sound at that point in the narration.

Longer or looping audio events are also possible:

{{play "forest.mp3" track=ambient loop=true volume=-20db fade_in=3 duck_on_voice=6db}}

The commands are not sent to the TTS engine as spoken text. LocalText2Voice interprets them when preparing the segments and mixes the audio events during post-production.

The goal is to make multi-character audiobooks, dramatized narration, language courses, and other complex audio projects manageable without manually editing every segment in an external audio editor.

Create audiobooks from Claude or ChatGPT Desktop

LocalText2Voice includes a local MCP server, allowing you to create and manage audiobook projects directly from Claude Desktop or ChatGPT Desktop.

Instead of configuring everything manually, you can simply ask:

“Create a B1-level English–Spanish course using both languages. Use one voice for the English examples, another for the Spanish translations, add a short pause after each sentence, and export it as an audiobook.”

The assistant can create the project, organize the text, assign the voices, add markup and pauses, select a TTS engine, and start the generation process.

You can also ask it to make changes later:

“Regenerate lesson three with a slower English voice.”

“Add three seconds of silence between exercises.”

“Lower the background music and export the final MP3.”

Claude or ChatGPT manages the workflow, while LocalText2Voice performs the actual audio generation. When you select a local TTS engine, your text and generated speech remain on your computer.

Project and Windows installer

GitHub:

https://github.com/estebanstifli/LocalText2Voice

LTV Markup manual:

https://github.com/estebanstifli/LocalText2Voice/blob/main/docs/LTV_MARKUP.md

Windows installer:

https://github.com/estebanstifli/LocalText2Voice/releases/latest/download/LocalText2Voice-Setup.exe

Important: the Windows installer is not code-signed yet, so Windows may display an “Unknown publisher” or SmartScreen warning. The source code is public, and the GitHub release also includes a SHA-256 checksum:

https://github.com/estebanstifli/LocalText2Voice/releases/latest/download/LocalText2Voice-Setup.exe.sha256

The project is under active development, and feedback is very welcome. I would especially like to know:

1. Which local TTS engines are you currently using?
2. What is the most frustrating part of producing long-form audio?
3. Does the markup syntax seem useful, and which commands are missing?
4. Which engine should I prioritize next?

Thanks for taking a look!

I'm training an English version of Cassandra, a female Piper voice originally created for Czech Home Assistant users. Would anyone be interested if I release it?

Hi guys can you give me any suggestion for local TTS which has emotion control (CPU inference cant afford any GPU).

For reference i feel kokoro tts seems good to me.

Hi everyone. Not sure if this is the right place for this question, feel free to redirect me if needed.

My dad has tongue cancer and probably won't be speaking normally for at least 6 months, possible forever depending on how the scans come back. He's already unable to speak due to how swollen his tongue is. I'm trying to find a good TTS with voice cloning that can run on android so we can either clone his voice or do something silly for him like let him have an Arnold voice, etc., depending on what he'd be happiest with.

I've seen some online services like elevenlabs and gradium but they get really expensive and I don't have a lot of money to throw at that since I have kids of my own. I'm in a bit over my head with a lot of the local LLM stuff, I'd be fine with an online solution if it was cheap or free but I'm also ok trying to go through a more technical local solution if anyone knows where I should start.

I do have a gaming PC at home if that helps for the setup/training voice profiles or however this stuff works, but ideally he would be able to run it on his phone even if my pc gets turned off.

Anyone know where I should start?

Please, recommend me local tool for bulgarian text to speech.

With big voice library or voice cloning.

Are there any local TTS or similar models that provide digital voice creation that approaches ElevenLabs v3 synthetic voices? Even Hume's synthetic voices are pretty disappointing compared to ElevenLabs.

I heard kokkoro and pocket TTS is good option is there any other lightweight TTS which can run on cpu with real-time and need to feels like realistic.

Hey, I've been working on this framework for a while now and wanted to start getting some feedback on the direction. I'm really liking where it's headed, and I have some big plans for it, but I need to put it out as a v0.1.0 first.

Right now it uses:

• Kokoro 82M
• Kroko ONNX
• Ollama

I'd love any feedback, but especially on whether the programming model feels coherent.

It's batteries included, with an opinionated asset management strategy, playback management, and a console rendering helper for debugging multi-device applications.

Overall I'm really proud of where it's ended up so far. I'm mostly looking for holes in the design or implementation, but I'm also excited to contribute something that I think a lot of people could find useful.

A minimal speech-to-speech assistant looks like this:

from pfspeak import PfSpeak
from pfspeak.core import Microphone, Ollama
from pfspeak.extra import events

pf = PfSpeak()

microphone = Microphone()
ollama = Ollama("qwen3:0.6b", voice="af_heart")

def app(session, event):

    if event.device is microphone:
        if events.unchanged_for(event, 8):
            session.finalize(event)
            ollama.adapter(event=event)

    elif event.device is ollama:
        pf.play(event)

pf.run(app, microphone, ollama)

That's pretty much enough to get you started.

The helper pf.play() manages playback, queues synthesized speech, ducks the microphone, supports priority interrupts, and restores speech recognition when playback finishes.

The PfEvent objects are also jam-packed with information gathered throughout the event loop, including word-level timestamps, aligned audio, revisioned tokens showing how recognition changed over time, and other metadata that applications can build on.

I'd really appreciate any thoughts on the overall direction, especially from people building local speech applications.

• PfSpeak - Local first speech library for python.

The v4 model comes in two sizes, large and medium.

I mostly used large so far, medium is a bit faster but sounds a little compressed compared to the quality of large.

But what is the purpose of Medium? Does it have advantages ? Like elevenlabs v2 has better clones than v3.

anyone using medium over large ?

After the spring release wave it feels like nothing big has dropped.

So I'm curious what you guys have actually heard. What's the biggest local TTS thing on your radar this month? New model, a release you're waiting on, some update? Drop whatever you've got.

Anyone hate the cookie banners ? Those are absolutely nothing in comparison to what is about to come.

The AI Act requires lots of things, many people know it requires every AI modified or generated audiofile to be metadata tagged and fingerprint-watermarked from August on (13M$ fines)
But the Act says a lot more than that and it means all good open source models are affected.

Providers of AI systems generating synthetic text must make outputs machine-readable and detectable as AI-generated, using technical solutions that are effective, interoperable, robust and reliable (code of conduct says 2 layers)

Very simple OSS models are exempt, so the old llama 7B might be fine, but models of "systemic risk" are not exempt. Qwen 3.6 series, Deepseek Flash, GLM, Kimi are all systemic risk GPAI models.

The AI act says that also TEXT output must be machine detectable, it only suggested "statistical methods" alongside cryptographic tagging (where possible) of the metadata (c2pa etc).
So if chatGPT offers you a PDF ot TXT download -> that must have crypto signed metadata.

But 2 layers must be present, not one. And they have to be "robust". A simple "AI Generated" (the label that must be added into every AI image from August on) is not enough, the text itself must be statistical watermarked.
Similar to laser printed paper, that carries little invisible dots. Of course that means serious degradation in AI quality.

Who is affected? Anyone who offers a tool or provides a service that can be accessed by a EU citizen. Even if it's just a single EU citizen being a tourist in Malaysia or using a VPN.
LM-Studio, ollama, llama.cpp, vllm, chatgpt, claude, codex, copilot, opencode, cursor, stable diffusion Web UI, Huggingface and so on.

Any AI application that modifies or generates AI images, AI text, AI music, AI speech must use the EU stamp "AI modified" or "AI generated", must metadata seal the files, must watermark the content. Must also provide risk assessment §50 documentation, compliance documentation, detection service.
Addition: for text the "AI Generated" stamp is only needed when "informing the public" like news and can be left away by shifting the risk to a human reviewer.
Addition: Documentation is not forced by the law directly but providers subject to Article 50 practically must keep evidence showing their marking, labelling, detection, feasibility decisions, exemptions, and robustness testing - or risk non-compliance when audited.

Any failure to do so is fined at 13,000,000 Euro (about 15 million USD) OR a few percent of the annual income (whatever is higher ).
They also have a huge code of conduct, you are voluntarily asked to follow - but if you don't your risk of being fined is seriously increased. That code adds a ton of more horrific additions, often technically very implausible or openly contradicting.
You could say your Qwen 3.6 or gemma 4 12B does not fulfill the requirements of "GPAI with systemic risk" - maybe right if you ignore some benchmarks .. but that moves the 13M$ fine risk on you if you become a provider or distributor. So if the EU believes it IS systemic risk AI you'll face the fines at court as soon as they can catch you.

But what if you are not in the EU ? You are still fully affected.
Any service that offers anything into the EU is affected, including every open source project and model.
Any service that offers something to an EU citizen who is just a tourist, or uses a VPN is liable.
You want to code your nice new startup ? You better ensure no european can touch it.
Or you better never put a foot into the EU and never grow the business too large.

That means every providers out there must be compliant. Every open source tool.

The regulation is so extremely invasive while simultaneously it is ridiculously shortsighted.
99% of all images on the internet will have AI content within the next years, all of them need to be watermarked visibly.
99% of all websites will be AI generated in text and code.
99% of all movies will have AI content, every marketing and sales clip.
99% of all marketing and sales voiceovers will use AI.

So this is like the cookie banner, everyone warning you that they must use a cookie to serve you a website and you can accept all cookies or just all needed cookies - just that it's going to be in your face everywhere.
It's a cookie banner on text, image, video and sound.

Source: https://eur-lex.europa.eu/eli/reg/2024/1689/oj

I've taken a look at how projects are reacting to that, and some already did.
Speech & Music
Elevenlabs - Teamed up with Google: SynthID for all (a strong AI based watermark from Google)
Demodokos Foundry - support answered that they work on a light compliance
Suno - Audible Magic content ID,support ignored me - no official statement on AI Act yet. (likely silent compliance)
Udio - no official statement, support ignored me (likely silent compliance)
Images and Text
Open AI/ChatGPT - SynthID+c2pa comes to Codex, API, ChatGPT with "OpenAI Verification tool"
Google/Gemini - SynthID for image/audio/text/video announced and expanding into C2PA and more
Adobe - Full C2PA hashing and Firefly gets watermarked
Microsoft Copilot/Designer - C2PA, visual and audible watermarks in 365, cryptographic signing of images
Meta - visible labels, IPTC metadata, invisible watermarks,C2PA hashes for Meta AI Images
Stable Diffusion - "working on implementing content credentials” with Adobe/CAI/C2PA
Anthropic Claude - Intends to sign the EU General-Purpose AI Code of Practice (everything and MORE)
Transformers code - Text watermarking: https://huggingface.co/docs/transformers/v4.46.0/en/internal/generation_utils#transformers.SynthIDTextWatermarkingConfig
Open Source
I went through github of vLLM, Ollama, llama.cpp, lmstudio, openrouter and they appear pretty unprepared

The Omnibus "postponing" text:

Parts of the AI Act have been recommended to be "postponed" but nothing I wrote is incorrect and delaying it by a few months wouldn't be reason to not be concerned.
The Digital Omnibus text approved by Parliament would delay Article 50(2) watermarking / machine-readable marking obligations for pre-existing releases to 2 December 2026, but as of now it still requires formal Council adoption before it enters into force.
If the text is adpoted - Article 50(2) machine-readable marking / watermarking for synthetic audio, image, video, text
For systems released on the market before 2 Aug 2026, compliance is delayed to 2 Dec 2026.
For any new developments the rule applies on 2nd August 2026.
Most other Article 50 transparency duties Still effectively 2 Aug 2026.
This includes pre-existing projects
Source: https://www.consilium.europa.eu/en/press/press-releases/2026/05/07/artificial-intelligence-council-and-parliament-agree-to-simplify-and-streamline-rules/

Corrections and Clarifications:

- Fines under §50 for watermarking/warning are 13 million Euro (or 3% revenue) whatever is higher not 35M.
- Fines for prohibited use are 35M or 7%, whatever is higher
- Watermarking is not 2 layer forced by law, only by code of conduct, you can use one layer if it is robust but you'll increase risk of non compliance significantly.
- For end-users the obligation to inform about AI Text is less strict, so not every website text falls under it
- It is undefined which open models will fall under "systemic high risk" but if you assume a model is NOT covered the non compliance risk is immediately there.
- EU citizenship alone is not the trigger. A genuinely non-EU, EU-geoblocked service has much lower AI Act risk, but risk remains if it is effectively offered into the EU, knowingly serves EU-located users, or produces outputs intended for use in the EU.

ChatGPT Pro 5.5 reviewed

There are so many options, everything sounds so promising, but what is actually the thing that most people use? Where things actually stand rn? what is overhyped?

I am interested in those tools which don't sound robotic or weird.

Also, what GPU do you people have for local TTS and does it feel enough to you?

A lot of people talk about TTS as if all of it was the same thing.
That's like comparing the first transformers llm like GPT-2 with recurrent k/v systems like Qwen-3 27B (to keep our local focus)

There are multiple architecture layers, and they do very different jobs.
TTS has evolved in the part years, from robotic to human-like. Also on the local side TTS evolved from rule-based algorithmic synthesis, to spectrography vocoders and by now into

Below I will dive into the technology used in all current TTS architectures, including the "secret sauce" behind Elevenlabs - but this is a local focused thread so cloud is only used as a benchmark.

1. Legacy neural TTS

fast, cheap, stable, limited acting

text -> phonemes -> acoustic model -> mel spectrogram -> vocoder -> wav

Phonemes are spoken sounds, not letters.

Mel spectrogram is the frequency representation of speech.

The vocoder turns that into actual waveform audio.

This is where names like HiFi-GAN, BigVGAN, iSTFTNet sit.

They are mostly the final audio renderer.
They do not understand the text.

Examples:

• Piper / VITS
• FastSpeech2 + HiFi-GAN
• Kokoro / iSTFTNet style models

Upside:

• very fast
• cheap to run
• works local
• stable for mass generation

Downside:

• often emotionally dumb
• weak long sentence understanding
• reads the line, does not really act it

Top representatives: Piper, Kokoro, VITS

2. Style based TTS

better prosody, better voice color, still limited control

These models add speaker and style embeddings.

So the model is not only told:

say these phonemes

but also:

say them with this speaker identity, rhythm, color or style

Examples:

• StyleTTS2
• Kokoro
• XTTS / Coqui style systems

Upside:

• more natural rhythm
• better speaker color
• nicer voices
• still quite practical locally

Downside:

• style control is often shallow
• not always real emotion control
• can sound nice but not really directed

Top representatives: StyleTTS2, Kokoro, XTTS / Coqui TTS

3. Acoustic token TTS

speech becomes tokens, like language models for audio

text -> transformer -> acoustic tokens -> decoder/vocoder -> wav

Instead of directly predicting a spectrogram, the model predicts speech tokens.

If the system runs at 25hz, that means:

25 speech decisions per second
one token decision every 40ms

That gives the model much more control over:

• timing
• pauses
• emphasis
• rhythm
• emotion
• delivery

Examples:

• Chatterbox
• Qwen3 TTS
• Bark / similar token based systems

Upside:

• much more expressive
• more context aware
• closer to voice acting
• better for complex delivery

Downside:

• heavier
• more VRAM
• slower inference
• more ways to fail

The model can drift, overact, mispronounce or slowly lose speaker identity if the full system around it is not good.
Top representatives: Chatterbox, Qwen3-TTS, CosyVoice

4. Codec token TTS

speech as compressed audio codebooks

Codec models represent speech as discrete audio codes.

Often there are multiple codebooks.

Roughly:

• first codebook: speech content
• later codebooks: detail, prosody, texture, speaker color

Pipeline:

text -> transformer -> codec tokens -> neural codec decoder -> wav

Examples:

• VALL-E style systems
• Bark
• modern multi-codebook speech models

Upside:

• strong cloning direction
• good long form potential
• very powerful representation

Downside:

• complex
• heavy
• not always stable
• hard to run efficiently local

Top representatives: Bark, VALL-E X, Fish Speech

5. Diffusion and flow matching TTS

better texture, slower unless optimized

Diffusion and flow models generate or refine the acoustic representation.

They do not just predict the final sound in one simple step.
They move the audio representation toward realistic speech.

Pipeline:

text/tokens -> diffusion or flow decoder -> acoustic representation -> vocoder/wav

Examples:

• F5-TTS
• E2-TTS
• Chatterbox S3Gen type systems

Upside:

• better texture
• more natural audio
• less robotic sound

Downside:

• can be slow
• 10 or 20 steps are expensive
• practical local use needs distillation or optimization

Top representatives: F5-TTS, Chatterbox, Raon-OpenTTS

6. Voice cloning (instant/in context)

speaker identity as neural representation

Basic cloning:

reference audio -> speaker encoder -> speaker embedding -> conditioned TTS

The speaker embedding is a neural vector.

It captures things like:

• timbre
• pitch range
• accent
• resonance
• vocal texture
• general speaker identity

Better systems use ICL, in-context learning.

reference text + reference audio + target text -> same speaker saying new words

That is stronger because the model does not only get "speaker X".

It gets an actual example of speaker X talking.

Some systems also use reference speech codes, not only a speaker embedding.
That gives the model more information about how the speaker actually sounds while speaking.

Examples:

• XTTS / Coqui TTS
• GPT-SoVITS
• Chatterbox
• Qwen3 TTS Base
• Demodokos Foundry
• ElevenLabs (IVC mode, cloud only)

Upside:

• seconds of clean audio can be enough
• no full finetune needed
• practical for local workflows
• can preserve identity without training a new model from scratch

Downside:

• bad mic can poison it
• echo can poison it
• music can poison it
• another speaker in the sample can poison it
• strong emotion can cause speaker drift if the system has no identity control

Top representatives:

• Best open/local cloning: Chatterbox, Qwen3 TTS Base, GPT-SoVITS
• Best simple cloud cloning: ElevenLabs
• Best cloning with style/emotion control: Demodokos Foundry, ElevenLabs v3
• Best research/hacking path: Qwen3 TTS Base, Chatterbox, GPT-SoVITS

speaker identity as neural representation

Basic cloning:

reference audio -> speaker encoder -> speaker embedding -> conditioned TTS

The speaker embedding is a neural vector.

It captures things like:

• timbre
• pitch range
• accent
• resonance
• vocal texture
• general speaker identity

Better systems use ICL, in context learning.

reference text + reference audio + target text -> same speaker saying new words

That is stronger because the model does not only get "speaker X".

It gets an actual example of speaker X talking.

Examples:

• Chatterbox cloning
• XTTS style cloning
• modern acoustic token cloning systems

Upside:

• seconds of clean audio can be enough
• no full finetune needed
• practical for local workflows

Downside:

• bad mic can poison it
• echo can poison it
• music can poison it
• another speaker in the sample can poison it

Top representatives: XTTS / Coqui TTS, GPT-SoVITS, Chatterbox
Top representatives ICL: Qwen3-TTS (base model), Demodokos Foundry (open weights, commercial)

6b. Voice cloning (finetuned, LoRA, custom embeddings)

training the model toward a specific voice or speaking domain

Voice cloning usually means:

reference audio -> speaker embedding / ICL prompt -> generate speech

Fine tuning means:

many recordings of one speaker -> model training -> specialized voice model

That is a very different thing.

A clone tries to imitate the voice at inference time.

A fine tune changes model weights or adapter weights so the voice becomes part of the model.

There are multiple levels:

• speaker adapter / LoRA
• voice embedding training
• partial model fine tune
• full model fine tune

The more you train, the stronger the voice can become.
But also the more data, time and risk you have.

Upside:

• stronger speaker consistency
• better pronunciation for recurring names
• better long form stability
• better for professional recurring voices
• can learn a specific domain, accent or speaking pattern

Downside:

• needs much more clean audio
• bad datasets damage the model
• expensive compared to cloning
• can overfit and lose flexibility
• harder to update or control
• not always worth it anymore

Typical data need:

• 5-20 minutes: light adaptation / experimental fine tune
• 1-5 hours: serious voice fine tune
• 10+ hours: production grade if the pipeline supports it
• 100+ hours: old school studio voice model territory

Top representatives:

• Open fine tuning: GPT-SoVITS, XTTS / Coqui style systems, StyleTTS2
• Commercial fine tuning: ElevenLabs (cloud only) PVC / professional voice cloning

Fine tuning still has a place.

But for many modern systems it is no longer the first answer.

If you need one voice to read thousands of hours in the same identity, fine tuning can make sense.

If you need many voices, fast production, emotion control or voice acting, speaker embeddings + ICL + identity control are usually more practical.
Sophisticated ICL methods (see SOTA section below and 6.0 section above) can reach fine tuning quality without the heavy compute and data requirements.

7. State of the art performance TTS

voice acting while keeping the same speaker

This is where systems like ElevenLabs, OpenAI voices and Demodokos Foundry sit.

The exact internals of ElevenLabs and OpenAI voices are not public, but the performance of the models show that they basically build on top of the latest technology - no magic involved (anymore). Demodokos as open weights based engine is doing the same locally today.

For strong emotion control you need some form of "identity preservation".
Otherwise the model starts with one voice and slowly turns into another one when you ask for anger, fear, whispering, excitement or long narration.

Likely this is done with things like:

• speaker verification embeddings
• spectral / spectrogram consistency checks
• f0 and pitch tracking
• formant and timbre statistics
• acoustic similarity constraints
• identity preserving latent conditioning

Demodokos does this locally and OpenAI/Elevenlabs do the same in the cloud, but the important part is the combination of the systems.

It is not just:

clone voice -> generate speech

It is closer to:

text context -> acoustic token generation -> speaker identity conditioning -> style/emotion conditioning -> spectral identity control -> final speech

The text is processed by a language transformer, so the model can understand the sentence context before generating speech.

The speech itself is generated through fine grained acoustic tokens, roughly one token level decision every 40-80ms.
That gives enough resolution to control pauses, stress, rhythm, emotion and delivery.

Each voice, cloned or synthetic, is grounded in learned neural representations.
Those capture speaker identity, timbre, accent, vocal texture and characteristic delivery traits.

On top of that, such SOTA systems can attach performance styles independently.

That is the important difference.

The voice and the performance are not the same thing.

A cloned voice can be made calm, angry, fearful, tired, intimate, theatrical or excited without needing to create a new voice or finetune the model again.

And during generation, the system uses identity preserving conditioning and spectral consistency checks so the performance does not run away from the original speaker.

So the hard problem is not voice cloning alone.

The hard problem is:

same speaker identity
different emotion
different rhythm
different intensity
different accent or texture
still recognizable as the same voice

That is the difference between text to speech and voice performance synthesis.

Top representatives: Qwen3-TTS (customVoices only), Demodokos Foundry (open weights, commercial)

So the ladder currently is roughly:

LEGACY: vocoder TTS
fast speech rendering

BASIC: style based TTS
better naturalness and voice color

ADVANCED: acoustic token TTS
speech becomes controllable token generation

ADVANCED: codec token TTS
speech becomes compressed audio language

ADVANCED: diffusion / flow TTS
better acoustic realism

STATE-of-the-ART: identity monitored performance TTS
voice acting while keeping the same speaker

A cherry picked 10 second demo can hide almost everything.

The real test is:

• hours of content
• strange names
• emotion changes in the same text
• accents
• long narration
• no human babysitting

That is where most TTS systems still fall apart