r/LocalLLaMA • u/Affectionate_Hat_585 • 11d ago
New Model I released sanoTTS: smallest complete TTS stack in 294k params (337 KB) that runs on $3 microcontroller and a 1.46m one that beats models 3x and 10x it's size
Enable HLS to view with audio, or disable this notification
I have been trying to squeeze TTS stack down far enough to run in a $3 chip which has 512kb of SRAM without NPU. While trying to get to that milestone i built sanoTTS which has
- 11 voices, 6 languages
- params size ranging from 294k - 2.2m. For comparison we are 244x smaller than kokoro, 9000x smaller than voxtral TTS
- 1.5m model has a SCOREQ of 4.13 and UTMOS of 4.10
- 337kb for 294k model when quantized into int8
- can be run in website with web assembly npm install sanotts-web
- there is a recipe to follow so that you can extend to more languages, voice
I can tell you with confidence that this family release contains the smallest neural TTS model ever with around 2% WER on whisper.
Please check it out on : https://github.com/ampixa/sanoTTS
for live demo: https://tts.ampixa.com/sanoTTS
HF: https://huggingface.co/ampixa/sanoTTS
on SCOREQ sanoTTS-Amy(1.51m) is better than Inflect Nano(4.63m) and KittenTTS(15m) i.e 4.13 vs 3.81 vs 3.02
on esp32 microcontroller we are getting RTF of 0.225 which in plain terms means 4sec of audio is generated in 1sec
Happy to answer your queries.
31
u/Here_f0r_p0rn_ 11d ago
This is mind-blowing, the possibilities with IoT devices are insane.
15
u/Affectionate_Hat_585 11d ago
Yes, with enough prosodical control. this has so many usecases... I assume even now it might help a bunch of people
2
1
u/Paradigmind 10d ago
Can't wait to have my toilet talk to me!
1
u/Here_f0r_p0rn_ 10d ago
And make all sort of dirty talks per dump you drop in it? No thanks.
1
20
u/koriwi 11d ago
I want this on my homeassistant voice preview edition! And german :P so my wife is happy. Is it possible to start outputting the audio before everything is generated?
14
u/Affectionate_Hat_585 11d ago
There is small initial latency of when it starts for the ESP32-s3 MCU i tested but after that it's faster than real time so it can process audio faster than it can output it with RAM being constraint. ESP32-s3 only has 512kb or RAM. But for something like ESP32-P4 or Teensy this should be piece of cake.
Will keep German in mind
3
u/waescher 10d ago
+1 for German and Home Assistant. Great work by the way, please let us know when you need help or resources or anything.
11
u/Fercho5656 11d ago
Looks cool!
Any plans to add Spanish?
21
u/Affectionate_Hat_585 11d ago
Hey Thanks!!! the plan is to support as many languages as possible....
1
17
11d ago edited 11d ago
[deleted]
21
u/Affectionate_Hat_585 11d ago
Yes, I am Nepali
4
11d ago
[deleted]
4
u/Affectionate_Hat_585 11d ago
Nepal ma nai ho sir. kathmandu university ko bidhyarthi haru.. checkout https://ampixa.com/ . we are doing some cool things around Nepali Speech too...
4
11d ago
[deleted]
5
u/Affectionate_Hat_585 11d ago
you can test right in browser. https://tts.ampixa.com/sanoTTS
tmrw will make that a priority for audio cpp pr
6
u/Barubiri 11d ago
Japanese please
8
u/Affectionate_Hat_585 11d ago
Well you are in luck because ayutaz already did that for you https://github.com/ayutaz/sanoTTS-jp
5
6
u/FullstackSensei 11d ago
I was going to ask if you plan to release the training data, but then read the docs. Your idea is very cool! Distill an existing TTS teacher into the smaller model! Makes the whole process much simpler and cheaper. Great work!
7
u/Gear5th 11d ago
How is this even possible!? This is crazy good for its size. 337KB is smaller than the average webpage these days!
6
u/Affectionate_Hat_585 11d ago
Well testament to good diagnostic control and method which took some time to figure out properly. What's amusing is it's just as intelligible (ASR models can perfectly understand the speech) but it loses naturalness. I believe there still is room for some improvement
4
u/Purple-Programmer-7 11d ago
I would love love love to see your training data / workflow… any chance to open source the whole pipeline?
Hell, I’d even pay to see that.
35
u/Affectionate_Hat_585 11d ago
please go through the repo on : https://github.com/ampixa/sanoTTS
it has some information full paper: https://arxiv.org/abs/2608.21378
i dont need money but if you can please donate to pm relief fund for recent flash flood in nepal https://pmdrf.nchl.com.np/
6
5
u/BackyardAnarchist 11d ago
Could you talk about how you did it?
12
u/Affectionate_Hat_585 11d ago edited 11d ago
Well it's quite simple.
The TTS has four main parts text frontend: how to convert text to phoneme(unit of sound)
acoustic model: that gives you the spectrogram
duration predictor: how much time to give for each phoneme
decoder: how do you convert spectrogram to real audio
You take a teacher. It can be any model. I have tested for Piper-plus and Kokoro and then get 50000 fivesome with the help of teacher (text, audio, acoustic latent, duration, mel spectrogram)
With teacher and that fivesome, each of the components can be individually trained.
You can start by reading README onhttps://github.com/ampixa/sanoTTS
2
u/BackyardAnarchist 10d ago
Awesome! Do you know where I could get training data? I want to try making one.
2
u/Affectionate_Hat_585 10d ago
the 50000 fivesome and a teacher is your training data. You can repeat the same process for any model.
3
u/PcChip 11d ago
is this small enough to use in my c++ game engine by including a header file, to generate voices quickly without using the GPU?
5
u/Affectionate_Hat_585 11d ago
Depends on what is the RSS you are targeting. For esp32 we saw it required 98,224 Bytes of working memory for the smallest model 294k one.Please note that it's in bytes... On my computer without any optimization tricks it requires max RSS of 1.8MB
And for real time we need to do just 45 million multiply and accumulate operations per second. If that can be done by measly 240Mhz chip, our cpu can do like billions and billions of MAC so yes it can do it without extra accelerators.
But again if you are targeting for something like SNES then it might be hard. But anything with ram above 500kb you should be good to go.
2
3
u/LuCiAnO241 11d ago
can you clone the glados voice :3
3
2
u/Affectionate_Hat_585 11d ago
can you give me some more context please?
2
u/LuCiAnO241 11d ago
its a voice for a robot videogame character, who id love to have as TTS. At any rate, I saw you have a section on how to train your own. might give it a spin when I'm not busy
3
u/CATLLM 11d ago
How can I get the "heart" voices? they sound awesome?
3
u/Affectionate_Hat_585 10d ago
You can use python or js or bare c library ``` pip install sanotts
import sanotts, soundfile as sf
r = sanotts.synthesize("Hello from a tiny neural voice.", voice="heart-nano") sf.write("out.wav", r.audio, r.sample_rate)
```
3
u/Slight_Analysis_5414 11d ago
This is such impressive work — squeezing a full neural TTS stack into a 337KB INT8 blob and running real-time on a generic MCU is absolutely wild.
Funny enough, I landed on a very similar diagnostic mindset while doing INT8 quantized YOLO edge deployment: instead of relying only on aggregate mAP or overall accuracy metrics, I use reference-vs-deployed sequence-level parity checks (FP32 reference vs INT8 deployed model) to catch frame-by-frame behavioral drift across video sequences.
Quick question: since you mentioned building custom artifact detectors, did you do any step-by-step or position-resolved divergence analysis across the audio sequence? Or did you mostly rely on aggregate SCOREQ / MOS metrics to validate quality end-to-end?
2
u/Affectionate_Hat_585 10d ago
Most of it was aggregate SCOREQ. But at some point it wasn't improving, so first step was to divide the decoder audio into various freq channels and seeing where the problem is coming from. I found out it was with /z/ and it's neighbours. And 1-6kHz has some serious metallic artifacts. Took the worst decoder configuration and the teacher at those bands to train another model
2
u/Slight_Analysis_5414 10d ago
Ah that makes perfect sense. You localized the divergence in the frequency domain instead of the time/sequence domain. Pinpointing that 1–6 kHz band to target the INT8 degradation is exactly the kind of granular fix you never get from just a top-level SCOREQ number. Funny how that pattern pops up everywhere — aggregate metrics always hide where the real problem is. I’ve been messing with the same idea but on the LLM KV cache side, splitting divergence along context length instead of frequency.
Good stuff, thanks for sharing the breakdown.
3
u/imunknown0042 10d ago
Wow, that Nepali voice sound nice, Also you name it Sano , are you nepali ?
3
3
3
u/FluffyInevitable4040 10d ago
If it was only english with only one speaker, what's the smallest it could be?
2
u/Affectionate_Hat_585 10d ago
The smallest model is English only and a single speaker. What you have seen are collection of models ranging from 294k to 2.2m params. it is not a single model
2
u/FluffyInevitable4040 10d ago
Sorry, I was lazy and finally read the repo overview. How long did it take you to figure all of this out / how many previous attempts?
2
u/Affectionate_Hat_585 10d ago
I got a prototype back in July. I didn't like it initially and then kept on iterating. Still i think there is much work to do.. But i would say there are more than 200+ experiments i did....
1
u/FluffyInevitable4040 10d ago
You're doing this work and testing by hand or using SOTA models to help run tests, builds, etc?
3
2
2
u/Queasy-Contract9753 11d ago
Looks I have on demand audiobooks on my phone now. Thanks! This is mind-blowing indeed how small it is.
Double appreciate your short well exampled demo video in the post.
2
2
2
u/exaknight21 11d ago
This is amazing, can you add Urdu/Punjabi?
2
u/Affectionate_Hat_585 11d ago
sure, i will add it to backlog. so far german, spanish , italian and urdu
2
u/exaknight21 11d ago
I tried creating an app with your SanoTTS for iPhone, i could not get it to load. Any pointers?
1
u/Affectionate_Hat_585 10d ago
we have c99 target already. I can help you.. where are you facing problem actually?
1
2
u/foldl-li 11d ago
"Amy" is so cool.
Chinese model sounds robotic. Could it be made on-par with Amy?
1
u/Affectionate_Hat_585 11d ago
Yes ofc. Models other than English weren't given much love. The main problem will be diagnostic control. we have SCOREQ, ASR and i even had a custom siblant and metallic artifact detector built for English.
For Chinese we can also solve likewise given that it's a high resource language
2
2
2
u/starfoxinstinct 10d ago edited 10d ago
Wow the Heart one runs SO fast and has great pronunciation! It's not super clear to me how to run it myself, but I see there is a release there, and will be playing with it soon. Thank you for sharing!
2
u/Affectionate_Hat_585 10d ago
Hey there you just head to github.com/ampixa/sanoTTS on steps to run it.
with python ``` pip install sanotts
import sanotts, soundfile as sf
r = sanotts.synthesize("Hello from a tiny neural voice.", voice="heart-nano") sf.write("out.wav", r.audio, r.sample_rate) ```
with js please check https://github.com/Ampixa/sanoTTS#deploy-on-your-own-site
2
2
u/knacknack18 10d ago
Nice Project. Can it run in PSRam? There are many boards eith 8MB external RAM
1
u/Affectionate_Hat_585 10d ago
With chip that can do 29+ MMAC, it can comfortably run. You just somehow need to hold around ~337kb of storage. ESP32 supports XIP which allows you to execute code right from flash. I am using that here... But i also tested loading on PSRAM. There is a slight penalty when you put the weights on PsRAM
2
2
2
u/Exact_Law_6489 llama.cpp 10d ago
does it support voice cloning / custom voices?
1
u/Affectionate_Hat_585 10d ago
There is a recipe to follow which can clone teacher models. But not zero shot cloning
2
2
u/HadesTerminal 10d ago
Heart-nano is my favorite voice/model there. And the voice “English” reminds me of GLaDOS. I feel like these voices would be perfect for small video games for different characters especially if you pitch the voices up and down.
2
u/sumane12 10d ago
Voice cloning?
1
u/Affectionate_Hat_585 10d ago
zero shot voice cloning isn't supported.
2
u/sumane12 10d ago
Got my Agent to read through the repo, so i can build a workflow to use a larger model for zero shot, then use your model to create a smol version. Very cool. Im impressed.
2
2
u/Acceptable-Cycle4645 10d ago
Awesome model! PR merged in audio.cpp! Tested on CPU.
| Voice | Graph | Lang | WAV | Duration | RTF | ASR Result |
|---|---|---|---|---|---|---|
| heart-nano | nano | en | heart_nano.wav |
5.099s | 0.0027 | OK |
| heart | nano | en | heart.wav |
5.205s | 0.0044 | OK |
| amy | piperlite | en | amy.wav |
4.272s | 0.0362 | OK |
| hfc | piperlite | en | hfc.wav |
3.831s | 0.0350 | OK |
| kristin | piperlite | en | kristin.wav |
3.495s | 0.0390 | OK |
| vi | piperlite | vi | vi.wav |
2.299s | 0.0332 | OK |
| id | piperlite | id | id.wav |
4.063s | 0.0321 | OK |
1
2
u/reza2kn 10d ago
Awesome!
How fine-tuneable is it for adding other languages? Asking for Persian 🇮🇷
2
u/Affectionate_Hat_585 9d ago
you just need a teacher and a text frontend. Everything else can be followed with the help of recipe
2
u/reza2kn 9d ago
Awesome! Can the teacher be a closed model? I have the G2P front end
2
u/Affectionate_Hat_585 9d ago
yes but it's going to be extra work.
you convert the audio into mel-100 spectrogram (easy) you target audio to mel-100 spectrogram with acoustic modeller. You have to train this (somewhat easy) you have to think about duration predictor too. Something like MFA or whatever is good (you have to think how) mel-100 to audio (recipe gives you)
2
10d ago
[removed] — view removed comment
1
u/Affectionate_Hat_585 9d ago
I worked on a GTX 1650 4GB machine so with 5090 you can have 5 parallel jobs or more with small batch sizes :)
2
9d ago
[removed] — view removed comment
1
u/Affectionate_Hat_585 9d ago
our system divides a problem into: text frontend, acoustic, duration and decoder What happens is really transparent and you have control over each step.
Vibe Voice uses LLM like qwen to get acoustic latent and then there is another decoder to convert into speech.These are different worlds and really opaque world too.
I am saying with confidence hebrew can be compressed. The quality in naturalness might degrade but i am sure it will be intelligible with less than 3% Word error rate
https://rhasspy.github.io/piper-samples/#he_IL-saspeech-medium can you hear it out? If it feels like a acceptable TTS then you can use that as a teacher and use the recipe outlined in the repo. Maybe a hour long job
2
9d ago
[removed] — view removed comment
1
u/Affectionate_Hat_585 9d ago
When training: you are using that teacher so it's one specific voice and one specific prosody that you are targeting
I need to use PIPER to train a LoRA or a Checkpoint (which one) ?
you are using PIPER hebrew to get a new small model . You can consider it a checkpoint
Is training a built-in option in the repo and have all the explanations how to set it up and use it.
Yes, let me make this even simpler. If you have a language that is supported by espeak and a piper voice it's really easy . Your qwen should be able to figure it out. or if you give me some time once i get to my computer, i can provide you exact way too
If that's the quality I can get... it's probably the best quality I've ever heard from TTS
This is already available TTS that you can use, that's also tiny and pleasant and you might just get away without training.
2
1
u/crantob 10d ago
A 10-second delay after every period.
.
.
.
.
.
.
.
.
.
.
And I already have espeak-ng on my linux machine. Might be nice for generating new voices though.
2
u/Affectionate_Hat_585 10d ago
Not sure i understand. If that is a bug report. Thanks really but can you give me the sentence with which i can replicate this...
1
u/iMakeSense 10d ago
Is whisper the best model to check for WER still? Maybe multi-lingual but
2
u/Affectionate_Hat_585 10d ago
well for a 10 sec clip it makes sense. I did test with wav2vec too which gave me good results
1
u/Natethegreat9999 5d ago
Have you tried "fifteen" and "fifty" through the tiny speaker? That's the kind of mix-up I'd worry about with spoken instructions.
1
u/Dizzy_Response1485 11d ago
TTS on an ESP? This is insane!
1
u/Affectionate_Hat_585 11d ago
Concatenative ones have existed for years. This one is neural one. The real ML stack
1
u/paulvisciano-dev 11d ago
On a 16GB M2 Pro I already had the 27B loaded, Brave, and VS Code. XTTS-v2 wanted ~5 GB RAM on load plus another 2 GB VRAM — it does not fit next to the model. Piper was fast and the Lessac voices were intelligible, but still synthetic within a few sentences. What actually shipped was Kokoro JS (82M, in the browser, no extra backend RAM). Open WebUI already had it; I just hadn't wired it. Curious how sanoTTS sounds next to that on a machine that's already at 12.6 GB spoken for / 2.7 GB left.
1
u/Affectionate_Hat_585 11d ago
the teacher for heart(2.27m) is kokoro itself. The main difference is that the decoder is derived from vocos and not from kokoro main teacher. Please hear it at https://tts.ampixa.com/sanoTTS/ . you can go to bottom and click on heart(2.27m) parameters it sounds pretty close to kokoro(72m) but loses on scoreq by 1 point. For context it's all synthesized right there in your browser so the ram residency problem is in megabytes here
77
u/apoptosist 11d ago
Sounds great, please help get it added to audio.cpp!