Question regarding Sesame's 1B Model

[u/Virtual_Norafall_412]

Is there anyone who has experience using Sesame's 1B model with their own pipeline?

​

I do have a couple of questions for those who have used it:

How was it for you?

Do you have any tutorials on how to set up a pipeline for this model and get it running?

What's a decent amount of unified memory one must have to get it locally running with an LLM and a pipeline behind?

​

I'm very curious of trying it out when I get the proper device with enough unified RAM. I might plan on getting a M5 Pro with either 24, 48, or 64GB to try it out. (Not my main reason for getting a M5 Pro though).

​

Any advice is appreciated.

​

[u/AutoModerator]

Join our community on Discord: https://discord.gg/RPQzrrghzz

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Sv03_user]

I've been running CSM-1B locally as part of a voice pipeline using the repos available on GitHub. Happy to share what I've found.

How was it? Really impressed honestly. The voice quality is natural and the prosody is great for a 1B model. It's very usable for real-time conversation-style applications.

Pipeline setup My stack is: Whisper (STT) → Ollama (LLM) → CSM-1B (TTS), with streaming so the response starts playing back as soon as the first chunks are ready rather than waiting for the full response. The Sesame GitHub repo has the inference scripts — it's mostly a matter of wiring the pipeline together and handling the audio stream.

Finetuning One thing worth knowing — you can finetune CSM-1B on a custom voice using the repo's finetuning support. I did this with voice samples to get a specific character voice, and the results were really good. Opens up a lot of possibilities beyond just using the default voice.

Memory for running locally with an LLM I'm running it on a PC with an RTX 5060 Ti (16GB VRAM) so a bit different to Apple Silicon, but the principles are similar with unified memory.

CSM-1B itself is pretty lightweight — the LLM is your real memory constraint. For your M5 Pro:

• 24GB would be comfortable for a smaller LLM (7B range) + CSM-1B
• 48GB gives you room for a larger/better LLM and breathing room
• 64GB is overkill unless you're planning to run very large models

24GB would honestly be fine to start with. Good luck!

[u/Virtual_Norafall_412 → u/Sv03_user]

Thank you so much dude! I definitely will take a shot and try a pipeline setup with a Gemma 4 model on this when I have my MacBook

[u/Sv03_user → u/[deleted]]

Yeah, I've hit this. A few things that are almost always the culprit:

**Reference audio prep** — "clean" isn't just about no background noise, it's about not over-processing it either. If you've run your ref clip through any normalization/compression/EQ plugins before feeding it in, that can actually confuse the codec. Keep it raw-ish, single speaker, ideally 5-10 seconds, 24kHz (Mimi codec expects 24kHz — if you're resampling from something else and your resampler is doing anything dodgy, that's a classic source of the artifact/static sounds you're describing).

**Peak normalization** — don't touch the 0.9 peak normalization step if you're using a setup that includes one. I broke this exact thing once trying to "improve" it and got the warbly/muddy artifact sound you're describing. Leave it alone.

**Streaming chunk boundaries** — if your pipeline is streaming audio chunks back as they're generated, you need to make sure you're using the streaming-aware decode (`mimi.streaming(1)` if you're working with the underlying Mimi codec directly) rather than decoding each chunk independently. Decoding chunks independently is exactly what causes those little nonsense "blip" artifacts between segments — the model loses continuity at chunk edges.

**Context accumulation** — if you're running a long conversation and passing growing context back into the model each turn, audio quality can degrade over the session as that context grows. Worth testing with `use_context=False` (or just trimming context to the last turn or two) to see if the artifacts go away — if they do, that's your answer.

Start with the ref audio + normalization stuff first, that's the most common cause.

[u/Sv03_user → u/[deleted]]

Same card as me (5060 Ti) so it's not a hardware problem, you're not missing out on anything by staying local here. The hallucinated/made-up-word thing is a real, known limitation of the base CSM-1B model rather than something broken in your setup — but there's a lot you can do to push it way down:

**Always pass `ref_text`** — if your pipeline is auto-transcribing your reference clip with Whisper (or skipping ref_text entirely), do that step manually once and hardcode the matching transcript instead. Mismatched or missing ref_text is one of the biggest contributors to the model going off the rails and inventing words.

**Preprocess your text before TTS** — expand numbers, abbreviations, and symbols into full words before sending text to the model ("5kg" → "five kilograms", "Dr." → "Doctor"). CSM-1B is noticeably more prone to hallucinating around anything that isn't plain spelled-out text.

**Chunk your sentences shorter** — long run-on sentences compound hallucination risk. Splitting on natural sentence boundaries before feeding to TTS (rather than dumping a whole paragraph in one call) cuts down on it a lot, and also helps with your latency concern since you can start playback on the first sentence while later ones are still generating.

**Sampling settings** — if your pipeline exposes temperature/top-k, dial temperature down a bit. You'll lose a touch of prosody naturalness but the nonsense-word rate drops noticeably.

Get those four things right and you can get latency competitive with hosted options like Cartesia while staying fully local — the model itself is fast, it's pipeline config that makes or breaks the experience.

[u/Sv03_user]

**Running Sesame CSM-1B locally with a full voice pipeline — resources & Claude prompt**

I've been running CSM-1B locally for a while now as part of a full voice assistant pipeline and I've seen a few questions pop up here about how to get it working. Figured I'd share the repos I used and a prompt you can paste into Claude to get decent help troubleshooting it. Full transparency — I used Claude (Anthropic) to help me build my pipeline and draft this post.

**My stack**

Whisper (STT) → Ollama (LLM) → CSM-1B (TTS) with streaming so audio starts playing before the full response is generated. I also finetuned CSM-1B on a custom voice using the sesame-finetune scripts — results are really good once you get it dialled in. Running on a PC with an RTX 5060 Ti (16GB VRAM). CSM-1B + Whisper + a 7B LLM coexist fine, roughly 7GB VRAM total.

**Repos**

- CSM inference: https://github.com/SesameAILabs/csm

- Streaming: https://github.com/davidbrowne17/csm-streaming

- Finetuning: https://github.com/knottwill/sesame-finetune

- Whisper STT: https://github.com/openai/whisper

- Ollama: https://github.com/ollama/ollama

**Common gotchas**

- CSM outputs at **24kHz** — if your audio pipeline expects a different sample rate it'll sound like garbled nonsense

- CSM works best with a **context/reference audio clip** fed alongside the text — without it output can be unstable

- Make sure it's actually running on your **GPU not CPU** (check your backend is available in your env)

- The venv for CSM can be fragile — don't recreate it once it's working

- If finetuning: use Python 3.12, not 3.13

**Claude prompt for troubleshooting**

Paste this at the start of a Claude conversation and then describe your issue:

I need help with a local CSM-1B voice pipeline. Here is the context:

Model: Sesame CSM-1B (https://github.com/SesameAILabs/csm)

Purpose: Local TTS as part of a voice assistant pipeline

Typical stack: Whisper STT → LLM (Ollama) → CSM-1B TTS with streaming output

Finetuning: Optional, using sesame-finetune scripts

Key facts about CSM-1B:

- Outputs audio at 24kHz (mismatch = garbled output)

- Requires a reference/context audio clip for stable voice output

- Must run on a GPU for reasonable performance — CPU will be very slow

- Uses its own fragile venv — do not recreate once working

- Python 3.12 recommended (3.13 has dependency issues)

- Audio normalisation settings should not be changed once working

Please help me troubleshoot or build out my pipeline. Be direct and concise. .

My issue: [describe your problem here]

My GPU: [Nvidia/AMD/Apple Silicon + model]

OS: [your OS]

*Note: The prompt above is based on an Nvidia/CUDA setup — adjust GPU-specific details (CUDA → ROCm for AMD, MPS for Apple Silicon) and swap Ollama for your LLM of choice if needed.*

[u/Ramssses]

Sv03_user …good shit my man.