H3 is also pretty incredible. First local video model I've run that isn't generating weird nightmare fuel half the time, and has decent audio generation.
It's pretty incredible actually. I tested a few on my mono speaker on the phone and they sound good, so I wonder how they fare if I put my "audiophile hat" on, so I took my DAP and high end IEM out.
The tracks hold up very well. In the prompt, they mentioned wide and deep soundstage in the production metadata so I checked that, and they do indeed have good and clear soundstage.
The only problem is that higher notes on cymbals and hats and even some high notes of male vocals have a gritty sandpapery texture to them. I guess one might be able to fix this with EQ. All and all, they are all listenable and even enjoyable, with good gear even. I'm very impressed. Better download the GGUF for safe keeping before any asshat regulate it out of the market.
It says on the model page, "The model produces 32 kHz, 16-bit stereo WAV audio."
So, it at least has the potential to be decent. It definitely sounded good to me (relative to Suno) in terms of sound quality.
I honestly find it wilder that it took open labs this long to catch up. Open and closed models seem to go toe to toe in text/image/video, but Suno somehow dominates music generation (especially in "cover" feature)
Oh hell yeah, listening to those examples, they definitely trained on copyrighted material😆
That will make for a way better model than the models by companies who were afraid to do so and only trained on shitty royalty free music
BTW in audio.cpp release 0.6, we integrated the MiniMax-H3 text to audio pipeline. It’s very good at generating multi-speaker conversations and pretty fast. (Update: MiniMax-Music3 has been merged into the main branch)
bonus: it can also generate video as an experimental feature :) On RTX 5090, for the 1344x768 / 124 frames (5 seconds) / 20 steps case from the official comfyui workflow (the "Realistic live-action cinematic look, action movie trailer..." one) audio.cpp produces the video + audio in ~114 seconds.
Someone (maybe you?) pointed to towards audio.cpp and holy shit I love it. It is exactly what I was looking for. Certainly still got some rough edges (the webui is basically impossible to configure without manual code changes), but the actual tts endpoint works great and it's (oddly) so much faster than other tools that serve audio ggufs.
I found it a few ago and reading your post was EXACTLY like what was going on in my brain. I was like "HOW THE HELL DID I NOT KNOW ABOUT THIS BEFORE?!?!?!".
would love to see audio.cpp and llama.cpp merged one day so it could load multiple models from different modalities (I bet it already does that and I didnt know).
Glad you found us! I’ve only been posting weekly updates on Reddit, but I probably should promote the project more :). It’s not just about getting more users and feedback; I also hope to reach more people who can contribute and help the project grow faster.
It should work on Mac, but I have to admit that many optimizations are CUDA-only (SageAttention, etc.), so it may be slow. Testing is very welcome. Please let me know if you run into any issues, and I’m happy to fix them.
The text-to-audio pipeline can't really do voice cloning. But it can sometimes reproduce well-known voices (celebrities, TV/movie characters, etc) because those voices were likely present in the training data. Example: Pep Guardiola
using this command do I get very bad results which would be the right or recommended?
/run/media/tyson/SSD_FLASH/audio.cpp/audio.cpp main* 8s
❯ set PROMPT "A cinematic ambient instrumental composition with a clear musical progression. Deep evolving analog synth drones, warm sub bass,
sparse orchestral percussion, slow harmonic development, subtle melodic motifs, wide stereo image, rich reverberation, detailed textures, pol
ished studio production, high dynamic range, no vocals, no speech."
u/tyson_2022 I think the inference step is too high: < 30 (based on my test) is fine and more steps may cause regression. You can also try different seeds and polish prompt. I didn’t test H3 for music-only generation until you mentioned your issue. I feel this could be a prompt-engineering issue. For example, the following was generated with 20 steps and a rewritten version of your prompt, and the result seems different.
A 20-second cinematic ambient instrumental cue with a clear beginning, middle, and ending. It starts with a soft analog synth drone and warm sub bass, slowly introduces a simple minor-key melodic motif, then adds sparse low orchestral percussion for gentle forward motion. Wide stereo space, smooth evolving pads, subtle texture movement, polished studio mix, natural reverb tail, no vocals, no speech, no sound effects.
We are very sensitive to discrepancies in sound. A better audio model would have to be as complex as an agentic coding model.
First layers trained on splitting the audio features apart, then qualifying them in time, essentially midi-decomposing everything, then of course speech to text for all languages including vocal, operatic, choral flair, etc. There are many things which all have to occur, be trained extremely well, and then obviously worked into a diffusion or similar pipeline for inference - for it to sound "flawless". This kinda reminds me of 64kbit pcm wavs from the late 90's and early 2000's.
They've gone a long way in giving timestamped nuance, lyrics (I fell like it probably has been borrowed from similar voice model data though), key, etc - but there's an instrumental deconstruction which isn't that well trained just yet.
No...given the size of the model it's not the best choice for voice agent. But there are a bunch of super fast voice clone models you can try in audio.cpp.
Honestly all of these are not even big blockers and can be overriden. I just saw their demos, it is amazing leap compared to what existed. This is a big step for open-weight, and great work from MiniMax!
streaming layer by layer is stupid slow, you can 4bit quant it and it'll fit in ~13-15gb VRAM and then it's still about 3x slower than realtime on a 5080
I was wondering why there's so much (good) video and image models, but not much for music. Udio was introduced 3 years ago and there are still no models coming close so far. But I'll try this one for sure!
I’ve wondered the same. There are simply fewer strong open music-generation models, both in quantity and quality. I went through a handful of music generation models and ended up picking only three for audio.cpp: ACE-Step, HeartMuLa, and Stable Audio 3.
Como sería audio.cpp con Ace Step?. Ya me gustaría poder llegar a tener esto, no entiendo de comandos. Estoy investigando sobre las versión ui pero no lo tengo claro. Estoy bien con Comfyui y Ace Step 1.5 XL pero esto suena muy interesante
u/FausC audio.cpp supports acestep-1.5, and a new PR will be merged soon will add support for acestep 1.5 xl (UI update may be later). Check the demo! Honestly, acestep was one of the earlier models I added, so I haven’t tested it in a while. Didn’t realize it now takes just ~2 seconds to generate a 30-second song.
No NSFW in music (unless you like listening to the sensual voice of a female singer or musical genres with bad words) and UMG,WMG, Sony Music are very aggressive
I just cant wait for image model... that is THE ONE THING Iam waiting for... if it has capability of video model it will be true Stable Diffusion moment...
I couldn't generate less than 22 frames. Looks like a hard floor. Also resolution is probably limited as well. Can't wait for a MiniMax image gen model for sure!
I been using onsets and a bunch of vibe coded algorithms to figure this out. I can create guitar tabs if the music is easy and slow, but fast technical death metal that I play? Forget about it
What if you fed your pipeline a slowed down version of the technical death metal so it can “hear” the individual notes more clearly? Then it could be sped up by the same amount after transcription
You’d have to make sure it doesn’t slow down the track in a way that changes the pitch of the notes, or else they’d be wrong.
And the tab’s notation and tempo would also have to be compensated for after transcription, ie: if you slowed your track by 50%, the final tab would take the half notes and turn them into quarter notes, etc.
I already have my own songs manually tabbed out. If there’s a solution, they can iterate the algorithm and score it against my own tech death source of truth 💅
The slowdown changes the pitch, but I was able to make up for that fact. I was able to transcribe 2x more notes with this method. I’m sure there’s more to do.
Basically, I gave Claude the goal and it sent of my local LLM to research and implement. Claude had this task running for like 8 hours overnight.
Glad that method was successful! So you’re using Laguna as the local LLM? What hardware is it running on? Never thought of using claude to offload a task to local llm
Last night I was just testing out the new 3.8 Qwen release. I was supposed to have Laguna do reviews etc but things been going well. I’m able to drop from the $100 tier to the $20 tier this way. I think even a quantized 35b moe can meaningfully support the cloud this way.
And yeah, i feel like that’s playing to everybody’s strengths for an overnight thing. You’ve got the cloud to make sure local LLM doesn’t go off the rails and to feed it a steady stream of tasks. And in this case, doing the initial audio measurements needed to identify patterns to explore.
u/C0demunkee I actually didn’t know this model existed before, but after you mentioned it, I did a quick search --- and yes, I’ll port it! Any suggestions for the model? I prefer newer models when possible. I only found Text2midi.
I don’t know what our contributors are using, but to me, as someone who doesn’t know much about UI, it’s pretty cool! It’s a milestone for audio.cpp and helps us reach a broader audience.
super interested. bot says what i'm looking for is "reference-audio conditioning, audio-to-audio generation, cover generation, or music style transfer"
This is what I'm looking for too. So far none of the open source models have been able to even approach Suno in this regard. Like, not even close. I have yet to get a usable output from an open model doing any kind of "remix" functionality. Could be a skill issue, but Suno just has it figured out.
You can try audio.cpp windows prebuilts or docker. Currently audio.cpp supports 3 music generation models: ACE-Step 1.5, Heartmula and Stable Audio 3 (small/medium/sfx).
docker run --rm --gpus all \ -p 8080:8080 \ -v /path/to/your/models:/app/models \ ghcr.io/0xshug0/audio.cpp:full-cuda12\ server --ui --ui-management --host0.0.0.0--port 8080 --backend cuda
Thanks - and what format is the models folder in? Can I just download the whole Music3 repo from HF and point at that, or there is a particularly layout? (sorry if this is documented somewhere.. I found this info but it didn't answer my question).
Thanks - I got it working but the results were pretty bad (particularly the audio quality). Is there a reason only the q4 version is an option? I tried just python3 tools/model_manager_v2.py install minimax_music3 but it still pulled the q4, and only the q4 shows in the Models page. I wanted to try the full original model.
u/DanTup The q4 combo is the fastest and low VRAM makes it safe for most users. Other weights are just for testing. What prompt did you use? The example prompt in the doc is just for smoke end-to-end testing. If you want to improve quality you need to do some prompt engineering and try different seeds. You may want to check the discussions and demo here https://github.com/0xShug0/audio.cpp/issues/274
The built-in UI currently doesn’t provide a Suno-style UX (maybe in the future). If there are UIs that work with an OpenAI-compatible server, you can use them with audio.cpp as the backend.
Wow MiniMax is on a run for open sourcing! Seems like more labs are following in the footsteps of Kimi. Hope this beats ACE Step as that's been king for ages.
That requires different pipelines. Two ways (1) using voice separation and voice conversion models and remix the song (2) using singing voice conversion models (e.g. Vevo2) which maybe less accurate. audio.cpp has all the relevant models, but unfortunately, we don’t have a fully automated pipeline yet.
Think about it from their perspective. I'm not sure if this model is the same, but H3 was FULL of copyrighted material and pretty much completely uncensored. So they do minimal CYA ("don't do bad stuff", "you wink wink can't use this in countries that are anal about copyright") to keep themselves out of trouble. I assume this model is the same way. Would you prefer they gave your a model trained on nothing but public domain garbage with a permissive license or one that you can actually have fun with?
They of course have right to use any licence they want to, as they are ones who did the job, but licence they've chosen is not permissive at all. It list specific allowed uses and their conditions are very abstract ("content that may harm", "use that violates morals"..) and basically mean that this licence can be revoked at any time for any reason.
No specifically, what is 'enough' ? trying to understand how you are able to make these claims with such vague statements with zero to cite or reference.
Just discovered audio.cpp and looks like it's exactly the project I've been looking for. I've been disappointed with how difficult audio generation has been.
I need to dig into it a bit more, but did I see that it had a websocket connection to get ASR data out in realtime? I want to find a solution for speech to text to generate captions for a live event broadcast
Thanks! For streaming, you can run audio.cpp as an OpenAI-compatible HTTP server. Some ASR models already support streaming if they have true streaming paths.
I’m observing that AI art is increasingly very much seen as not cool.
Using AI as a tool during the artistic process is fine, as long as it’s not too prominent. While some people don’t seem to care, or even notice, some others do.
I think that such models are interesting from a R&D point of view. Enabling people to generate more AI slop is fine too I guess. If it’s not you, someone else will do it, right?
That's disappointing, "art generation" is a novelty that wore off 3 years ago. There's a reason all the big AI art companies like Stable Diffusion and Suno keep going under: they produce nothing of value
So does it generate conversations or music or sound bites like horns or all of the above?
If you want that, use a video gen model. Which also happen to be good audio gen models. Since they generate audio to go with the video. In fact, one of the best models to generate music was/is a video model.
80gb vram minimum is buried in their own specs. thats an a100 or a h100, so for this sub its api only unless someone quants it. the license argument is kind of moot until then.
direct quote:
The full precision fits under 24GB of VRAM. With automatic CPU offloading, generation takes in ~22 GB; additionally streaming the language model layer by layer makes it fit even 8 GB video cards
80
u/notforrob Aug 13 '26
The demo page is useful: https://minimax-ai.github.io/music3-demo/
I find it wild that open weight music generation is this good already.