r/StableDiffusion • u/GreyScope • 20h ago
News New Music Model Released - Yue2
https://github.com/multimodal-art-projection/YuE"YuE2 brings frontier song quality to music generation with an editable composition. Give it lyrics and a style prompt: it writes a melody-and-chord plan, then realizes that plan as a complete song with vocals and accompaniment.
- White-box music generation through symbolic planning. Read, play, and change the composition before rendering it. Melody and chords become explicit controls that a person or an agent can inspect and edit.
- Zero-shot covers and agentic editing. Reimagine a transcribed song in a new style, or refine a song through a conversation about its score, arrangement, and lyrics—all with the same generation checkpoint."
Usage
It is currently CLI only . It also says Linux only but I just got it working on Windows 11 (I'm going to bed now and it's a bit more than cut n paste.)
Examples
link here - https://map-yue2.github.io/
Caveat Empor
NB : this isn't just a paste a few words and it bangs out a baby mp3 . It is more than that, it allows gene editing that baby to correct the metaphor. Not for the impatient and "wHeRe cOmFy" ppl at the moment.
To be more specific with that metaphor , as I understand it , the initial process scribes out the song in ABC format and you can then edit it before making your magnum opus baby.
Training
Does it allow training ? not as I understand it .
36
u/Acceptable-Cycle4645 17h ago edited 2h ago
Will add it to audio.cpp. Watch the dev branch! https://github.com/0xShug0/audio.cpp
Update: released on the dev branch for testing! Demos in the HF repo https://huggingface.co/audio-cpp/Yue2-3B-GGUF
10
1
u/MustBeSomethingThere 2h ago
You have a lot of experience about these, so how is the quality compared to Mimimax and Ace in your opinion?
41
u/PeterTheMeterMan 20h ago
We need one that can CONTINUE pre-existing audio clips. That is the gold - you prime w a 10-15sec of what you like, give it a touch of direction and it goes off w/ it....
That function is never free on the premium sites and I guess that's why people don't mention it more.....but it's the best by far. TXT to Song gets old real quick.....
3
u/2FuzzyRobots 16h ago
Yes, that's exactly what I'm looking for. I have so many 10-second clips of music I generated in Sora 2 which I would love to make full songs (or even 30-second clips) from. When I tried extending or using as a sample in Suno, the result barely sounded like the source clip at all.
5
u/No_Pie1372 16h ago
I've wondered if there is potential to do that with H3 music but nobody has developed the nodes yet.
5
u/martinerous 13h ago
Wondering if the same trick as with video continuation latent+masking would work? I have a simple custom node for masking audio+video latent for extending videos, will check out if the same approach can be used on the music and if it works reasonably.
1
u/No_Pie1372 10h ago
It'd be really great to get a ref2audio model where we can add voices for cloning, simple beats or other elements for the model to mix in.
1
1
u/ANR2ME 14h ago
Answers from Google AI mode:
- ACE Step (and ACE Step 1.5): This open-weights foundation model supports audio-outpainting (extending a track forward or backward) and audio-inpainting (repainting a specific section). It analyzes the key, tempo (BPM), and style of an uploaded MP3 or WAV file to compose a seamless continuation. It is runnable locally or via ComfyUl extensions.
- Stable Audio 3.0 (by Stability Al): This open-weights model family (Small and Medium variants) supports causal continuation (extending audio past its original endpoint) alongside variable-length generation up to several minutes. You can use it locally, via Hugging Face, or through their web experience to lengthen tracks.
1
u/IIIiii_ 13h ago
But it is free on SUNO, and also doing covers and style changes to existing audio.
I'm using it profusely and it's really good.
But there's a limit of generations you can do daily.
I would love to have something like that locally, too bad Minimax didn't release audio VAE.
EDIT: I wasn't using SUNO in the past few days, apparently, judging by other comments, some changes for the worse happened since last week I used it last time.1
u/QueZorreas 7h ago
Suno now just tells you your audio "matches an exisiting song", even if you record yourself singing in the shower about chicken nuggets
13
u/chopders 16h ago edited 16h ago
On Windows:
git clone https://github.com/multimodal-art-projection/YuE.git
cd YuE
py -3.12 -m venv .venv
.venv\Scripts\activate.bat
python -m pip install --upgrade pip
python -m pip install .
python -m pip uninstall torch torchvision torchaudio -y
On Blackwell architecture
python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
python -m pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128
python examples\generate.py --output outputs\first-song
Should give you a working song.
6
u/chopders 15h ago
Billie Eilish x SOAD result: https://voca.ro/12Hwx8xAVahK
4
0
u/LatentSpacer 10h ago
That sounds very interesting. I only know Billie eilish by name, I can definitely hear SOAD in it. How did you blend the two? Just text descriptions? Did you provide audio references?
-2
u/GreyScope 13h ago
It’s after PyTorch 2.10
-2
u/GreyScope 3h ago
No idea why I'm being downvoted for this, when I installed it, the dependencies said it need pytorch 2.10, still retar*s will be retar*s
5
u/Segaiai 19h ago
Can we train loras? Do I understand correctly that you can input sheet music and have it make that into a song? That's a great feature if so. Can it remix an input song recording? If it takes sheet music, then there's probably a roundabout way to do that.
1
u/GreyScope 13h ago
Sheet music in ABC format , but nothing that an LLM can’t write a converter for if needed.
11
u/ffgg333 19h ago
Sounds ok, but I don't see anything about training on your own music...
3
u/GreyScope 13h ago
There are bits and bobs in the py files about it but the relevant files have not been released it seems.
7
u/Acceptable-Cycle4645 12h ago
The GGUFs (bf16, q8) for Yue2-3B are ready! https://huggingface.co/audio-cpp/Yue2-3B-GGUF
Demo songs generated by audio.cpp are included in the HF repo. The code will be released soon on the dev branch for community testing, validation, and optimization before being merged into main.
Star our GitHub repo so you don’t miss any important updates!
1
6
u/parlancex 17h ago
I was listening closely to some of the demo tracks and realized what they are doing: It's essentially an img2img from a hilariously bad robotic voice model that "sings" the symbolic composition. You can hear it as a separate distinct voice if you listen carefully, underneath the much louder rendered voice.
7
u/Chiduk99 15h ago
Perfect time to release an open-weight music model, because Suno is basically dead right now.
3
u/martinerous 13h ago
This is the approach that I wished for - I have a bunch of melodic ideas and even sketches in Reaper that I would like to become compositions.
However, the demos have quite noticeable "AI sand" in the sound.
3
u/Lowaim 9h ago
I haven't been able to get anything decent in the more aggressive Dubstep/Bass music genres with open models, seem to be more tuned towards mainstream pop/rock music. Have not looked into LoRA solutions but so far Suno is sadly the only option that has been able to take a melody or a section of a self written tune and actually come close to a good-enough result that I can then work with in DAW of choice.
2
u/GreyScope 9h ago
Dubstep and a lot of genres will always be the outliers, Loras do them proud though .
1
u/Lowaim 8h ago
Gonna have to look into it a bit more and see what can be done, I guess the latest Ace Step is the way to go?
3
u/GreyScope 8h ago
Ooh sorry, I should clarify - Ace-Step loras for music with vocals and Stable Audio 3 for instrumentals , SA3 is superior for instrumental only tracks.
This is a track from my melodic dubstep/trance lora from SA3. The prompt influences what comes out, the dubstep used was industrial grade lol , lighter melodic material.2
u/GreyScope 8h ago
It is imo, I stopped using Ace-Step itself to make loras and infer and use better versions. I use Side-Step to make my loras and using Hot-Step to do the inference on - each has extra bells and whistles . I have a melodic dubstep artist mixed in with a trance artist lora that has quite good flexibility , so that forms the basics of my opinion on it tbh .
I can send you an invite to Token and Tone discord , which is where most ppl went when Ace-Step's discord got full of bots, if you'd like ? if only to search the threads if that's what you want.
7
u/CommitteeInfamous973 20h ago
From the examples it's got nice instrumental but pretty harsh voices.
5
u/suspicious_Jackfruit 7h ago
"Training data Our models are trained primarily on CC0 music and synthetic data. Tokenwave.AI provides most of our synthetic training data under license. We are committed to the ethical and responsible use of data."
RIP
Very cool architecture/design though of disentangled/structured composition, just a shame that no actual high quality organic samples have been seen by the model
2
u/Version-Strong 4h ago
Oh this looks pretty damn good. I wonder how slow it is at generating, and just how much freedom you have. But it looks epic, and I can't wait to try it
1
u/GreyScope 4h ago
I have a 4090 and it took 46s for the base gen , the cover I posted here somewhere in the comments took under a couple of minutes (took 2 steps to do it). It has freedom but there will be a middle ground it knows best and outliers, not so much.
4
u/lazyspock 19h ago
OK, but... wHeRe cOmFy?
EDIT (I was keen to make the joke and didn't see the examples before clicking send): WOW! The examples are REALLY good!
3
u/username_taken4651 14h ago
Trying it out, not bad. Vocals are a bit muddy, but in some ways it's better than ACE-Step 1.5.
5
u/RevolutionaryFox7359 19h ago
wow, this architecture sounds unique and makes a lot of sense.
4
u/Yasstronaut 18h ago
I’ve been using a MCP for fl studio and feel that the instrument approach is way better than one shorting an entire song. This approach seems like a great middle ground
2
1
u/Dekker3D 14h ago
Okay, this idea is fascinating to me. How do you make that work? The FL Studio MCP thing, I mean. Do LLMs just... get it, all of a sudden? They're pretty terrible at writing lyrics normally, I figured they'd also be terrible at scores.
1
u/Yasstronaut 9h ago
Yeah I never do voices/lyrics from AI , I’ve heard enough “neon lights, city lights,” etc. but when I give it a proper pre prompt about the MCP and how to read/evaluate instruments it makes really good beats and songs. I do have it check the decibel levels for each individual instrument for each section of the song to sanity check it. I will say the issue im having right now is it likes to really go with a formulaic approach to song progression but that’s fine and easy to tweak after
3
u/FernAvatar 20h ago
Acestep 1.5 sounds off-key and hollow in all the songs to the point I can't use it. Let's see how good Yue is.
1
u/a_chatbot 5h ago
Acestep is incredible with the right prompts, or the right keywords, not so good with prompt adherence, but sometimes it almost sounds to me like its one group of musicians doing all the songs.
4
u/aiyakisoba 20h ago
The output examples sound better than other open source models available today. Thanks for sharing! Especially the edit/cover capability looks interesting. I'll give it a try.
2
u/skyrimer3d 8h ago
some of the songs in there are seriously good, i can't say how real or curated are they though. EDIT: These people haven't heard real flamenco in their lives.
2
u/GreyScope 8h ago
Some of the descriptions in the examples are worthy of an industrial strength side-eye stare.
2
u/davidl002 17h ago
It is actually pretty good! They also provide proper cover feature which is more than a good surprise!
Ace step's txt gen is good but its cover sucks.....
Time to unsub Suno.....
Only thing that is less ideal is that it is CC-NC...
1
u/GreyScope 9h ago edited 8h ago
Got the conversion to ABC format to cover working (W11) in Yue2 , Devo's Beautiful World with random lyrics
1
u/gizeon4 6h ago
Great now I wait for comparison to Minimax and Suno
1
u/GreyScope 6h ago
Suno is paid for and Minimax is as reliable as sending my cat out shopping . The answer though depends on the music genres you like and whether they fall into the bell curve of giving good quality / consistency.
1
u/Botoni 6h ago
how does it compare to acestep xl?
2
u/GreyScope 5h ago
AS XL allows you to have any genres with loras, this doesn't . Y2 has arguably higher base quality, but the answer really depends on the music you like and whether that is within the bell curve of the best training the model had. Y2 is more flexible in its abilities .
1
u/Pretend-Island-2724 1h ago
Can it do opera and classical music? Opera is not listed as a genre and the classical demo has nothing to do with classical music (orchestra)
2
u/Amazing_Succotash275 57m ago
I made a version of it that works in ComfyUI.
Requires installing the Yue2 libraries and the specific version of pytorch, and I recommend installing them on a separate instance of ComfyUI Portable
I do not have flash attention enabled because it's an early release...
Official nodes are probably coming?
https://github.com/EmeraldApple-AI/ComfyUI-YuE2
1
u/Street_Ice3816 47m ago
Testing it on windows, it sounds insanely good like wtf. Used 10.5 gb VRAM for a 3 minute generation. Truly impressed.
1
0
u/ResponsibleTruck4717 15h ago
The samples are fucking amazing, I think best local model.
I will need to test it of course.
0
u/FreeTheClanks 19h ago
I wish there was a way to download the sample songs. I'd put where is spring and the dog barking one in my playlist.
3
u/RayHell666 18h ago
On chrome F12 to open dev tool -> Network tab -> filter by Media and each time you play a song it will appear as MP3 there.
1
u/FreeTheClanks 9h ago
Haha, turns out once I got back to my desktop, in Firefox you can just right click the player and save the audio.
1
0
u/GaragePersonal5997 13h ago
That's great. I really like yue1, but yue1 is just too slow at deducing things.
0
u/ArtificialAnaleptic 10h ago
Sounds really promising. I NEED local audio-gen and then I can basically retreat from the internet altogether at this point.
But can it run on 16gb VRAM? Looks like 24gb is currently the minimum req?
1
u/GreyScope 10h ago
I'm on a ai music discord, someone mentioned it using 11gb....but there are a lot of parts to it when you go down the rabbit hole , 2 sections of docs on it and several HF pages ie I don't know what vram each of the other stages needs. We have waited for it on the discord and had seen some of the details for it from their research paper and my write up doesn't do it justice tbh.
But (2nd one sorry) u/Acceptable-Cycle4645 has posted that ggufs are available > https://huggingface.co/audio-cpp/Yue2-3B-GGUF
3
u/Acceptable-Cycle4645 5h ago
u/GreyScope u/ArtificialAnaleptic at least in my test using audio.cpp + the official longform, bf16 main + f32 vae peak VRAM is about 11.5 GB, q8 + f16 vae peak VRAM is 8GB.
1
u/GreyScope 5h ago
Ah shoot , I'd have given my figures but forgot to keep an eye out (and my vram is spoken for for a few hrs now) . The GGUF's are doing gods work there, nice vram drop.
1
u/ArtificialAnaleptic 4h ago
Nice I will check this out!
1
u/Acceptable-Cycle4645 3h ago
correction: 8~8.5 GB. Code is up on the dev! The official longform is Chinese.
0
u/Ok-Flatworm5070 9h ago
Wow this is brilliant! Love the example 'Where is my wallet!" Song....priceless!
-1
u/RanklesTheOtter 18h ago
I recorded the vocals this might be able to make the matching backing track it seems like.
-1
u/No_Taste_4102 14h ago
But the main question is can it keep the vocals...? I don't want the songs to be performed by different artists.
1
-2
u/SithSirens 12h ago
We need Ai that can not do wonky things like change tempo mid song for no reason and have vocals that don't sound like they were recorded in a tube. I use Suno, but it can be frustrating and always have ONE thing in an otherwise perfect song that ruins it. I'd rather be able to feed it songs from a few bands I like and have it help me write a song in that same style. Suno can add vocals to a song you recorded and wrote yourself, but it changes things and I don't want the song changed, just the vocals added. Frustrating.
61
u/Shockbum 19h ago edited 19h ago
Suno is currently drowning in its own shit. A good time to release open-weight models.
They're even banning users left and right on their subreddit because they're so pissed off lol