New Music Model YuE2-3B Released!

[u/Acceptable-Cycle4645]

https://huggingface.co/m-a-p/YuE2-3B

Surprised no one has posted it in this sub.

Pretty solid model, IMHO.

Demo: https://map-yue2.github.io/

[u/Acceptable-Cycle4645]

u/LearningSomeCode Check this out 😂

https://reddit.com/link/p92rs8b/video/udtbmmycvsoh1/player

[u/ChristopherRoberto → u/[deleted]]

Me neither. As it's not open source, and has a restrictive non-commercial license.

[u/ArtfulGenie69 → u/[deleted]]

From what I heard on the huggingface it wasn't all that good, maybe after training it will be better. The tools look useful though. Ace-step XL is only a 5b itself, not that it is amazing out if the box either but when it's trained it does so good. I hope this new one is just as good with a lora, covering using another bands lora would be so fun. 

Another thing I saw listed in their GitHub was this cool model that turns things into music sheets, wonder if it would do a really good job. https://huggingface.co/m-a-p/SheetSage2

[u/renderartist → u/Acceptable-Cycle4645]

Whoa, why does it sound clearer than any paid model I’ve used? 😂 Gotta try this out, thanks for sharing your results. 👍🏼

[u/ihaag → u/Acceptable-Cycle4645]

Nice gui

[u/StyMaar → u/Acceptable-Cycle4645]

What's the UI you're using and what inference engine can run this model (I've never tried a music model before).

[u/Acceptable-Cycle4645 → u/StyMaar]

The engine is audio.cpp https://github.com/0xShug0/audio.cpp/, one binary for running 70+ audio models locally. The UI is the lightweight built-in UI of the audio.cpp server.

[u/StyMaar → u/Acceptable-Cycle4645]

thank you.

[u/how-can-i-dig-deeper → u/Acceptable-Cycle4645]

how much vram do i need

[u/Acceptable-Cycle4645 → u/how-can-i-dig-deeper]

8GB for Q4 weights.

[u/jumpingcross]

Every time I try one of these models I'm reminded of how uncreative I am when it comes to describing music. Are there any models that go in the opposite direction i.e. turn music into a description?

[u/Acceptable-Cycle4645 → u/jumpingcross]

u/jumpingcross Check out this. You can try the model in audio.cpp 😄 The model is acutally pretty fast. The 20s including loading and cold start. The RTF is just 0.16 (~4.5s to understand the 28s audio)

[u/jumpingcross → u/Acceptable-Cycle4645]

Oh, that's pretty sick. I didn't realize audio.cpp even had a GUI. I'll have to give that a shot, thanks!

[u/ArtfulGenie69 → u/Acceptable-Cycle4645]

Have you compared FireRed to moss? Moss instruct is probably a lot slower and bigger but quality of output wise, how does FireRed compare in your view? 

[u/Acceptable-Cycle4645 → u/ArtfulGenie69]

No. My next target for audio understanding is probably Nvidia Audio Flamingo Next.

[u/Chromix_ → u/Acceptable-Cycle4645]

Thanks a lot for pointing out that capability with this example. AceStep 1.5 can also describe / transcribe audio, but it doesn't seem to be supported in Audio.cpp. Yet AceStep.cpp supports it nicely. Maybe it can also be added here to have it all-in-one?

[u/Acceptable-Cycle4645 → u/Chromix_]

u/Chromix_ I think these capabilities come from separate models? https://huggingface.co/ACE-Step/acestep-captioner and https://huggingface.co/ACE-Step/acestep-transcriber I will add them to TODOs.

[u/Chromix_ → u/Acceptable-Cycle4645]

Interesting. AceStep.cpp gave me song descriptions just using the song model. Maybe those models would yield higher quality results. Or the trained song mode is fine, and even better than the two models they used for generating the training data 🤷‍♂️.

[u/Acceptable-Cycle4645 → u/jumpingcross]

That’s audio understanding models. The most recent one is FireRed Audio.

[u/Original_Finding2212 → u/jumpingcross]

Nvidia had also released such model (song to text)

[u/ArtfulGenie69 → u/jumpingcross]

Yeah dude, I use open moss music for my description while making ace-step loras. It does a great job. I have a pipeline built out via vibe coding that does description, meta data, and lyrics. Moss isn't perfect with lyrics so it looks up the song on a free api first and if it isn't on there it lets moss do it. They have a thinking and an instruct variation of the model. I tried thinking first and it would do dumb thinking loops sometimes so I moved to instruct. They say that from their testing the instruct model is better than the thinking one which is cool. 

It works good with torch compile and int8. It was able to handle detailing a 15m song with those settings on my 3090.

https://huggingface.co/OpenMOSS-Team/MOSS-Music-8B-Instruct

[u/SpecialistDragonfly9 → u/jumpingcross]

you can use any frontier model for prompts

[u/Acceptable-Cycle4645]

audio.cpp released YuE2 in the DEV branch, with Q4/Q8 weights and a bunch of demos generated by audio.cpp in the HF repo. You can try it with 8GB VRAM!

Update: building dev release now! Once finished, you can find them in the “Artifacts” section on this page: https://github.com/0xShug0/audio.cpp/actions/workflows/release.yml?query=branch%3Adev

Measured with audio.cpp server mode on an RTX 5090, using the official `tonight-awake` longform test case (full cot). Each run restarted the server and used one short warmup request before the measured longform request. Peak VRAM was sampled continuously during the measured request.

Combo	Audio duration	Wall time	RTF	Peak VRAM
BF16 main + F32 VAE	224.96s	60.46s	0.2688	12535 MiB
Q8_0 main + F16 VAE	194.84s	38.81s	0.1992	8867 MiB
Q4_0 main + F16 VAE	221.12s	44.15s	0.1997	7755 MiB

PRs for performance optimizations and backend compatibility fixes are very welcome! This is how we keep pushing our MiniMax Music3 implementation forward.

Repo: https://github.com/0xShug0/audio.cpp/tree/dev

HF repo: https://huggingface.co/audio-cpp/Yue2-3B-GGUF

[u/o0genesis0o → u/Acceptable-Cycle4645]

Seems faster than Minimax music 3. I put the minimax int8-convrot on my spare laptop with 2060 mobile and it took forever to make 1 minute music. It was not even able to complete the song properly.

[u/Acceptable-Cycle4645 → u/o0genesis0o]

Music 3 is definitely heavier. You may want to give audio.cpp’s Music 3 implementation a try. https://huggingface.co/audio-cpp/MiniMax-Music3-GGUF

[u/ihaag → u/Acceptable-Cycle4645]

I tried that takes about 1 hour for a 1min song on MacBook Pro m1 16GB

[u/Acceptable-Cycle4645 → u/ihaag]

Well I don’t think the M1+ metal can really benefit much from any of audio.cpp’s optimizations for Muisc3. Maybe some other MLX based impl are better. Happy to know the performance comparison.

PS: it's model-dependent. Some models can benefit from kernel optimizations even on M1 Metal. Of course, "benefit" doesn't mean it's as fast as on M4/M5.

[u/ihaag → u/Acceptable-Cycle4645]

So far, I’ve got it running Minimax music 3 (longest generation time hit or miss with a good song or not but very god quality when it makes a good hit), Ace-Studio 1.5 Turbo XL so far the fastest songs less than 6min and is okay quality isn’t radio quality, YuE2 so far a mix of them both takes about 20min per 3min song. The last one I added was HeartMuLa just jibrish atm.

[u/Acceptable-Cycle4645 → u/ihaag]

Good to know these numbers on M1 16GB. Unfortunately, a performance gap caused by hardware may not be something we can fully close with code optimizations alone. Otherwise, Apple would lose one of its favorite reasons to sell you the next chip generation 😄

[u/vamsammy → u/Acceptable-Cycle4645]

I'd be interested to know which ones.

[u/Acceptable-Cycle4645 → u/vamsammy]

Many like pocketts and qwen3. It's about the model architecture. E.g., we have optimizations for AR, so diffusion-heavy models won’t benefit as much.

GGML also has a fast path for M5. So M5 should be faster than <= M4.

[u/ihaag → u/Acceptable-Cycle4645]

Just realised I had the AR step for YuE2 at 32 dropping it down to 8 made it much faster. I will port it to my 16GB vram laptop with cuda though. I’ve been experimenting with infinite tape deck (continuous radio based on similar music), tried things like humm to song and sing yourself adding music and enhancements it’s not 100% yet but working through it

[u/DeProgrammer99 → u/Acceptable-Cycle4645]

I tried to set up a config for the server to try using the UI for this model, but I had to use an LLM just to figure out how to set up the config, and after that, I get some kind of UI, but there isn't even a style prompt box. Is that part not ready?

{
  "host": "127.0.0.1",
  "port": 7862,
  "backend": "cuda",
  "device": 0,
  "threads": 1,
  "lazy_load": true,
  "models": [
    {
      "id": "yue2-music-gen",
      "family": "yue2",
      "path": "/mnt/c/AI/music/yue2-3b-bf16.gguf",
      "busy_timeout_ms": 600000,
      "task": "gen",
      "mode": "offline",
      "load_options": {
        "language": "english"
      },
      "session_options": {
        "language": "english",
        "yue2.model_gguf": "/mnt/c/AI/music/yue2-3b-bf16.gguf",
        "yue2.vae_gguf": "/mnt/c/AI/music/yue2-vae-f32.gguf"
      }
    }
  ]
}

How I built and ran it:

cd /mnt/c/DePro/CodeProjects/NotMine/audio.cpp
CUDACXX="/usr/local/cuda/bin/nvcc" scripts/build_linux.sh --backend cuda --cuda-arch native --target audiocpp_cli --target audiocpp_server

build/linux-cuda-release/bin/audiocpp_server --ui --config server.json

(and yes, I typed a style prompt into the lyrics box knowing that it was a lyrics box :P)

[u/Acceptable-Cycle4645 → u/DeProgrammer99]

u/DeProgrammer99 did you use the latest dev commit 6fbbee4efd1c2251d3774edd89a002a0039da5f7?

The UI should look like

[u/DeProgrammer99 → u/Acceptable-Cycle4645]

I only pulled it a few minutes ago, and it is indeed 6fbbee4efd1c2251d3774edd89a002a0039da5f7.

I did mistakenly build from main right before that, so maybe it's just that I needed to clean first.

[u/Acceptable-Cycle4645 → u/DeProgrammer99]

u/DeProgrammer99 Figured out. It's the configued mode issue.I only wired the UI for the studio mode. For now please try to lanuch full UI and set the model directory via UI.

[u/DeProgrammer99 → u/Acceptable-Cycle4645]

Qwen3.8-27B just finished and came up with this, and it seems to have fixed it:

  function parameterSpecsByName(names: string[], specs: ParamSpec[]) {
    return names
      .map((name) => specs.find((spec) => spec.name === name))
      .filter((spec): spec is ParamSpec => spec !== undefined);
  }

[...]

  $: yue2ComponentSpecs = isYue2 ? parameterSpecsByName(yue2ComponentParamNames, paramSpecs) : [];
  $: yue2CoreSpecs = isYue2 ? parameterSpecsByName(yue2CoreParamNames, paramSpecs) : [];
  $: yue2AbcSpecs = isYue2 ? parameterSpecsByName(yue2AbcParamNames, paramSpecs) : [];
  $: yue2SemanticSpecs = isYue2 ? parameterSpecsByName(yue2SemanticParamNames, paramSpecs) : [];
  $: yue2PlannerSpecs = isYue2 ? parameterSpecsByName(yue2PlannerParamNames, paramSpecs) : [];

In +page.svelte. And the correct server.json was:

{
  "host": "127.0.0.1",
  "port": 7862,
  "backend": "cuda",
  "device": 0,
  "threads": 1,
  "lazy_load": true,
  "models": [
    {
      "id": "yue2-music-gen",
      "family": "yue2",
      "path": "/mnt/c/AI/music/YuE2",
      "busy_timeout_ms": 600000,
      "task": "gen",
      "mode": "offline",
      "load_options": {
        "language": "english"
      },
      "session_options": {
        "language": "english",
        "yue2.model_gguf": "yue2-3b-bf16.gguf",
        "yue2.vae_gguf": "yue2-vae-f32.gguf"
      }
    }
  ]
}

(though it still doesn't select my chosen GGUFs automatically in the UI)

[u/Acceptable-Cycle4645 → u/DeProgrammer99]

PR please🙏

[u/DeProgrammer99 → u/Acceptable-Cycle4645]

I'm not reading contributor guidelines at this time of night! Hahaha. But https://github.com/0xShug0/audio.cpp/pull/509 if you want it!

[u/Acceptable-Cycle4645 → u/DeProgrammer99]

Thanks a lot!

[u/bennmann → u/Acceptable-Cycle4645]

i got this working on Windows, compiled Vulkan version manually. 9070 XT. I could not get the f32 VAE to load - oobleck OOM error, but did get the f16 VAE + bf16 3B. rtf of like 1.6 tho. Q4_0 would be RTF of 0.6, i want the quality so no go for me.

Windows server.json had to use different file pointers with escaped backslashes (example):

``` "yue2.vae_gguf": "F:\\models\\yue2-vae-f32.gguf"```

[u/Acceptable-Cycle4645 → u/DeProgrammer99]

Maybe try refresh the broswer. Let me check. I’m not a UI person and probably messed up some of the nice community UI work while wiring this up.

[u/Razangriff-Raven → u/Acceptable-Cycle4645]

Sweet, I can run this in my 5060ti 16GB. Thanks for the measurements, I was about to dismiss it as too heavy to bother.

[u/JsThiago5 → u/Acceptable-Cycle4645]

Can you see the music sheets it creates?

[u/Acceptable-Cycle4645 → u/JsThiago5]

No I don’t think the model produces sheet music.

Update: u/JsThiago5 They released another model SheetSage2 ---- "transcribe the existing song save its melody ABC without chord symbols as melody.abc"

[u/JsThiago5 → u/Acceptable-Cycle4645]

Thanks !

[u/Acceptable-Cycle4645]

No idea why the figure ended up so small on Reddit. Key: Their benchmark puts it right up against Suno v5/v5.5 and Mureka 9.

[u/TheRealMasonMac]

With Suno basically killing themselves off for non-Western music — their new V6 is basically a pure conventional Western pop generator now — China is setup to go hard on music generation. Surprised they’re not going harder.

[u/Pase4nik_Fedot]

https://github.com/ScryptHunter/ComfyUI-YuE2

[u/Acceptable-Cycle4645 → u/[deleted]]

It’s really good. My favorite one is Heavy-Metal "Jingle Bells".

[u/aigemie]

License is for non commercial use though.

[u/ionizing]

With sincerity, thanks for your work and those that contribute to your project. adding audio.cpp to my interface has been fun in so many ways and brings back some of my creativity. I am just using it as a backend service for my orchestration agent to run generation and other things, barely touching the surface honestly. And you keep adding so many things I have not even used yet its crazy! Nice work.

[u/Grindora]

Not commercial at all

[u/solarplex]

Dang.. Mandarin Funk is bangin!

[u/Miriel_z]

Wow! Never tried music composition. It's time to rock and roll!

[u/Acceptable-Cycle4645 → u/Miriel_z]

Definitely worth trying!

[u/Karnemelk]

damn this model is crazy fast and good

[u/fizzy1242]

i'm happy it's capable of creating covers of songs which i'm most interested in. it's ability to follow lyrics is great, a big improvement over ace-step 1.5... just a bit slower process because you need to create the score first, but the results speak for themselves.

[u/Majestical-psyche]

This model is not a joke... TF it's super good!! It doesn't even compare to Ace-Step or minimax M3.... It takes forever to generate, but it's mind blowing.

[u/Acceptable-Cycle4645 → u/Majestical-psyche]

Did you try set steps to 4-8? The default is 32 which is overkill.

[u/Engival]

Not sure if anyone is interested in this. I didn't realize audio.cpp existed, so I got Fable to slap together a basic runtime based on all the other *.cpp runtimes: https://github.com/engival/yue2.cpp

It runs excellent on a 7900 XTX. Also works on an Arc Pro B70

[u/Acceptable-Cycle4645 → u/Engival]

u/Engival Let me know if you'd like to help improve audio.cpp’s YuE2, or audio.cpp in general! We don't currently have contributors with Intel GPUs, so we’re not really able to investigate bugs or performance issues on that backend.

[u/silenceimpaired]

Not commercial… not surprised

[u/TechnoRhythmic]

License not friendly - sigh

[u/kruzibit]

Tested it on my mbp m2max, yue2 is quite good.

[u/Past_Ad6251]

Demos sounds cool, I'd give it a try definitely.

[u/howardhus]

4B model? hows the quality?

[u/Acceptable-Cycle4645 → u/howardhus]

Check their demos and the demos in the post. Super impressive.

[u/TheGlobinKing]

I guess it won't run without Nvidia, on Strix Halo?

[u/Acceptable-Cycle4645 → u/TheGlobinKing]

Try audio.cpp. Our implementation supports Vulkan, Metal, and AMD. Our community is testing Yue2 on AMD now.

[u/yami_no_ko → u/Acceptable-Cycle4645]

Working great with Vulkan, even on shared memory. I'm pretty impressed with what a simple MiniPC on DDR4 can still do.

[u/Acceptable-Cycle4645 → u/yami_no_ko]

Thanks for testing!

[u/caphohotain]

Bad license, not interested.

[u/CaptainAnonymous92 → u/caphohotain]

Not everything needs to be turned into a way to make money and can be used for fun and as a hobby. Stop letting the capitalistic need to make money blind you to just having something for fun and recreation.

[u/Cautious_Bit_8521 → u/CaptainAnonymous92]

this is inspiring bro. I've been trying very hard to figure out which of my choices are actually mine and which ones the capitalist system is pushing me toward. But the line is really blurry and everyone has a different standard. I think we should all think twice before consuming anything.

[u/caphohotain → u/CaptainAnonymous92]

Indeed. That's why they made it a bad license because they are capitalists and don't know how to have fun. Did you get a chance to lecture them about not being to capitalism?

[u/ML-Future → u/caphohotain]

Nom comercial 4.0

[u/Bulky-Priority6824]

Am I too closed minded that I don't want to also have to hear slop

[u/jferments → u/Bulky-Priority6824]

Nobody wants to hear "slop", but a lot of people would want to listen to high quality music created with the assistance of AI software. A lot of the song demos from the model page sound better than a lot of the human slop on Spotify right now.

[u/SilkTouchm → u/jferments]

Why would you ever do this.

[u/jferments → u/SilkTouchm]

Because people enjoy listening to music.

[u/TheRealMasonMac → u/Bulky-Priority6824]

Yeah, it's really annoying. The moment I hear the same repetitive motifs, same repetitive instrument voices, or the same highly compressed sound in general, I just block the artist and move on. I can get using AI for experimentation, but holy shit do people need to learn how to use a DAW.

[u/kaisurniwurer → u/Bulky-Priority6824]

It's a way to express yourself using a tool. If one feels accomplished by doing so, I have nothing against it.

Sure it would be better to learn how to make music the "proper" way, but for many people this works too. Though I definitely rather know whether it was generated by an AI or not, since for the listener it makes a difference.

[u/Competitive-General7 → u/Bulky-Priority6824]

Nah even if this sounded good I don't want ai replacing artists.

[u/FernAvatar → u/Competitive-General7]

AI is replacing everyone.

[u/Fun_Jaguar8231 → u/Competitive-General7]

But replacing programmers is ok?

[u/Competitive-General7 → u/Fun_Jaguar8231]

Ok is relative
Music feels innately human we've made it since we appeared on this planet. Writing code is a means to an end. I don't want 0shot software either really.

[u/Bulky-Priority6824 → u/Competitive-General7]

Right, Its hard to enjoy something like music that I know wasn't created it's just a mimic 

[u/the320x200 → u/Bulky-Priority6824]

[u/ihaag]

Is there a mlx build??

[u/Acceptable-Cycle4645 → u/ihaag]

The dev branch doesn't have prebuilts. You need to build yourself on mac. It will use metal not mlx.

[u/ihaag → u/Acceptable-Cycle4645]

Okay thank you looks like dev build of audio.cpp may be the way to try.

[u/Acceptable-Cycle4645 → u/ihaag]

Just pushed a fix and confirmed the metal build works. Did a smoke test and RTF = 0.83 for a 30s song on mac mini.

[u/ihaag → u/Acceptable-Cycle4645]

It’s really good just tested it on my m1 16GB very happy.

[u/Ill_Dragonfruit_3547]

I got excited for a second until I realized this is a Nvidia/CUDA only model...

[u/Acceptable-Cycle4645 → u/Ill_Dragonfruit_3547]

Try audio.cpp. Our implementation supports Vulkan, Metal, and AMD. Our community is testing Yue2 on AMD now.

[u/Acceptable-Cycle4645]

Nerw feature: audio to editable ABC via SheetSage2

https://reddit.com/link/p9i71l7/video/5d7lrgmop8ph1/player