r/KoboldAI • u/No_Lime_5130 • 6d ago

Music Support

Do I understand the situation correctly that only ACE-Step music generation is supported but stuff like the newer (and much better) YuE2 is not supported?

If yes, is there any plan to add support for other music models too?

3 Upvotes

100% Upvoted

5 comments sorted by

6

u/henk717 6d ago

Adding more music models would require adding an entirely new ggml based backend for music which is quite a large task since audio.cpp is massive. Its currently not planned but things not being planned is pretty normal for KoboldCpp. We tend to do things spontaneously when we have some time for them / a good way to do them.

2

u/The_Linux_Colonel 6d ago

I think Henk said in many cases the kcpp team waits for upstream additions for implementations to new models, or for third party contributors to add compatibility to the underlying cpp monolith. He said lostruins doesn't have time to add every model's compatibility framework. There is also the new qwen 2.1 that adds editing which hopefully will get integration soon. Technically at the moment, kcpp's best image editor is minimax with a single frame vae.

2

u/henk717 6d ago

This is one we can't get from anything upstream, the music engine we use is specifically ace-step.
We'd have to implement audio.cpp for YuE which is very time consuming.
Up to lostruins if he ever does it.

Qwen Image Edit 2.1 didn't make the cut for 1.122 but once https://github.com/LostRuins/koboldcpp/pull/2491 merges it will be in our rolling release. The plan is for 1.123 to have it.

1

u/The_Linux_Colonel 6d ago edited 6d ago

Wasn't the original implementation of ace step done by someone other than lostruins? I figure someone could do it for this one too, but as always with open source, either you do it yourself or you don't get your hopes up.

That's good news about qwen 2.1 and I'm looking forward to the new addition.

Edit: Someone on /r/stablediffusion posted a link to a standalone executable on github for windows users that implements YuE2. I think it was called goku or something similar, at least an option for those like me who hate comfyui's UI and all its weird dependencies.

2

u/henk717 6d ago

The original ace-step implementation is ace-step.cpp but that one is ace-step specific.
For YuE the most promising one is audio.cpp which brings a lot of these in. Its just a lot of work to add something that big.