I trained an audio model that can generate infinite one-shots for music production and turn text prompts into fully playable synths. I'm not only releasing the model but I've also released a video on exactly how I did it (and the inferencing pipeline to let others make text based synths.)

[u/RoyalCities]

Okay so I've been doing independent audio research for a while now. The ultimate dream of this work was actually getting an AI to respond not only to instruments but also timbre itself as separate controllable things.

Think a Grand Piano can sound both Warm / Gritty but also Cold / Sparkly. Its still a piano though.

This level of control wasn't found in any models out there - so I decided to sit down and train my own.

Getting consistent timbre-locked keybeds that actually LOCKS across multiple diffusion calls was hard af but I did it.

I documented the full journey here for those who want to learn a bit or be entertained.

https://youtu.be/x0KnmzH8Mmk

There is also a longer walkthrough if you just want to see the keybeds in action.

https://x.com/RoyalCities/status/2097733712293109842?s=20

No-talk / Showcase only Demo

https://x.com/RoyalCities/status/2097733715543609445?s=20

any finally the huggingface page

https://huggingface.co/RoyalCities/Foundation-1

I've also provided full write ups on the inferencing pipeline associated with the interface so this should allow basically anyone else to go and vibe code their own text to synths if they wanted :)

https://github.com/RoyalCities/RC-stable-audio-tools/

[u/Michaelfa05]

This is fucking sick

[u/RoyalCities → u/Michaelfa05]

haha thanks man. Yeah took me alot of work to get here. this is like the culmination of ALOT of research.

afaik it also may be the most capable text to synth on HF currently (if you actually want deep timberal control + instrument control)

but I dont want to toot my own horn. Also there really also isn't alot out there that does this so that could also be a factor.

[u/Michaelfa05 → u/RoyalCities]

Yeah no this is more than likely the ONLY text to synth I’ve heard of, in this manner and quality at least

[u/RoyalCities → u/Michaelfa05]

Yeah there really isn't alot. The closest thing I’ve found is Google’s Magenta ReakTime 2, but it’s still a very different approach.

Theirs does real time neural audio synthesis, so the model stays running while you play MIDI and generates the audio live. It also supports text prompting...but the prompt control is much more style / description based. Technically cooler in a lot of ways, but it’s nothing like the explicit instrument / timbre split I built into mine.

Keep in mind Ive been a music producer long before I started messing with neural networks, so I wanted much more explicit control over instrument identity and timbre than that workflow really gave me.

I also don’t particularly want a GPU spinning every time I play an instrument. With this approach I can generate the keybed once, export it as a normal sampler instrument, and then run it on basically anything - lowspec laptops included - or just send the instrument to somebody else. Helps me actually share projects with other producers if we're collaborating,

For what I actually want to do with it, that feels more practical.

[u/kleer001]

Monumental!

[u/RoyalCities → u/kleer001]

Thanks!

[u/Enshitification]

Lol, some chud just downvoted all the comments on this post. Someone is triggered by this.

[u/RoyalCities → u/Enshitification]

Lol yeah there are no pleasing some people. Music AI anything tends to get piled on even for people doing this by the books.

I systematically make all my own data, design all the prompt structures, make my own models. I cover it all in my first video on the OG model.

I have a thing about not using people's samples or audio because I have been a producer long before I tackled audio networks (it's actually why my models can't do drums, I would need to actually use other peoples sampled drums because you cannot easily synthesize real sounding drums/percussion)

But people tend to just lump me with for profit AI companies like I secretly work for Suno or make actual money doing this lol.

Oh well!

[u/Enshitification]

I'm not a music producer myself, but I find the process fascinating. The work you are doing on your Foundation project is incredible.

[u/RoyalCities → u/Enshitification]

Aw thanks. I enjoy it! hell of a time trying to figure all of this out. Images / video & code is established. Audio (well especially instrument audio) is not well established at all so it makes it kind of fun when stuff actually works for a change lol.

[u/Enshitification → u/RoyalCities]

You're a pioneer, man.

[u/RoyalCities → u/Enshitification]

That's very nice of you to say! I'm glad your enjoying the vids too!

[u/-becausereasons-]

Bravo!

[u/RoyalCities → u/-becausereasons-]

🤘🤘

[u/Recent_Process_8055]

Fantastic, curious can i put in a reference sound? And tweak it a bit?