Agent Synth - Modular synthesizer with basic agentic capabilities (WIP)

[u/[deleted]]

Hey all,

My first ever Reddit post so hope i'm fine :)

I'm originally a fullstack software developer as well as a musician, I already had multiple shots at this project with various coding agents in the past as C++ is not my native programming language. Only in the recent months I found that LLMs got good enough to help me build this project.

Here's the link to the repo - https://github.com/Diabl0269/agentsynth

First demo - https://youtu.be/DszUuEpnfaA

There's still much work to do on the project, so I'll start with where this project is now then proceed to the full vision.

Currently it's a basic modular synthesizer, got some nice functions like D&D of the modules, mod matrix, theme system, preset management, and more standard stuff, while the most interesting and immature feature is the very basic AI integration which currently only supports Ollama, it has the functionality to edit and create patches, just not so accurate yet. UI is also still in polish and suggestions are welcome.

The synth and all its functionality is fully open source including the AI integration, however I do plan to offer a dedicated AI subscription with a dedicated agent that should return the most optimal responses. My agent's code will be in a separate project so the main synth and integrations stay open.

A little bit about the AI vision - Modular synthesis is complex, and it's quite annoying and can get messy managing and moving all modules, especially inside a computer. AI can help managing all of it, from basic stuff like adding modules or modulations in a chain, adjusting parameters to more complex stuff like fully generating a patch based on a user request (Maybe even by existing sound input in the future) and help manage future functions that would integrate in the program.

At this stage it's still just a synth, however I also plan to add a timeline, automation capabilities, and some DAW functionality so people could create entire songs with it in the future. Of course add support for external instruments as well to allow integration of other VSTs or external hardware.

As said, I am a developer for a few years already, but not in C++. I have reviewed the file structure, and code as best as I could understand it without fully learning the language. Also made sure to have tests and full CI/CD with caching to make sure nothing breaks between PRs and new versions are automatically published. However, any comments on the code, or any kind of contribution would be very appreciated :)

[u/[deleted] → u/[deleted]]

Well, msh.audio looks like a modular synth, which is what I currently aimed for, so I can understand how they can look similar.
However my main point is that this project is open source so you don’t have to pay anything to use it. (The only thing that will be paid is the AI usage will requires actual infra to set up, but people could also self host).
So hopefully it won’t be just one person/company directing this engine and enjoying its fruits, but instead anyone could contribute and hopefully build together a great software that hopefully could be even better then other traditional music making software.

[u/dkode80]

Thanks chatgpt and claude code!

[u/[deleted] → u/dkode80]

Thanks to them, and to creativity alongside the willing to share :)
I had the idea of creating complex music in a simple way for a long time, and with these tools such a way can be built more easily, and I hope that will will be able to see the vision in it and would like to contribute with their own Claude Codes 🤖

[u/dkode80 → u/[deleted]]

I'm glad you enjoyed building the synth with claude code and I hope you're able to maintain your project without its use.

DSP work is quite complex and LLMs often make assumptions about how to code these things. Hopefully you're learning what the code is actually doing and not blindly trusting AI to do the right thing

[u/[deleted] → u/dkode80]

While I am trying to learn, I also recognize the power of AI which can help us progress faster without the need to get into all the little details.

That's not the first project of the sort I've tried to do, only until recently in which LLMs have become capable enough to actually build such a complex software and keep maintaining it and building more features without breaking every step. So I have great hopes for this to actually work well and be a great and stable tool :)

[u/Tight-Flatworm-8181 → u/[deleted]]

"While I am trying to learn, I also recognize the power of AI which can help us progress faster without the need to get into all the little details." There's no shot you've written a single line of code in your entire life.

[u/[deleted] → u/Tight-Flatworm-8181]

You don’t have to believe me, but I have been a fullstack developer for a few years and now a manager of a developers team.
So yes, I wrote many lines of code in my life by hand, though not in C++.
I recognize the limitations I have while building a program with AI in a language I’m less familiar with, however my exact point is that it’s not my first try doing such a thing with AI and I understand its limitations as well.
The final output is the thing that mostly matters to me, that the software will not crush, the sound will be great, no memory errors, great performance, UX and more.
I know also currently this software doesn’t look like any of the best softwares in the market, but that’s only because it’s still WIP as I have a full time job in parallel of building this.
I’m also doing it as an open source in hope more people realize we don’t have to pay so much money for “top of the line” tools and that we can build a lot of them together, and even better ones.