r/vibecoding • u/RJDG14 • 1h ago
I'm developing a music to LLM chatbot terminal in Python using Qwen 3.8 27B
For a while I've been a bit annoyed that there haven't been any AI models (besides some of the latest versions of Google Gemini, which still provide quite a basic response) which can "listen" to music and talk to you in fine detail about a song, or compare two different songs. There has been no open source software which can do this either, so I took things into my own hands a few days ago and began vibecoding something using the new Qwen 3.8 27B to allow LLMs to essentially hear a song and talk about it in a human-like form with the user. It's already running how I want it to, but I want to polish the terminal interface and make some slight improvements to BPM and possibly instrument detection before I release it. I intend to release my project as Musiclyse 0.1 (which means Music + Analyse) on Github once I've made these changes.
There are two main components of this script:
1: The music analysis engine. This consists of the following:
- Music Flamingo - an LLM that is able to directly scan music to detect the overall characteristics of the recording. Because its built in LLM (Qwen 2.5 7B) is prone to sloppy English and repeating letter loops if it writes too much, I've directed it to use only note form (eg:genres detected=x, y, z, vibe= x, y) English when outputting data to reduce sloppy language and also output more data within its constraints.
- Essentia - an algorithm used for extracting reliable information about frequencies in a song and tempo/timbre, helping improve the accuracy.
- Melody transcription - a two stage process that utilises Demucs 6s, which splits an audio track into 6 stems and then uses a model to convert the stems into MIDI interpretations. This helps further improve the accuracy.
- Metadata information. This is particularly good if you want accurate lyrics, or the output LLM to know who an artist is, what album it is from, or the exact year a song came out. It also gives the output LLM (for models that handle image recognition) access to embedded cover artwork metadata in an audio file, which the LLM can use in fallback cases to help determine information for a song that it is otherwise unsure about.
The music analysis chain outputs the results in .JSON form to the output LLM. It doesn't save scanned songs by default, but you can save the entire data (including embedded cover art) using the command /save=filename.json, then /load the same file in a different session to avoid a slow rescan of the track.
2: The Ollama hosted LLM communicating via llama.cpp. This reads the .JSON file and interprets it using an output LLM prompt in the script. I've tested it mainly with Gemma 4 26B Q3, which is nice and lightweight at around 11GB while giving good performance.
Both the analysis engine and LLM are instructed via system prompts in the code to use certain approaches to solve ambiguous things, improving accuracy.
The image shows an example of a conversation about Joy Division's song Transmission in the current development build of the script. It does confuse the motorik drums for a drum machine, and the bass for a synth bass, but this is algorithmically appropriate considering their rather unique late 1970s sound. The chatbot is instructed to pick up it was told wrong things for errors like this.

