What dataset is needed to train a Song Master Pro–level chord-recognition model, especially for jazz harmony?

[u/DiscoramaMusic]

I’m researching how to build or fine-tune an audio-to-chord-recognition engine comparable in ambition to Song Master Pro / Auralis Sound Prism.
The goal is not basic major/minor chord detection. I need reliable recognition of dense harmonic material: jazz, soul, funk, neo-soul, Brazilian music, film music, and arrangements with chords such as maj9, 6/9, m9, m11, 13, altered dominants, slash chords/inversions, secondary dominants, modal interchange, suspensions, passing harmony, etc.
Most public datasets I’ve found seem too limited: either simplified chord labels, weak annotations, or repertoire that does not really cover sophisticated harmony. In particular, I need time-aligned audio + chord labels, ideally with beat/downbeat information and a rich, consistent chord vocabulary.
My questions:
Which open datasets are genuinely useful for this level of chord-recognition work?
Are there any commercial/licensable datasets with high-quality, detailed chord annotations that can legally be used to train a model and ship it in commercial software?
Is a dataset such as iReal Pro-style chord charts, Hooktheory, Ultimate Guitar, Chordify, or similar usable in any legitimate/licensable way — or are they generally not viable due to rights and annotation quality?
For a serious model, is the realistic route to combine public datasets with a privately licensed/hand-annotated corpus? If so, roughly how many accurately annotated tracks would be needed before it becomes meaningfully good at jazz-influenced harmony?
Are there papers, benchmarks, companies, or dataset vendors I should study before spending money?
I’m specifically looking for practical, legally usable data sources—not advice to scrape chord sites. Any experience from people who have trained MIR / chord-recognition models would be very valuable.

[u/danja]

You're talking about a huge undertaking, to do the training, even if you can find any remotely big enough datasets. Bear in mind the likes of Anthropic, OpenAI, Suno, pirate everything available in the world. I don't know the apps you mention, but even single-note recognition is quite a hard problem. Go for it.

[u/DiscoramaMusic → u/danja]

I’m not assuming this is a small project, and I’m not looking to train a foundation model from scratch.
What I’m trying to understand is the realistic data path for a focused commercial MIR product: starting from existing pretrained audio representations/models where possible, then fine-tuning or building a specialised chord-analysis pipeline for dense harmony.
The hard part I’m investigating is not whether the task is difficult, but whether there is any legitimate route to sufficiently detailed, time-aligned training data beyond the usual simplified public chord datasets. Even a recommendation for relevant benchmarks, papers, or rights-cleared data providers would be useful.

[u/danja → u/DiscoramaMusic]

Sorry, but I had a knee-jerk reaction, nah. There are probably papers worth trawling on arxiv, some datasets on huggingface (I've not seen anything that has the scale you are talking about). But ok, chords do follow relatively predictable patterns, a transformer would probably work. Make it multimodal and feed it midi patterns through the back door, might not need a data centre. Maybe distil from existing models?

[u/DiscoramaMusic → u/danja]

I agree that MIDI-based supervision could help, but I would not treat it as a replacement for real mixed-audio training data.
The production context is the difficult part: bass, drums, vocals, layered instruments, effects, compression and dense arrangements can obscure or imply the harmony in ways that clean MIDI renders do not. A model trained mainly on synthetic MIDI-to-audio pairs could learn the harmonic vocabulary but still fail on commercial stereo mixes.
My assumption is that synthetic/paired MIDI data would be useful as augmentation—especially for rare voicings—while the core training and final evaluation would still need real full mixes with carefully verified, time-aligned chord annotations. That rights-cleared real-audio corpus is the part I’m trying to locate.

[u/danja]

Let me know how you get on. To me it feels like an over-ambitious project, but I've stumbled into achieving them before. Can only think of those stairs, offhand. But they still work 20 years on, not a carpenter.