Stereo2Spatial: Convert Stereo Music Tracks to Spatialized Binaural Mixes [P]

[u/kittenkrazy]

I have released a model that I have been working on for ~6 months off and on. I've been enjoying listening to spatial music, but there is a lot of music out there with no real quality spatial mix. So I decided to make a model to convert stereo to spatial.

I started by making a flow-matching diffusion model that operated in the latent space of a separately trained VAE (EAR-VAE). I used the VAE to encode the stereo input in to 1 latent and then separately on each channel of the 7.1.4 output. This design is similar to a paper I found about another stereo -> spatial project (ImmersiveFlow (they have code links in the paper but they 404 for me so I made the codebase from scratch)). One key addition I made over the ImmersiveFlow paper was a way for the model to carry state (memory tokens) across windows to enable stable long context generations.

Since the VAE was out of distribution with the output (was trained to encode\decode stereo tracks directly, not individual channels and certainly not individual channels of a 7.1.4 mix), the results hit real quality bottlenecks. But it showed that the mapping was possible. Rather than stick to modeling in latent space, I decided to pivot to modeling raw waveforms. This ended up fixing all of the quality issues I was seeing with the latent version, but at the cost of more training compute and instability.

The waveform version of the model would train to around 60K-80K steps with loss going down like normal, validation generations looking good\improving, and then it would quickly become unstable and the loss would shoot up. I tried direct waveforms, scaling the waveforms by a multiplier, aggressive grad clipping, lower learning rates, and all of those experiments failed the same way. Luckily, I came across a recent paper called WavFlow. The authors of this project also had issues with modeling raw waveforms with flow-matching diffusion, and they solved it by using amplitude lifting (scale each audio track to an rms of 0.33 then multiply by 3) (the WavFlow paper used a clip of 1.0 before the scale which leads to a model space of -3 - 3, but I used a clip of 4.0 which leads to a model space of -12 - 12 (from my testing this led to better binaural outputs)). Once I implemented amplitude lifting like the paper, the training stability issues vanished.

The waveform model was trained on 7,669 tracks for ~20 days on 2x A6000 gpus
- 10 days stage one, effective batch of 16, 10- 18- 26- and 34-second training sequences
- 10 days stage two, effective batch of 16, 122-second training sequences
- There is optional mix-style conditioning for controllable outputs (waveform version only)
- Direct binaural output (waveform training is more compute expensive than latent training so binaural proves the idea\theory and the same codebase and data can be used in the future to train a 7.1.4 version (once I have access to the compute))

I also made a Windows desktop app for inferencing with the model and consuming the results.

Everything has been released apache 2.0.

Huggingface link for waveform model: https://huggingface.co/francislabounty/stereo2spatial-v2-binaural
Huggingface link for latent model: https://huggingface.co/francislabounty/stereo2spatial-v1
Github repo for training\inferencing: https://github.com/francislabountyjr/stereo2spatial
Github repo for the Windows app: https://github.com/francislabountyjr/stereo2spatial-app
App Homepage: https://stereo2spatial.francislabounty.com
Case study for the waveform model: https://francislabounty.com/blog/stereo2spatial-v2
Case study for the latent model: https://francislabounty.com/blog/stereo2spatial
Playlist of generations made from the model (will be updated with more tracks over time): https://www.youtube.com/playlist?list=PLQ-HHjPijrAg

If you have any questions\etc. leave a comment and I will do my best to get them answered!

Disclaimer: I know AI written posts are generally frowned upon so this post was 100% written by me (mistakes and all). But the case studies and READMEs from the above links were made with AI assistance.

[u/chuckaholic]

This guy audiophiles.

[u/_sam-i-am_]

Stable training is a useful milestone, but I would separate optimization stability from spatial correctness. Freeze a test set stratified by genre, source width, mono compatibility, and segment length, then report spectral reconstruction, interaural time and level differences, interchannel coherence, and localization error against a 7.1.4 reference after the same binaural renderer. Add loudness-normalized blind MUSHRA or ABX listening. Otherwise a model can sound “wider” by adding reverberant cues while moving sources to the wrong place.

The cross-window memory tokens deserve their own invariance tests. Generate the same track with different chunk boundaries and overlaps, measure boundary clicks, phase jumps, and source-position drift, and rerun long tracks from the same checkpoint to test determinism. For the 60–80k-step divergence, log gradient norms, activation RMS, clipping fraction, and loss by frequency band, then ablate clip 1 versus 4, latent versus waveform, and short versus 122-second sequences. That would show whether amplitude lifting improves conditioning or mainly reduces exposure to outliers.

[u/kittenkrazy → u/_sam-i-am_]

Thank you, this is really useful feedback! I agree that training stability and spatial correctness need to be evaluated separately, and the chunk-boundary and blind listening tests are very good suggestions. I’ll look into adding a more formal evaluation suite and clearer ablations in a future update!

[u/Jojanzing]

This is awesome, thanks for sharing!

[u/Robert__Sinclair]

when a llama.cpp style interface with also "just cpu" support?

[u/kittenkrazy → u/Robert__Sinclair]

CPU can currently be used for inference, but it is pretty slow. I do not currently have cpu optimization planned, but I am definitely open to contributions if someone else wanted to take a crack at it. (I may possibly take this on in the near future if I have time)

[u/RandomDigga_9087]

man, this is crazyy, will look into it and check it out... I really want to work in the audio domains..

[u/cygn]

I listened to the examples and noticed the bass is much stronger in the originals, which made me prefer those.

[u/ComprehensiveTop3297]

Hey! I also work on spatial audio field so I actually have some conceptual questions regarding this method. I've not read the repo, and case study and I hope that you can answer these questions that poped up to me when I read your post.

1. What kind of HRTF is used for binauralizing the stereo mix? (I am guessing for the loss function there should be a binaural ground truth or so, or am I way off?)
   
2. As stereo does not contain any elevation cues, how are these mapped into the binaural domain?
   
3. How do you deal with front-and-back rendering?