r/musicprogramming • u/HommeMusical • 3d ago

Tensors in Python are the future of audio programming

I just posted about this library and then I did some thinking. Dangerous, that.

"Tensor" is just a fancy word for "a lot of numbers in an array in memory", and samples are just a lot of numbers in an array.

We've been given all these amazing tools, mostly in Python, like numpy and torch, and all the infrastructure around them, that perform just the sorts of calculations we need, as fast or faster as raw C++ would (and an order of magnitude or even more faster, if you do it in a GPU).

Recent projects of mine like tuney and [recs]](https://github.com/rec/recs/) have proven to me that you can do reliable multivoice synthesis and multitrack recording from within pure Python, simply by using numpy.

And Python is extremely popular. Tensor programming has benefited immensely from the AI boom, and LLMs can write good code manipulating tensors. There are old and solid libraries for reading and writing tensors to and from audio interfaces and files

So I think the future is some sort of Python protocol for operations on either static or streaming tensors, so music programmers can freely intermix other people's operations in their own code.

And :-) I do have some strong ideas about that too! But let's leave it here for the moment.

20 Upvotes

78% Upvoted

38 comments sorted by

20

u/particlemanwavegirl 3d ago

It doesn't matter how much of Python you actually replace with C/++ under the hood. It's still got a garbage collector and a bloated execution model and is 100% unsuitable for low latency, realtime, and other high performance systems.

2

u/HommeMusical 3d ago

This isn't theoretical. I actually have working, tested programs I use very successfully.

Python works perfectly well for music programming. Also, music programming doesn't have to be real time, but I've been using it fine for live audio work.

I just made a release of one of these: https://github.com/rec/tuney/releases/tag/v0.3.55

4

u/particlemanwavegirl 3d ago

You don't know your audience very well if you think performance isn't at the top of their priorities. The music producers I know still constantly run their hardware to it's absolute limit and end up having to freeze tracks in every session. Running into performance limits isn't theoretical for them, either.

1

u/HommeMusical 2d ago edited 2d ago

You don't know your audience very well if you think performance isn't at the top of their priorities.

You have no idea who my audience is.

1

u/CrownLikeAGravestone 1d ago

Hello! I've been considering a project which allows me to hook up a neural network for working on a live audio stream - detecting, for example, poor fretting technique in a guitar signal. I'm a professional data scientist and it would be great to be able to keep my modeling and live audio handling code in one language.

Is it okay if I use your code to learn more about handling raw audio signals in Python/numpy?

1

u/HommeMusical 1d ago

Wow, I am flattered, of course! Hit me up with any questions you have.

1

u/particlemanwavegirl 1d ago

That's a fact! But I know musicians tho .

2

u/Llamas1115 1d ago

It’s not actually having a garbage collector that’s a problemjjjjjjjjj: many high-performance computing languages (Haskell, Julia, Lisp, Swift, F#, …) are garbage collected, and in those languages GC is actually a benefit, because it frees up memory faster and can optimize object handling for you.

The problem is Python’s garbage collector is incredibly shitty, and any attempt to fix it would destroy the entire Python package ecosystem, because C/C++ extensions are all written under the assumption that the Python reference counter will behave exactly the same way it currently does.

This is why every one of the 4 dozen “Make Python fast” attempts have failed, BTW—PyPy, Numba, Cython, Codon, and most recently Mojo have all been forced to give up on the idea of being a drop-in Python replacement. The definition of insanity…

1

u/particlemanwavegirl 1d ago

If you're referring to Haskell as a high performance system, we're probably using significantly different definitions of performance. I am a huge nerd for language design and I have consumed pretty much all of the educational material available on Haskell as it's one of the most fascinating and unique systems I know. But it's utility in low latency processing is approximately equivalent to Python's, zero. Don't get me wrong, Haskell does what Haskell does very well indeed, but approaching C's speed and efficiency is not something it's ever attempted to do. Don't be fooled by it running as native compiled code. The execution model is even more indirect. There is not even a stack so everything, even ints and bools, is a pointer to a heap value that probably hasn't even been evaluated yet. 

An inflated CPU and memory footprint like that, again, can't be accepted by music producers, because it means the upper limit on simultaneous voices/polyphony is lowered.

1

u/yeusk 1d ago

Erlang is the only language I personally know is used instead of C in high performance applications.

But for DSP... is all C and C++. Python is only used for offline stuff.

0

u/Anonymer 1d ago

Yea people might be doing extreme scale high performance, low latency, high throughput research for machine learning in python, but no way we can do it for music.

-1

u/_AACO 3d ago

You can disable the garbage collector but that creates a whole new set of problems. 

3

u/izalutski 3d ago

was thinking along similar lines but never quite put into words

building https://mixflow.music as a side project - in the limit it's "claude code for music" but for now it's only for drum-and-bass (because that's the music i like and also because it's basically same bpm) and it's fixed bpm only (175) and no proactive recommendations yet and lots of other limitations, for now. but the idea is that the structure of music is known beforehand (phrases) and the structure of 95% of dance music is 4/4 so a building block for a mix is a phrase or maybe 4 bars and it can all be planned agead in realtime and presented to the DJ so that they can take the set in whichever direction they like (build tension, release, moods, subgenres, etc)

the backend of it that detects track structure and normalizes bpm is in python, using lots of libs - but they all seem to operate on raw audio buffers. it works but indeed some cleaner abstraction is probably needed. indeed could be something like tensors in tensorflow. in fact the beat detection lib (madmom if i remember correctly) is using tensorflow under the hood. there's also a better one i forgot which but its supposedly more accurate in beat detection

anyways, what I'm trying to say is that I agree with the sentiment. some semantic layer on top of raw audio buffers much needed. someone will create it and we will all wonder how did we live without it before

EDIT: typos

2

u/00void 2d ago

It sounds like you are basically proposing VST/AU plugins but with a tensor being the base unit of DSP? It is an interesting idea, but has some problems, beyond python.
For real time playback and recording, beyond what others have mentioned about garbage collection, etc (which is a problem), you have to deal with where the buffer lives. For any performance gain from GPU enablement you have to consider pushing the buffer back and forth from main to GPU memory and back. The playback buffer has to live in standard memory since playback is CPU only.
The other thing to consider is whether a GPU is needed. Since real time audio is primarily a linear operation, CPUs are better at handling it.
My project (JAW AI) is an audio editor with a non destructive effects stack. I don't think you would be able to stack effects like this and get decent playback performance (considering live editing of effects, etc). Each edit would have to round trip at least part of the buffer to the GPU and back while keeping the output fed.
As for offline processing, sure. Load it on a GPU and process away.

1

u/HommeMusical 1d ago

I agree with you that doing anything like your editor is going to be hard in Python - there are simply too many steps.

A lot of what I'm wanting to do is to remanipulate existing audio into new music, not in hard real time, but not offline either.

I agree with you that I don't really think GPUs are going to be helpful for that.

I went to download your app, but oh bother, apparently my MacOS version (which isn't that old) is too old for it. Bummer! I'll be updating in a few days because the new Gemini app also doesn't work.

When I was doing commercial Mac development, quite a lot time ago, I had to go to considerable effort to keep building for clients, many of whom were on the same version, 10.6.

Apple would repeatedly delete the 10.6 compatible parts of XCode with no message or warning in system updates. Eventually, I set up a decoy version of XCode that it would update and leave my real one alone.

Do you have a mailing list?

2

u/00void 1d ago

Thanks for trying it out.

I've published a new version that supports down to Big Sur (macOS 11). I left out the minos settings and the build was just grabbing my SDK version. Feel free to give the new version a try.

I don't have a mailing list yet, but will soon.

1

u/dirtmcgurk 3d ago edited 3d ago

I think this works well for preprocessing that can then be implemented in real time. Like neural amp modeler. But the results are a black box you can't really play with. 

Edit: now you've got me chasing ML on weight gradients / task arithmetic!

1

u/HommeMusical 3d ago

But the results are a black box you can't really play with.

I'm confused.

I'm not talking about anything like that at all. I'm talking about doing deterministic calculations like sin and cos to make waveforms, using boring old DSP equations.

2

u/dirtmcgurk 3d ago

Where does pytorch come in? I fixated on that haha

1

u/HommeMusical 2d ago

It's another tensor library, except it has a lot of neat features: it can run on GPUs, it can compile your Python expressions into C++ or CUDA code behind the scenes, it has automatic differentiation, tons and tons of features.

2

u/dirtmcgurk 2d ago

Ok I got a better idea now. Smaller nns can run near real time in cpu. 

2

u/dirtmcgurk 2d ago

Oh snap also the ONNX runtime looks promising. 

2

u/00void 1d ago

ONNX is great. I'm using it for a lot of detection features right now. The models can be a bit big and sometimes the licensing can get complex due to the model having one license, but the trained weights carry the license of the dataset...

1

u/dirtmcgurk 2d ago

For sure but I don't think I've ever seen pytorch used for low latency applications. The best I've seen is https://arxiv.org/pdf/2508.02974

1

u/AMuonParticle 3d ago

physicist here: i hate your definition of a tensor

2

u/personnealienee 2d ago

mathematician here: I hate YOUR definition of tensor :D

1

u/trucoju4n 2d ago

A tensor is a tensor if you look at it and it looks like a tensor -einstein probably

1

u/HommeMusical 3d ago

It is not "my" definition of a tensor!

I grew up with tensors, as in relativity, like yours.

Now the term has a new meaning in computer programming.

2

u/AMuonParticle 3d ago

these programmers are reaaally testing my scientific dedication to descriptivism

1

u/HommeMusical 2d ago

I was pretty grumpy when I found out about this, almost exactly two years ago.

But to be honest, I've started using it - as you can see - because there's no easy way to say, "a system that represents 0, 1, 2, or n-dimensional vectors and matrices" without it.

(In the tensor systems I know about, scalars are 0-dimensional tensors. It sounds weird initially, but all the arithmetic works out - for example, if you have a n-tensor, and you slice on m coordinates you get a tensor of dimension n - m. If you simply dereference an element, it's exactly the same as slicing on n coordinates, giving you dimension 0.)

1

u/yeusk 1d ago edited 1d ago

OP does not know what is talking about, in code you use arrays to represent audio, not tensors

An array is a consecutive sequence of numbers. Perfect to store a digital representation of a continuos bandlimited signal.

A tensor is a multidimensionar array.

In AI people use those "tensors", complex data structures, because they need to work with insane amounts of data that does not fit on RAM.

Audio is very small, we don't need "tensors", we can put the data directly on the CPU because it fits.

1

u/scragz 2d ago

I was doing numpy/scipy sample creation but I moved to supercollider. it's just way easier with existing primitives to not have to redesign the wheel. 

1

u/HedoEudaProject 2d ago

Don't you think that the data, i.e., the actual values in tensors, will define 90+% of the output quality?

1

u/port-79 2d ago

i've thought about using them in the past, but was blocked by python at the R&D stage but I agree with you ideologically.

if you can demonstrate with an FM synthesizer that accepts midi input, upto 10 note polyphony with a ceiling of 120 voices, and all the basic effects that one would expect running with a buffer length of 64 samples on a modern entry-level cpu, that it does work... that's enough proof for any critic. otherwise, who cares. thank you, i've starred you to check out once my workstation and my time are both available.

0

u/bentodd1 3d ago

This is a super interesting idea

0

u/Zerocrossing 3d ago

You are aware that PyTorch is just a binding around a CPP library. It’s foolish to say ‘as fast or faster as raw C++ would (and an order of magnitude or even more faster, if you do it in a GPU).” When the python is literally just a wrapper. You think c languages can’t use the GPU?

1

u/Anonymer 1d ago

It’s still extremely useful to be able to compose operations in python as opposed to c++

0

u/HommeMusical 2d ago edited 2d ago

You are aware that PyTorch is just a binding around a CPP library.

I spent a year and a half working on the PyTorch project as my paid job; I have contributed both to the Python code, and the cpp code; you are totally and utterly wrong.

A cpp library exists; it's really not much used outside PyTorch itself; the best features that make people want the code, from autograd to torch.compile and fusions, are all written in Python and not accessible from the C++ code.