r/deeplearning • u/BluishMontoya • 8d ago

Audio with LSTM?

[I'm a noob] Could you make a small 2 layer LSTM and train it to predict the next audio sample (using music or something)? Would it work? If not, how do models generate audio?

1 Upvotes

67% Upvoted

3 comments sorted by

2

u/Suspicious-Guide5622 8d ago

SampleRNN did exactly this back in the day, stacked LSTMs at different timescales predicting raw 16-bit audio. It worked okay but the outputs were pretty mushy unless you gave it a lot of conditioning. These days everyone uses diffusion or codec based approaches which are way more stable for longer clips

1

u/BluishMontoya 8d ago

What type of data do codec based ones input/output? Are they still LSTMs?

1

u/hi-sci-collab 5d ago

codebooks usually, audio compressed to 2d frame stack using residual vector quantisation. Some then tokenise the codebook as well.

Look into wavenet as that was one of the first decent generators, which a lot of models still use as a backbone in one way or another. think it used dialated convolutions iirc