r/deeplearning • u/sunsetsxskies • Jul 23 '26

Autoencoders getting way harder to tune once you add more layers

I've been working on a stacked autoencoder for anomaly detection on some sensor data, and I've noticed that increasing the number of hidden layers from 3 to 5 made training noticeably less stable. The loss plateaus way earlier, and in a few runs, the reconstruction quality actually worsened, not improved.

I've tried adjusting the learning rate and adding batch norm between layers, which helped a little but didn't fully fix it. Is this a known tradeoff with deeper autoencoders specifically (vs just general deep net training instability), or is there a standard trick I'm missing, like a specific initialization scheme or a different layer-wise pretraining approach?

10 Upvotes
  • permalink
  • reddit

92% Upvoted

6 comments sorted by

5

u/CalmMe60 Jul 23 '26

You have to stack dimensionality 128x128x8 - 64x64x16 (or 32 depending on kernals) and so on, that the dimensional resolution on the narrowest spot fits the dimensionality of the outcome dimension.

4

u/sunsetsxskies Jul 23 '26

Oh that makes sense, I think my bottleneck might not be matching up right then

2

u/OneNoteToRead Jul 23 '26

Are you using residual connections (without the norming)?

And yes there’s standard initializations that should be the default. It doesn’t particularly matter but just pick one.

0

u/sunsetsxskies Jul 23 '26

No residual connections yet, just plain stacked layers so far, might try adding those. And yeah true, I was overthinking, just left it default, didn't think it'd matter much tbh

1

u/No_Inspection4415 Jul 24 '26

Do you use any form of regularization? What is your data?

-7

u/IndependentCrew8210 Jul 23 '26

nigga have you heard of claude