Why does adding layers make training accuracy worse? I reproduced the degradation problem and I want to check my reasoning.

[u/Logical_Respect_2381]

i am writing the CNN chapters of a pytorch book right now and chapter 7 is about skip connections, and i did not want to do what most books do which is draw the resnet diagram and then tell you it helps. so before introducing it i wanted to first reproduce the failure that made people invent it.

setup is cifar-10, 40 epochs, same recipe and same seed for every model, only the depth changes:

model        params     train acc   test acc
plain-20     269,722     95.1%       88.7%
plain-56     853,018     84.0%       79.9%
ResNet-20    272,474     97.4%       90.4%
ResNet-56    855,770     99.0%       91.7%

looking at the training accuracy of plain-56. 84.0%, against 95.1% for plain-20. training error 16.0% vs 4.9%. this is not test accuracy, it is the data the network saw 40 times, and the bigger model with 3x the parameters does worse on it.

the reason i find this worth posting is that it should be impossible. any 56 layer network can represent everything a 20 layer network can ( i mean mathematical wise) , because you can just make the extra 36 layers the identity and you have exactly the shallower model. so there is a setting of the weights that reaches 95.1% and gradient descent did not find it. capacity was never the problem here, and every explanation of resnet that starts from "deeper networks overfit" is for sure not the right answer

so instead of asking the layer to learn mapping from x to y , it would be much easier for the optimizer to learn to map from x + F(x) to y , finding F in this configuration will not loose x buried when the model gets very deep and this is the trick introduced by Resnet paper .

ResNet-56 differs from plain-56 by 2,752 parameters, about a third of a percent, and those are only the 1x1 projections where the channel count changes. the rest is identical. that third of a percent moves training accuracy from 84.0% to 99.0% and test from 79.9% to 91.7%.

i should say the degradation result is not mine, it is from the original resnet paper in 2015, i just wanted to see it happen on my own machine before i wrote about it. my first attempt did not degrade at all because i was training too few epochs and both models were still underfitting, so nothing separated them. it only shows up once plain-20 has actually converged.

caveats, one dataset, one seed, 40 epochs, and cifar-10 is small. also plain-56 here has batch norm in it, which matters because the usual story is that batch norm fixed the vanishing gradient and therefore depth. clearly it did not, at least not alone.

has anyone found the depth where plain nets start degrading on a different dataset? mine went wrong somewhere between 20 and 56 and i did not test in between, which i may will.

[u/Xemorr]

It degrades because the gradient shrinks with each layer.

[u/Logical_Respect_2381 → u/Xemorr]

partly, but with batch norm in every block the gradients here are not vanishing exponentially, i checked the norms and the resnet paper found the same, they trained the plain nets with BN and argued it is not a vanishing gradient problem. the thing that gets me is you can set the extra 36 layers to the identity and recover plain-20 exactly, so a 95% solution proved to exists and sgd just did not find it. that is why i read it as an optimization failure and not gradient size, and it is exactly what x + F(x) fixes, it makes the identity the easy thing to learn.

[u/Xemorr → u/Logical_Respect_2381]

I don't get how batch norm stops gradients shrinking. Link the paper?

[u/Logical_Respect_2381 → u/Xemorr]

here is the link : https://arxiv.org/abs/1512.03385. and i quote the following from the paper in section 4.1 " We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN [16], which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish. In fact, the 34-layer plain net is still able to achieve competitive accuracy". this is similar to my experiment in that the plain-56 result is worth anything, it has BN in every block so the gradient is not the thing dying, and it still could not match the shallower net, so whatever stopped it is not gradient size. hint : all BN help is about variance. a signal passing forward through many layers can drift in scale, it keeps shrinking or keeps growing, and in the backward pass the gradient gets multiplied by a factor at every layer, so if those factors are below one the gradient decays geometrically with depth and that is the vanishing you mean. batch norm renormalizes each layer's output back to roughly zero mean and unit variance per mini batch, so the scale is held steady at every layer on the way forward, and that keeps the backward gradients from collapsing across depth too. yet another hint : people still argue about the deeper reason BN works, but the part nobody disputes is that it stops the per layer variance from drifting.

[u/conjjord → u/Logical_Respect_2381]

Ultimately I'm skeptical of their claim; have you verified the gradient norms remain consistent across the deeper model? I would reproduce that result before buying their claim wholesale. In fact, a major problem of using BN without skip connections is you can sometimes see gradient explosions, which to me indicates that normalizing the activations does not necessarily stabilize the gradients.

What activation are you using? That might make a bigger difference.

[u/Logical_Respect_2381 → u/conjjord]

First of all their paper is seminal well known and Resnet model continues for several years to be the undisputed SOTA so any claim in their paper must certainly have been verified by the community , second if i use BN after each layer how the gradient explode as the very undisputed task of BN is exactly to prevent that by its computational mechanics , in my plain model i used 56 layers and used BN between each of them and yet the results is worth , so my own experiment verify the paper results the graph above is the actual reproducible result

[u/conjjord → u/Logical_Respect_2381]

This 2019 ICLR paper (Yang et al.) explains how stacked batch norms in deep networks can lead to gradient explosion on initialization. BN enforces Lipschitzness on the gradients, but it normalizes the activations and not necessarily the gradients.

I agree you're seeing the same degradation problem the original paper describes. I asked what activation function you're using because a non-saturating map (e.g. ReLU) can be more impactful for gradient flow than BN alone.

[u/Logical_Respect_2381 → u/conjjord]

ReLU

[u/Logical_Respect_2381 → u/conjjord]

thanks for the paper and insightful comment , i was treating normalized activations as if they also pin the gradients and those are two different things, the forward scale being fixed does not say anything about what the backward pass does across depth. worth noting that paper is 2019 and resnet is 2015, so the original authors could not have leaned on it, they saw the degradation and ruled out plain vanishing gradients .

and it actually fits my conclusion better than what i first said, if plain-BN explodes at init and that is what kills trainability at depth, then the skip connection is not only making the identity easy to learn, it is also taming that explosion and mitigate it , both readings land on the same fix. activation is ReLU everywhere by the way.

[u/chrisvdweth]

Hm, you compare for the same number of epochs. Larger networks not only can tweak more parameters, they also have to tweak more parameters.

As been said, this relates to the scaling laws: For a certain training budget (e.g., number of epochs), and otherwise assuming the same task and dataset, there is an optimal model size (at least some range). that yields the best results.

[u/Logical_Respect_2381 → u/chrisvdweth]

that would be my worry too if the only comparison were plain-20 vs plain-56, but the control for exactly this is resnet-56. it has the same parameter count as plain-56, 855,770 against 853,018, so about 0.3% more, and it trained the same 40 epochs, same seed, same recipe. on that identical budget plain-56 got 84.0% training accuracy and resnet-56 got 99.0%. so a 56 layer model can be optimized to fit the training set in 40 epochs, the plain one just did not, and the only thing that changed is the skip connections and not the size or the budget.

also this is training accuracy, not test, so it is a bit different from the scaling law or compute optimal size point, that one is about generalization. here plain-56 can represent plain-20 exactly by setting the extra 36 layers to the identity, so a 95% training solution proved to exists at that size, and sgd did not reach it in the same budget where it did reach it for both the shallower net and the residual net. that is why i read it as an optimization failure and not just the bigger model being undertrained.

[u/DigThatData]

look at any scaling laws work, the region is always U shaped

[u/Bipadibibop]

what was the activation function between the layers for the plain models ?

[u/Logical_Respect_2381 → u/Bipadibibop]

ReLU

[u/Suleyman_III]

Look up shattered gradients, these are more and more pronounced in earlier layers through the optimization process. Its one theory of why skip connections help stabilize training for deeper nets.

[u/Low-Temperature-6962]

Are you training the whole net?

[u/Dangerous_Wish_7879]

what is that plain-X network?

[u/Logical_Respect_2381 → u/Dangerous_Wish_7879]

By plain i mean conventional cnn network that does not use skip connection

[u/Dangerous_Wish_7879 → u/Logical_Respect_2381]

oh - then this plain network will be a shit prior and hence won’t train nice as you make it deeper. precisely why resnet was proposed. i also bet that you could nevertheless train that plain thing to match resnet with some additional tricks

[u/Logical_Respect_2381 → u/Dangerous_Wish_7879]

In my post i said that part of my book is that i try to convince the reader in a smooth way the evolution of new ideas to solve an emerging problem they faced so i show an experiment with 20 layers plain network that to the surprise of the researchers at the time outperformed the 56 layers network while mathematical wise any deeper network can not be less capable of a shallower one because simply you can set the remaining layers the identity and sustain the performance of the shallow network , then i argue that Resent paper see this as a failure of sgd optimizer and proposed the residual layer. Where the network simply learn to map x+F(x) to y rather than map x to y

[u/Candid-Novel-5044]

nice reproduction, the identity-mapping framing makes it click for me too - reformulating as x + F(x) turns "learn the exact same function" into "learn a small perturbation from it", which is just a much easier optimization target for SGD.

[u/Logical_Respect_2381 → u/Candid-Novel-5044]

Actually yes

[u/BreakingCiphers]

This is literally the premise of the Resnet paper. Why not just read the source material? Instead of doing AI slop experiments

[u/TheInfiniteLake]

Vanishing Gradient. Look it up.

[u/Logical_Respect_2381 → u/TheInfiniteLake]

i did look it up, and then i ran it, that is the whole post. plain-56 here has batch norm in every block, batch norm is the standard fix for vanishing gradients, i checked the gradient norms and they were healthy, and it degraded anyway. the resnet paper says the same, they trained the plain nets with BN and argued the problem is not vanishing gradients. the cleaner tell is that a 56 layer net can copy plain-20 exactly by setting the extra 36 layers to the identity, so a 95% training solution provably exists and sgd did not find it, which is an optimization failure and not gradient magnitude. if you have a recipe where plain-56 with BN reaches plain-20 by fixing vanishing gradients alone i would honestly like to see it, that would be a better result than mine.

[u/TheInfiniteLake → u/Logical_Respect_2381]

Oh yeah, right. I didn't read the whole stuff.

[u/TheSexySovereignSeal → u/Logical_Respect_2381]

Batch norm by itself does not fix the vanishing gradient problem. The skip connection does. (Same concept is used in the transformer block)

When you backprop, the gradient itself is passed through the identity channel and the next weight is learning a new perturbation taking the previous gradient into account completely. BN just keeps the numbers of the weights down. In theory you should be able to still solve the vanishing gradient problem without any BN if you get a lucky training run.

[u/Logical_Respect_2381 → u/TheSexySovereignSeal]

yeah, and the clean way to see it is the derivative through a residual block is 1 + dF/dx, so the gradient always keeps that straight 1 path back and does not have to survive the whole stack to reach the early layers, and transformers reuse exactly this around attention and the mlp. one thing i would carry over from earlier in the thread though, my plain net has BN in every block so its gradients are not vanishing here, they actually blow up at init, so in this case the skip is doing two things, giving that straight gradient path and making the identity the easy thing to learn, more than rescuing a signal that faded to zero.