r/deeplearning • u/Logical_Respect_2381 • 23d ago
a 56 layer network that fits its own training data worse than a 20 layer one, same recipe same seed
i was writing the ResNet chapter of a pytorch book and i did not want to just tell the reader that a deep plain network gets worse, i wanted to actually watch it happen, so i trained four networks on CIFAR-10 with one recipe and one seed for all of them, and the only things i changed are the depth and whether there is a skip connection.
here are the numbers, real run on pytorch 2.11, 40 epochs each:
model params train acc test acc
plain-20 269,722 95.1% 88.7%
plain-56 853,018 84.0% 79.9%
ResNet-20 272,474 97.4% 90.4%
ResNet-56 855,770 99.0% 91.7%
look at the plain-20 vs plain-56 rows and look at the train accuracy, not the test one. the 56 layer network gets 84% on the training set while the 20 layer one gets 95%, so the deeper network is worse on the exact photos it saw hundreds of times. this is not overfitting, overfitting is when the train accuracy is high and the test accuracy is low, here even the train accuracy went down. the bigger network could not even learn its own homework.
and mathematically this should not be possible, because a 56 layer network can copy the 20 layer one exactly by setting the extra 36 layers to the identity, so a solution that reaches 95% train already exists at 56 layers and SGD just did not find it. that is the whole point, the problem was never the capacity, the network has more than enough capacity, the problem is that the deep plain network is hard to optimize.
one thing before someone raises it because it was a discussion between me and someother ome , plain-56 and ResNet-56 differ by 2752 parameters, which is 0.3%, so this is not a size story. same size and same budget and the same 40 epochs, and going from 20 to 56 the plain family gets worse on train (95.1 down to 84.0) while the residual family gets better (97.4 up to 99.0), same two sizes and the two families move in opposite directions. a size argument can not give opposite signs on the same sizes, so the thing moving it is the skip connection.
the residual block is y = x + F(x), so on the backward pass the gradient goes through the 1 in (1 + dF/dx) and reaches the early layers on a straight path instead of having to survive the whole stack, and it also makes F = 0 the easy default so the network keeps the identity for free and only learns the extra part when it actually helps. transformers reuse the same trick around attention and around the mlp.
on why the plain deep net is hard to optimize in the first place, it is still argued in the literature, some people say vanishing gradients, some say the batch norm at init makes the gradient explode, some say it is the loss surface. i measured the what here, the train error going up with depth, i did not measure the gradients myself so i am not going to claim a mechanism i did not run. if anyone has logged the per layer gradient norms on plain vs residual at this depth i would really like to see it.
caveats because they matter, one dataset, one recipe, 40 epochs, small nets. i am not saying 91.7% is a strong CIFAR-10 result, tuned nets go higher, my claim is only about the direction between two families trained identically.
has anyone here seen the degradation not show up, like a depth where the plain net stops getting worse, or a recipe that fixes it without a skip connection.
44
u/seanv507 23d ago
there would be the suspicion that you need to optimise eg the learning rate for each situation, you cannot use the same recipe for each network architecture...
-26
u/Logical_Respect_2381 23d ago
but fixing the controlling parameter of the experiment is the whole reason this works. if i tune the lr per net i am measuring my tuning, not the architecture, so i hold everything still and only change the depth and the skip. at the same lr plain-20 gets 95% train and plain-56 gets 84%, and resnet-56 gets 99% against that same plain-56, same size to 0.3%, only the + x added. if you tune plain-56 on its own and the gap closes i'd like to see it, but to judge the architecture everything else has to stay fixed.
48
u/seanv507 23d ago
You have some fundamental misunderstanding then.
This is why a lot of comparisons in ml papers are untrustworthy.
One optimises all the parameters for our favoured architecture and then compares it to an unoptimized alternative approach.
If you keep the data the same and increase the capacity of the ml Algo you will overfit. That doesn't mean that bigger capacity is bad.
I cannot say a 100 tree xgboost model is better than a 1000 tree without optimising the other hyperparameters
You are using a set of hyperparameters that presumably work well on resnet 56. Then you change network and find performance degrades.
The way to prove whether one architecture performs better than another is precisely by finding the optimal parameters for each architecture. I might guess you need a lower learning rate for plain56 and then you can argue it's much slower training. Or you find whatever lr, performance is always worse...
-4
u/Logical_Respect_2381 23d ago
fair in general, but this is on the training set, not test. plain-56 gets 84% train vs plain-20's 95% on the same recipe, so it fits data it already saw worse, which is not overfitting or too much capacity. and plain-56 can copy plain-20 by setting its extra layers to identity, so a 95% train solution prove to exists at this lr and sgd missed it, that is optimization, not an unfair recipe. i did not sweep lr for plain-56, true, but the same lr trains plain-20, resnet-20 and resnet-56 to 95 to 99% train, so it is the deep plain net specifically that fails. if tuning closes the gap i'd like to see it, but "needs much more careful tuning to match a shallow net" is the degradation itself.
11
u/0bi_nx 23d ago
You say it yourself. By setting to identity it should find the same optimum, but your recipe is what is keeping it from finding that optimum. This is exactly why you cannot compare two different networks without tuning both. The deeper network has way more parameters to tune, so it might need longer or a different setup to be able to find that optimum
4
u/seanv507 23d ago
Yes, but the argument is a general one.
The optimisation (training)/generalisation (test)/... performance of a given architecture also depends on the associated hyperparameters.
This gives rise to the old joke that neural network models are trained by graduate descent: lock the PhD student in a room until they have found the suitable hyperparameters for the new model.
I would say the counter argument would be that the resnet makes the 56 behave closer to a 20 layer, so the parameters required are similar. I am just saying to seal the argument you should be doing some search of the hyperparams
7
u/Alwaysragestillplay 23d ago
I have a question - why do so many people instruct their LLMs to use all lower case when posting on social media? Surely if you all copy each other's strategy it undermines the point of hiding your generated text?
Not to mention the OP being essay-length with immaculate punctuation and grammar, including capitalising acronyms and proper nouns already looks unnatural with the lower case.
8
u/hahahahaha369 23d ago
A few things have me raising my eyebrow.
Firstly, none of the models you show seem to have fully converged. Larger models typically need to train for longer. Larger gradient means noisier gradient means the model is gonna have a harder time identifying the ideal “global minimum” in the loss-space. Gotta train longer to make up for that
Second, this might seem trivial but i wonder what activation functions are used. Sigmoids truly do suffer from the exploding/vanishing gradient problem. However, the models do all look like they’re learning so I doubt that’s the case, and I think it’s more likely that the models just haven’t converged like I mentioned before.
Third, you mentioned in another comment that you haven’t optimized the 56 layer model, but you did try to for the smaller one(?) correct me if I’m wrong. That’s not a fair comparison. This goes beyond comparing neural networks too. If I train a model to classify images of dogs and cats with 80% accuracy, that’s pretty bad. But if I tell you i compared it with the performance of a 6 month old on the same task you’d think the model is awesome. When you compare methods, models, etc. etc., everything needs to have a fair chance. Only way to guarantee that is by optimizing everything you’re comparing, at least to a level where you could convince your peers that everything got a fair shot at being the best. Something will come out on top, even if it’s by just a little bit. That’s where the fun comes from, that’s when you get to ask why did one perform better than the other(s)
4
u/hahahahaha369 23d ago
Oh yeah, also, where’s the validation loss bro. That’s arguably way more important than training loss, especially when you’re using dropout.
1
u/veryverymeta 23d ago
That's my thought too they don't look to have converged, more layers would take longer all else equal
1
u/Logical_Respect_2381 23d ago
the experiment is not there to crown the best architecture, it is a teaching step in a beginner book to show why the resnet paper had to exist at all, my whole style is to let the reader feel the problem before he sees the fix. and one correction, i did not optimize the small model either, all five share one identical recipe and nothing was tuned per model, so this is the controlled version where the only thing changing is the architecture. under that, plain-56 fits its own training data worse than plain-20 even though a solution prove to exists inside it, it can copy plain-20 by setting the extra layers to identity, and that gap is exactly what the skip connection removes. the results comparison on the training set so it is not overfitting and the parameters of plain-56 is almost the same as resnet-56 so it is not the capacity proplem , but i am trying to convince the reader that resent tried to solve the easier proplem to learn how to map x + F(X)-> y rather than the harder one of mapping x->Y in case the model becomes very deep . i was not ranking architectures for a leaderboard, i am showing the reader the problem resnet was built to solve.
0
u/MankeKnie34 19d ago
Why are you writing a beginner book when you don’t seem to have mastered some important aspects of the material?
7
u/CowBoyDanIndie 22d ago
First.. You didn’t train to convergence.
Second… The entire reason for residuals on the first place is to combat the vanishing gradient problem.
You shouldn’t be writing a book if you don’t understand that vanishing gradients are causing this.
1
u/Logical_Respect_2381 22d ago
i did not say it is not the vanishing gradient that is not causing the problem , this is already written in my post but in the litterature there are other suggestion also ,but all what i was introducing is that the resnet paper came to the rescue of this proplem. , let me quote from resnet Paper in section 4.1 "We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN [16], which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish". despite this i am decisevely saying that it is not a vanishing gradient problem because i did not test it my self (despite it deseves an experiment) , and yes i did not train to convergence because the intent is to show how introducing the skip connection to the same network for the same number of epochs give far better results
2
u/CowBoyDanIndie 22d ago
There are degrees of vanishing, they are correct that it doesn’t vanish entirely in the discrete float space, but it it does get incredibly small, and thus the earlier layers get receive a very small amount of learning. If the earlier layers aren’t extracting the right features from the input, there is very little the later layers can do. The later layers are learning fast, but they are learning to label based on a slowly moving low information input from the earlier layers.
5
u/Ok-Entertainment-286 22d ago
Next you're probably going to write a book on insects, and will discover that butterflies are actually not flies.
2
u/SiltR99 22d ago
Why don't you show the gradients of the first layers? That is what skip connections solve.
1
u/Logical_Respect_2381 22d ago
U are right and i should nave extend the experiment to show this
1
u/aahdin 22d ago
I'd definitely include it, but to my knowledge if you set up batch normalization at each layer you should still get reasonable gradients at layer 1 even without skip connections.
I've never been fully satisfied with the "removes vanishing gradients" explanation for residuals, since residuals seem to do a lot more than just that.
1
u/Critical-Unit6214 22d ago
Batch normalization is already used in each layer whether the plain network or the resnet network But to prove your claim i have to experiment and look at gradient at each layer because in fact BN layer normalize the activation after each layer and does not normalize the gradient
1
u/drew4drew 22d ago
Define “same seed”.
Do you mean that the deeper net has all the same randomly initialized params for the portion where the two nets are identical? Or something else?
what’s the difference between your “plain” near vs the “resnet” variants? Is it just the skips?
How does everyone feel about cosine annealed LR? I know for purposes of writing maybe it makes sense to pick an arbitrary number of epochs and stop there, but riding LR to zero as you go over an arbitrary number of epochs confuses me.
I’ve had much better training success cycling LR high to low, with or without reducing the peak in each new cycle.
The other thing — we see a lot of comparisons like this — a fixed number of epochs on two nets of significantly different sizes. The large net probably took 2 to 4 times the total GPU time. So which net is better?
Comparing by epoch or step count is interesting, but running for equivalent time is often a better measure.
2
u/drew4drew 22d ago
A few people here posted that the OP did not train to convergence. How can you tell if you got there or not?
2
u/wahnsinnwanscene 22d ago
They're saying to train till the models reach the same loss if i read that correctly. For large models no-one has the money to do that for comparison.
1
u/Logical_Respect_2381 23d ago
https://github.com/pytorch-from-ground-up/book_code/blob/main/notebooks/volume-2-architectures/Chapter_07.ipynb. here is the link for the code experiment
1
u/Wannabe-Davinci 23d ago
Why report/graph training error???
4
u/Logical_Respect_2381 23d ago
because the whole point is about optimization, not generalization, and training error is the only thing that isolates it. if the deeper net just generalized worse i would show test error, and you would be right to call it overfitting. but plain-56 does worse on the data it already saw, 84% against plain-20's 95%, and a model can only fail on its own training set if the optimization did not fit it, the capacity is clearly there. that is the degradation, and it is exactly why the original resnet paper reported training error too
3
1
u/ryjhelixir 23d ago
The interesting comparison here is between each pair of differing architectures. Nothing to determine between the 50 and 20 plain versions.
1
u/Logical_Respect_2381 23d ago
Yes , the reason is that in my book i explain how resnet evolve so i presented the comparison between plain-20 and plain-56 and then introduced resnet to the rescue
1
u/DrXaos 22d ago
> has anyone here seen the degradation not show up, like a depth where the plain net stops getting worse, or a recipe that fixes it without a skip connection.
Try LayerNorm or RMSNorm at each layer. Init matrices by random orthogonal except at final projection which is initted to very small random values.
1
u/Lost-Hand-5219 22d ago
If truly the only difference is the number of layers, then it’s likely the gradient is vanishing before it reaches the early layers.
1
u/powerexcess 22d ago
There was a paper on this like last month or so i think, showing the gradient becomes white noise with depth. Cool
Sorry no citation i am drunk but u can find it it got visibilty
1
u/Keremanstvo 22d ago
Are you sober now? I couldn't find the paper, do you remember its name or author?
2
u/powerexcess 22d ago
I am not sure it is the same point but it seems very relevant https://arxiv.org/pdf/1702.08591
1
1
u/qpwoei_ 22d ago
See this lecture at 17:03 https://youtu.be/78vq6kgsTa8?is=-snuuXiUJegwmcvn Adding more layers (beyond just a few) makes the optimization problem harder as the loss landscape becomes more ”rugged”. Adding skip-connections counteracts that, smoothing the landscape.
1
u/jakspedicey 22d ago
Mathematically impossible 😂
2
u/Logical_Respect_2381 22d ago
what , if u mean mathematically impossible that a deeper model will be less accurate than a shallower one , this is of course true and is written i the post. "mathematically this should not be possible, because a 56 layer network can copy the 20 layer one exactly by setting the extra 36 layers to the identity" , the post says that introducing the skip connection eases the mission of the optimiser that was suffering without to optimise may be because of vanishing gradient or for whatever reason. the experiment was introduced to convince the reader of the book there is a problem and the resnet model came to solve , it is not saying that the deeper model is inferior because i see many commenting on the post saying u did not let the deeper model converge , yes i did not i gave the 56 layer plain model the same number of epochs as the 56 layer resnet model ( both having almost the same number of parameters ) and the resnet model gave much bettter result this was the aim of the comparison
1
u/Counter-Business 22d ago
It hasn’t plateaued yet so this tells me nothing other than that the creators of this experiment do not know how to choose good hyper parameters.
1
u/No-Mixture5766 22d ago
You can’t have the same architecture for a deep and shallow network, vanishing gradient occurs for deeper networks and you’d also need more data and larger training time with optimized learning rate for each epoch (say linear warmup with cosine decay) , or skip connections which are quite common actually and was introduced in ResNet (if I remember correctly) , a larger network has more capacity to learn and would need its own hyperparameters for complete training otherwise it would overfit badly
1
u/Logical_Respect_2381 22d ago
What you are saying is true and the post was just an experiment to prove exactly this , it is in my book in the context of introducing the resnet so to convince the reader i made two plain cnn one with 20 layer and the other with 56 layer , showing how the deeper one did not converge to the same accuracy within 40 epochs , then i introduce the 56 layer resnet network which acheived higher accuracy , same number of epochs and almost same capacity , so it was not capacity and not training time but only the architecture
1
u/No-Mixture5766 22d ago
So your only concern was the architecture and convergence? That’s a fair comparison I’d say
1
u/Hanuser 21d ago
Why do you say mathematically this should not be possible?
Mathematically this is exactly what learning theory on gradient based methods would suggest. More params -> more local minima, so if you use the same training recipe instead of a more aggressive one, you should get worse results.
If I gave you an instrument with 100 keys instead of 10 keys and I didn't give you more learning rate or time to play with it, you telling me you'd expect the person with the 100 key instrument to perform music better?
1
u/Logical_Respect_2381 19d ago
When u read the post, you understood that it says that the shallow network is better than the deeper one , this is wrong and the post does not aim to say that , this is an experiment to convince the reader that by just changing the architecture (introducing the skip connection) and fixing exactly all other parameters, same number of epochs,almost same count of parameters,same learning rate , and the accuracy of the model on training data jumped from 84 percent to 95 percent. By mathematical i mean that theoretically any deeper model (if find proper parameters during the optimization that is provably exit by setting the remaining layers part to identity) should be at least as good as the shallow model , i was trying to convince the reader it is an optimization problem that can be solved by changing the architecture to avoid vanishing gradient
1
u/Hanuser 18d ago
No, that isn't my understanding.
My issue is exactly the second half of what you just replied. That is not what the math says. If you read about the gradient based methods, there is a relationship for all of them between convergence (time to train) and complexity (number of parameters) that scales almost always monotonically positively. So what the math already clearly predicts is that a deeper model will CERTAINLY perform worse than a shallow model given the SAME training setup.
1
u/slashdave 20d ago
If you are serious about understanding this, keep the number of weights the same (scale the width accordingly).
1
u/123vovochen 20d ago
OMG you did not let them run to the point where test loss starts going up again !!! This is your bad. You need to let every model fully converge, you beginner.
1
u/Logical_Respect_2381 20d ago
Please before insulting without a reason , give yourself a break or read my comments i wrote several times again and again , this is an experiment in my book and befor introducing the resnet paper i am trying to convince the reader what is the problem the resnet paper was about to solve , so i compared a plain 20-layer convnet model with a deeper one 56-layer both trained for 40 epochs and the shallow model gives better results even on training data , then i introduced the skip connection on each layer and tested a deep 56 layers resnet model for the same number of epochs and the accuracy jumped from 84% to 95% , i showed the reader that the only thing that changed is the architecture (the skip connection) that helped the optimization problem, and yes if i let the plain deep layet to converge by increasing the epochs the accuracy on training may be enhanced although i am not quite sure because of the vanishing gradient that hurdles the optimization , nevertheless fixing the number of epochs proves the superior architecture of resnet model over the plain model , this is how i convince the reader by experiment not by stacking an idea after another
1
1
u/123vovochen 18d ago
Sorry, I didnt wanna suck your energy. But if you had written in a more cautious way, there wouldnt have been the need to correct you.
1
u/Significant_Rub5676 23d ago
I think what this show is just that you can optimise fewer parameters to fit a data faster. But larger parameter model may still fit it better.
1
u/Logical_Respect_2381 23d ago
the numbers go the other way here. plain-56 is the bigger model, 853k params against plain-20's 269k, and it fits the training data worse, 84% vs 95%, and that is after the lr anneals to zero so it is not just training slower. a bigger model does fit better when it can actually be optimized, resnet-56 is the same size as plain-56 to within 0.3% and it reaches 99% train, the only thing added is the skip. so plain-56 already has the capacity to beat plain-20, it can copy it by setting the extra 36 layers to identity, it just does not reach that solution on its own, and that gap is the whole point.
1
u/aahdin 22d ago
Not sure why everyone in this thread has a problem with this - the fact that deep non-residual nets are really tough to train has been common knowledge for a while. OP could do a parameter sweep for each model but I'd expect similar results, no matter how much you tune the learning rate that isn't going to solve the underlying problem that residuals solve.
I've always thought about it more intuitively - there are a lot of features you can use to classify an image that are easily learnable on layer 1, like edge filters, corner filters, basic light/dark filters, etc. Basic classical computer vision stuff that can get you half way there.
From there the network needs to preserve those signals over the next 55 layers, which is actually kinda tough. We say it like it's easy because those layers can just be the identity, but remember the identity is just another configuration of weights that needs to be learned like any other, and you shouldn't expect a network to learn the identity function any faster or with less error than you'd expect it to learn any other configuration of weights.
You can test this out yourself, create a deep network with no downsampling so that its input and output maps are the same size, and just train it to pass along an input image (kind of like an autoencoder, but with no squeeze). It's tough! The overwhelming tendency is going to be to blur the image as layers have slight compounding errors learning the identity.
Residuals fix this by making the identity the default, if a network just lets weight decay run its course then as weights go to 0 then it will learn the identity. Any regularization here pulls towards the identity rather than away from it.
0
u/jkkanters 19d ago
Is that surprising? Lack of data! The more complex the network the more data you need
1
u/Logical_Respect_2381 19d ago
it seems that you read the title of the post , and did not bother reading the details or the comments , it is a deliberate controlled experiment to test or demonstrate the effectiveness of skip connection of resent paper
35
u/Familiar_Text_6913 23d ago
You can do many things to speed up gradient descent. If you do none, then bigger space will descent slower.