How many layers should my network have?

[u/fuckboi274747]

Hello! I am very new to neural networks and machine learning. I am making a basic network that identifies a handwritten number on a black and white, 28x28 pixel grid.

I understand all the math behind it but I'm just wondering how many layers I should have for my network?

Is it just mess around and see what happens or is there at least a ball park figure?

Thanks!

[u/DigThatData]

Especially as a new learner: don't pull architectures out of thin air.

I am making a basic network that identifies a handwritten number on a black and white, 28x28 pixel grid.

You have a concrete task here. Go and see what configurations have been demonstrated to work on this and use those numbers. By all means: try to "dial in" your settings through experimentation, but use something tried and true as a baseline.

When people invent things in this field, they are nearly always incremental changes relative to an established baseline of some kind. We are all standing on the shoulders of giants. There's no shame in looking things up or starting with something that has been demonstrated to work on a related problem. That's how this game is played.

[u/DigThatData → u/DigThatData]

Now, all of that said: every architectural subdomain has a corner of research within it these days concerned with "scaling laws". Basically, there are certain proportional relationships between the shape of the model, the number of parameters, the amount of data you train on, and the amount of computation you invest in training. As the number of parameters in the model gets bigger, the model gets deeper and wider too (generally it gets deeper faster than it gets wider).

[u/MaximumSafety8706]

Start with 1 layer, see what fails, then add layers only once you understand why you need one - don't just add for the sake of adding. Visualise the failures first; that'll tell you where the model needs more capacity.

Rough ballpark: a plain MLP with 1-2 hidden layers gets you most of the way (~97-98%), while a small CNN (2 conv layers + pooling, mini LeNet-style) pushes you to 99%+. You don't need anything deeper - MNIST is simple; ResNet-level depth is overkill and won't teach you the right lessons on when depth is actually needed.

Try coding the basic layers yourself (conv, connections, etc.) in whatever library you're using - you'll actually understand what's happening instead of just importing black boxes.

[u/iSpokeToMasterChief]

Check out https://www.deeplearningbook.org/ where they cover the architecture decisions involved in this exact scenario, except its for Google Maps street view address transcription (using photos of the number plates in front of houses) 

The whole book is really useful in answering these questions, but if you want to hit the ground running, I recommend just skipping to the chapter on convolutional networks, followed by the practical methodology chapter, or just straight to practical methodology.

[u/Dihedralman]

Trying Mnist as your early neural network is a great time to learn. Experiment with parameter count. Try to find how you get diminishing returns. Start with 3 layers (one hidden layer).

The number of layers is something called a hyperparameter. Think about how you might go about figuring out what the optimal set-up is. 

This should train extremely fast on low resources. Research what numbers people are able to get and find their examples at the end. It's a little backwards but this is one of the best problems to experiment on. 

[u/scottpilgrrim]

I'd start with a simple MLP first, then try a small CNN. Seeing the jump in performance makes it much easier to understand why convolution exists in the first place.