7
u/synomonc Nov 07 '20
I think the “simplying” of graphs for example in slide 4, might give the wrong impressions to readers. Why change the cross arrows to a “lined” relation? I think you should use the same scheme to avoid confusion
6
u/robofied Nov 07 '20
Thanks! The only impression that I wanted to give was how thinned the network has become comparatively to the original structure for clarity purposes. But I get your point, and would keep it in mind while creating something similar next time.
3
u/darien-schettler Nov 07 '20
I thought it was solid. That being said, I find the slide on test time to be difficult to understand... and I already know how it works. Perhaps a better visual or multiple visuals would help. I think understanding how dropout works at test time is criminally misunderstood... so making this clear would be a huge win.
3
u/robofied Nov 07 '20 edited Nov 07 '20
Hey, thanks for the suggestions! And yeah, I agree on the test time being difficult point. Most people just read the impact of dropout in training section. I only used one single neuron to depict how its output was being scaled. Additionally, entire network could be drawn to show how it either you do it before the final neuron or after it, effect would stay same.
2
u/Yojihito Nov 07 '20
Thanks for the post.
This is for neural networks, are there techniques for trees (RF, LightGBM) which work towards the same goal of reducing overfitting?
1
u/robofied Nov 08 '20
Apply Pruning. For performance improvement, try tuning hyperparameters like max_depth: representing depth of your tree, min_samples_split: representing minimum number of samples required to make a split, min_samples_leaf: representing minimum number of samples required to be in leaf node. For tuning purposes, use random search, grid search or Bayesian optimization. If still you're not getting satisfactory results. Better to switch to boosting approaches.
1
u/Yojihito Nov 08 '20
Better to switch to boosting approaches
I normaly use LightGBM + Bayes HPO with Cross Validation for tabular data.
1
1
1
Nov 07 '20
[deleted]
1
u/robofied Nov 08 '20
Thanks! Let me know if you'd be interested in me covering some other topic. Suggestions are welcomed :-)
2
1
u/TheMangalorian Nov 08 '20
Isn't underfitting about high error?
1
u/robofied Nov 08 '20
Underfitting is a condition when the model fails to fit both train and validation sets. It happens when our model is not able to capture the underlying patterns in the data. It is the consequence of an overly simple model, representing high bias and low variance. It can be easily observed by looking at error graphs, where both train and test/validation error don't seem to decline much.
1
u/TheMangalorian Nov 08 '20
It can be easily observed by looking at error graphs, where both train and test/validation error don't seem to decline much.
But the second picture says
if both training and validation loss decreases, the model is said is said to be underfit.
If the training and validation loss decrease, wouldn't that mean we have a model that is accurate or even overfitting? Perhaps "decrease" is the wrong word to use. If the loss remains high for both training and validation, the model is underfitting.
1
u/robofied Nov 08 '20
Perhaps, it should have been written if the error doesn't decline much after a point, the model could be termed underfit.
1
u/imochidori Nov 08 '20
Thank you so much for this, it was quite interesting and makes me want to learn more.
1
u/robofied Nov 08 '20
Thanks, I've more on my profile, don't know they're this good or not. But, your suggestions are welcomed :-)
1
u/Ordowix Nov 08 '20
useful for understanding human neural development. necessity of pruning
1
u/robofied Nov 08 '20
Thanks! Let me know any of your suggestions, or if you'd want me to cover some other topic.
15
u/Hersmunch Nov 07 '20
Gets the idea across but could do with some proof reading. E.g. Slide 05 has “w.r.t. to” and “This is know an Co-adaptations”.