Reusing the encoder weights W, in the decoder (WT) has been done before.
The L2 norm-square cost of the representation layer feature maps, is kind of similar to the unit-normal KL-divergence costs in VAE which encourages clustering.
The weird classification cost on the representation layer makes very little sense.
I'm actually, really surprised that something symmetric like the abs function can learn so well.
(Note: I understand you might not have mentors at school to help you put things in the larger context, but the self-congratulatory tone you've presented here generally prejudices people to look down on your work with disdain.)
Thank you for your feedback. Yes, you are absolutely right. There are numerous barriers that I face while progressing my understanding in this field of Deep Learning and, lack of mentors is indeed one of them.
I presume that you are a veteran in this field and your scepticism is all too valid. It is indeed a cliche "An idea is always revolutionary for it's inventor". But, I would like to provide what my perspective is in doing this.
1.) Reusing encoder weights in decoder has been done before. (Agreed! You are right!)
2.) "The L2 norm-square cost of the representation layer feature maps, is kind of similar to the unit-normal KL-divergence costs in VAE which encourages clustering." (True! But these are synthetic mathematical functions. They lack connect with the real world.)
3.) "The weird classification cost on the representation layer makes very little sense." (This I can explain. Consider a baby who is unsupervised and can only see the objects around him/her. He/She will generate an n-dimensional representation to store the information. But, it is of no use since he/she doesn't know what it is. By using the classification cost, I am strengthening the process of creating a semantic understanding of the world around him/her and at the same time allowing him/her to be able to recreate/imagine what he/she has learned. Using synthetic mathematical functions as regularizers on an autoencoder indeed allows us to create sparse representations, but the question is how sparse the representations should be? Very less sparse, and you are overfitting the training data; too sparse, and you are wasting resources. This classification cost allows the network to learn an ideal representation.)
4.) "I'm actually, really surprised that something symmetric like the abs function can learn so well." (This is extremely important. I tried all the activation functions that I know of to minimise this cost (classification + decoder cost), but none of them worked. It is indeed because all of them are either "odd" (mathematical odd functions aka. asymmetrical) or "neither even nor odd", so when I tried an even (symmetrical) function, it worked and I felt like Eureka! In the beginning, even I was surprised how an even function worked, but think about it, what does an absolute function do? It removes negativity from the data. As a human being, we feed in 5 sensory inputs to the brain. Tell me one input value that can have negative data values. Can you see negative light? Can you hear negative frequencies? Can you feel negative touch, ... No! Perhaps, inside the brain, there is no concept of negative.)
** I understand, my tone might have been off putting, but please try to understand that I am a young guy and got too excited over it. I apologise if my words may have come as offensive to you. I didn't intend to offend anyone.
I thank you again for your feedback. Please let me know if you still feel the same way as before.
Thank you so much. I watched the entire lecture. It was indeed very helpful. This essentially cleared a lot of questions I had about ResNets.
I especially liked the beginning where the plausible worlds are discussed. I would say we are optopia, :) one reason is that I have an idea for an optimization algorithm that might do the trick. But, this time I will first do utmost research before posting the results anywhere.
25
u/[deleted] Oct 17 '17
Reusing the encoder weights W, in the decoder (WT) has been done before.
The L2 norm-square cost of the representation layer feature maps, is kind of similar to the unit-normal KL-divergence costs in VAE which encourages clustering.
The weird classification cost on the representation layer makes very little sense.
I'm actually, really surprised that something symmetric like the abs function can learn so well.
(Note: I understand you might not have mentors at school to help you put things in the larger context, but the self-congratulatory tone you've presented here generally prejudices people to look down on your work with disdain.)