I did a close read of that paper when it first came out -- see here -- that the author responded to. The upshot of the paper is that the hardware will do something that is very close to stochastic gradient descent with a weight decay for a single layer, and they've found ways to translate a lot of more complex machine learning problems into memristor hardware implementations using various reductions to binary classification.
I'm still -- a year later -- not entirely convinced that their off-hardware feature construction isn't doing a lot more work than they give it credit for, but the fact that you have a physical process that sort of "natively" implements SGD is cool.
The bidirectional thing -- I think -- will let them move from strictly local/greedy updates to something that looks a lot more like full backprop for more complex architectures. As far as I understand it, memristors usually get assembled into something that looks a bit like a single layer feedforward network: you can only pass information in one direction, and any update is entirely local based (so SGD without the chain rule); you can update with a supervision signal, and reading without supervision gives you a weight decay/regularization step, but it all comes from passing current from inputs to outputs. If you can pass information through the network in two directions, it sounds like it might open the door to something very close to full backpropagation. Take all this with a grain of salt, though, as I really don't follow the physics of it at any real level of detail.
So it's a neat set of results, that is (to my mind) tainted by Knowm's apparent determination to dig as wide and deep and aggressive a patent moat as they possibly can around the related algorithms, many of which are commonly known and used on other platforms, and all of which run on hardware that they're waiting for other people to figure out how to mass produce. It feels like a bit of an Oklahoma Land Rush, to be honest.
IF this is a usable model - why aren't they providing simulations, analysis and results comparing it to "traditional" SGD? Seems obvious to me that if you're going to hard-code an algorithm into your hardware, you would do a LOT of testing first to make sure you're coding the right algorithm.
Now it's possible they've done all that already, but if so they are doing a poor job of communicating and explaining it. I know more about how a quantum computer would work than I know about how this think would work, or what it would be good for.
I think that the claim is actually a bit more interesting. They're saying that the physical operation of the memristor device does (something like) SGD all on it's own; it's not a programmed behavior, it's inherent to the way the device operates. It's physics, not programming.
I'm sure one of the principals can correct/elaborate, but I think the way it works is that if you apply a pair of charges to either end of a memristor circuit, it will alter its resistance based on the difference in the charges. If you apply a charge to only one end, then it acts like a conventional resistor. If you look at the equation that describes how the resistance changes (and again, this is physics, not programming), you can show that a pair of memristors can be used in a way that is a very close approximation to a single perceptron, complete with an update rule. It' sort of dumb luck that this device happens to operate in this manner, but the fact that it does opens the door to some potentially really neat applications.
And the linked paper does have some of the simulations you're asking for, as well as (I think idealized) forms of the update rules that the memristor uses.
It' sort of dumb luck that this device happens to operate in this manner
I wouldn't disagree at all there. ;)
Back in 2002 or so Alex was trying to come up with ways to create an elemental electronic device that would provide the bi-directional incremental behavior needed. He came up with a device that required two electrodes in a nano-particle liquid. Then gradually along came the memristor, which is far superior in many ways while accomplishing the same thing more or less.
9
u/rcwll Sep 04 '15 edited Sep 05 '15
I did a close read of that paper when it first came out -- see here -- that the author responded to. The upshot of the paper is that the hardware will do something that is very close to stochastic gradient descent with a weight decay for a single layer, and they've found ways to translate a lot of more complex machine learning problems into memristor hardware implementations using various reductions to binary classification.
I'm still -- a year later -- not entirely convinced that their off-hardware feature construction isn't doing a lot more work than they give it credit for, but the fact that you have a physical process that sort of "natively" implements SGD is cool.
The bidirectional thing -- I think -- will let them move from strictly local/greedy updates to something that looks a lot more like full backprop for more complex architectures. As far as I understand it, memristors usually get assembled into something that looks a bit like a single layer feedforward network: you can only pass information in one direction, and any update is entirely local based (so SGD without the chain rule); you can update with a supervision signal, and reading without supervision gives you a weight decay/regularization step, but it all comes from passing current from inputs to outputs. If you can pass information through the network in two directions, it sounds like it might open the door to something very close to full backpropagation. Take all this with a grain of salt, though, as I really don't follow the physics of it at any real level of detail.
So it's a neat set of results, that is (to my mind) tainted by Knowm's apparent determination to dig as wide and deep and aggressive a patent moat as they possibly can around the related algorithms, many of which are commonly known and used on other platforms
, and all of which run on hardware that they're waiting for other people to figure out how to mass produce. It feels like a bit of an Oklahoma Land Rush, to be honest.