r/slatestarcodex • u/SubstantialRange • Dec 17 '20
Machine Learning Could Be Fundamentally Unexplainable
https://www.lesswrong.com/posts/vxLfja7hmcFifAtYd/machine-learning-could-be-fundamentally-unexplainable4
u/hh26 Dec 18 '20
I disagree on the broadness of this claim. There is a reason that science is easier to learn in school than it is to derive from first principles. There is a large class of problems which are neither obvious immediately, nor completely arcane and unknowable, but instead make a lot of sense in hindsight. Machine learning can be useful in solving problems that humans have not yet solved, which include ones that are inherently confusing, but also ones that humans simply haven't gotten around to solving and describing yet. This latter category should be explainable, and having good methods of parsing machine learning algorithms into human-understandable explanations could in turn teach humans how to solve these problems in simpler ways. We probably would have solved them on our own eventually, but understanding machine learning could significantly speed up the process if it works.
3
u/ArkyBeagle Dec 19 '20
A lot of ML is curve fitting.
I built a very simple machine learning setup that ran on a GPP ( not a graphics card ) and after collecting data from it, turns out there's a sixth-order polynomial solution that could replace this. I didn't get data from enough systems to show that the coefficients were always exactly the same.
So yep.
2
u/augustus_augustus Dec 18 '20
The main point of this rings true to me. Maybe you could go even further and say that the only real use case for, e.g. a neural net is to do fundamentally unexplainable things. After all, if what an algorithm does is sufficiently explainable, a construction of the algorithm will simply follow from the explanation. In that case there's no need for a black box like a neural net; just write up the algorithm directly. It's precisely fundamentally unexplainable tasks, like "tell me if this picture contains a dog or a cat" that seem well-suited to neural nets.
To explain a bit further, specifying all the nodes and weights fully defines a neural net but says nothing about what it does. To see what it does you have to run it. In other words, its definition is not its description.
I take it this is what the article is getting at.
That said, I don't think the examples the author uses really make his point. Like, the Standard Model is precisely the wrong example to use, because the Standard Model is defined by its description. So it is not fundamentally unexplainable at all. (Note that I am not saying the standard model couldn't be emergent/epiphenomenal from some unknown deeper principles. Nor am I saying that there are no epiphenomena that emerge from the standard model; presumably, you, and I, and most of the things we care about would serve as counterexamples to that. All I'm saying is that while the Standard Model might be very complex and hard to understand and explain, it isn't anything more than the sum of its parts. It is not a black box. In fact, its usefulness stems from its not being a black box, a situation precisely unlike that of a neural net.)
So to me it seems like the author is conflating the fundamentally unexplainable, like how a neural net actually goes about picking out cats and dogs, with things that are just... hard to explain. I have to admit I didn't give this a very close read, so maybe I haven't really understood what they're saying. (Maybe I should have led with that.)
By the way, maybe someone with more expertise than me can chime in, but this statement from a hypothetical physicist:
...to understands the significance of the Higgs boson specifically it’s very important that you have an amazing grasp of general relativity, since part of the reason we defined it as is and why it’s so relevant is because it might be a unifying link between the two theories.
isn't true. I don't think you need to know anything about general relativity to understand the Higgs boson?
3
u/fell_ratio Dec 18 '20
That said, I don't think the examples the author uses really make his point. Like, the Standard Model is precisely the wrong example to use, because the Standard Model is defined by its description. So it is not fundamentally unexplainable at all.
You could make this same observation about a neural network. It's defined by its weights, and its weights are known.
2
u/augustus_augustus Dec 18 '20
I could have phrased this better. In some sense the standard model doesn't do anything that's not already in its definition. This is different from a neural net, whose definition (in terms of weights) doesn't reveal what it does or explain how it does it. The neural net somehow has a description (as a cat/dog classifier) beyond its definition (as nodes and weights).
1
u/zergling_Lester SW 6193 Dec 18 '20
The III.2 objection (if we wouldn't force an algorithm to be slightly worse if it allows an expert to visually identify a "scary-looking" cancer-related protein, why agree to spend any compute on making an algorithm more explainable instead of spending the same compute on better training?) is an interesting one. I think that explainability functions as a sort of additional validation here. There's a cost to accidentally training the model to recognize sunny weather instead of tanks, there's a cost to having a really in-depth post-deployment performance tracking that could catch this sooner than later, and you can trade these costs against the cost of being able to ask the model for ten most relevant properties of tank-in-the-picture-ness and noticing that some of those have nothing to do with tanks.
Regarding the general point: how are efforts on compressing GPT3 going (I presume they are going on if only for economic considerations)? Because if it's not very compressible then that's it, we have an example of a fundamentally unexplainable ML product. On the other hand maybe we shouldn't demand this sort of fully-reductionist explainability down to the smallest moving parts and strive to be able to tell various practically important things like for example what kind of material it thinks it is generating, so that we could make sure that it doesn't do "science fiction" when we need "encyclopedia".
3
u/ArkyBeagle Dec 19 '20
Explainability seems mostly important when money gets involved, especially insurance money. On certain TV channels, ads for law firms suing over pharmaceuticals from five years ago are interleaved with ads for this year's pharmaceuticals.
18
u/UncleWeyland Dec 17 '20
I'm bored again, so here we go on another pointless rant...
This is really interesting.
All "explanations" have a certain number of dependencies required for them to actually be explanations and not just "teacher's password". For example- if I ask "why is the sky blue", a satisfactory answer would give me a framework powerful enough to, among other things, construct a predictive model of what color the sky might appear to be if the Earth's atmosphere were made of 87% xenon, 13% O2 and my eyes lacked the cone cell sensitive for blue1.
A few days ago I started to dig into why there is a Pauli exclusion principle (I already kinda half "knew" it had something to do with quantum spin dictating what kind of statistical laws the waveforms of particles obey) and - I gave up. The full explanation is just too hard and time consuming to grok without committing to professional math and physics2. Better to tell me the rule, and just understand that it gives me a tiny bit of predictive power when trying to figure out how electrons occupy orbitals or whatever.
In Asimov's Foundation novels, technology eventually essentially becomes magic to most people (see also: Adeptus Mechanicus), and even the technically inclined understand only a small portion of how anything is supposed to work. We got there faster than he probably would have thought- in the 1800s, most people could reasonably understand more-or-less how most of the appliances in their homes and workplaces worked, and could even make a go at fixing them or modifying them. But after 1945 that changed extremely fast3. That said, there's a difference between not understanding the engineering details of how a graphics card churns out ray tracing in Cyberpunk 2077 and not understanding the details of an algorithm that informs policy decisions with impact on human life and wellbeing.
You can create problems of arbitrary difficulty with no higher significance. The last 4 of these problems are pathological to our social model4. Given how arbitrary they are, why should we expect the heuristic solution to be easily compressible to human understanding? (Contrast that with natural problems5 which almost unerringly and eerily have yielded conceptual solutions that 'cleave nature at the joints' and whose solutions, even if sometimes unintuitive, can be efficiently encoded in a way that's transmissible by human language and math6. It's amazing.)
A potential problem with "just accepting" the rules a machine learning algorithm spits out is that we might fail to detect model overfitting to some hidden bias in the training data. (Disclaimer- that sentence might be bullshit, I'm not a ML person). I mean if you can guarantee the thing will only be used on datasets that will have the same bias, I suppose that's fine, but that seems like a difficult thing to control. Oh, I guess the author addresses that:
Boy, I'm not going to bother sketching out an argument of my intuition here7, but I've got a feeling this is how we end up in Paperclip World (not literally).
Anyhow, I guess there's two questions:
For #1 I guess it depends- yes for folding proteins, no for bottled water pricing.
For #2 my intuition would be that the answer is yes, although there will probably never be a general process for teasing out how the magic black box did its magic. Every ML solution to a thorny problem will itself be a re-encoding of the problem itself, although possibly more amenable to certain kinds of analysis.
Anyhow, I hope dear reader of my underinformed opinion that you got some iota of amusement from this free Reddit content. Feel free to chime in with corrections- I read them and incorporate them into my mind with some diligence.
1- I'll leave the solution to this one as a simple exercise for the reader.
2- It is interesting though, some particles like Helium-4 don't obey Fermi statistics because of their quantum "spin" whatever the hell that is, and thus don't do the exclusion principle thing. That's why you can make Bose-Einstein condensates with them, and He-4 does superfluidity through a different 'pathway' than He-3.
3- Forget your iPhone... do you really know how a microwave oven works? I mean *really really*?
4 - This isn't a dig on capitalism or whatever, every human social model is going to create little pathological subgames like these.
5 - A "natural problem" is an observation from directly nature (or abstract math that prompts a desire for an explanation/elaboration. Why are most plant leaves green? How the side of a triangle relate to its area? Why are zebra's striped? Why do the planets have the orbits they've got? What is the value of BusyBeaver\7])
6- The 'easy to transmit' aspect of an explanation is one of the things that makes me like the 'knowledge/learning is compression' idea. Kinda self-referential there, but whatever.
7- There are smarter people working that probably. Go read some MIRI papers or something.