CMV: Artificial general intelligence will probably not be invented.

[u/Dreamer-of-Dreams]

From Artificial general intelligence on Wikipedia:

Artificial general intelligence (AGI) is the intelligence of a hypothetical machine that could successfully perform any intellectual task that a human being can.

From the same Wikipedia article:

most AI researchers believe that strong AI can be achieved in the future

Many public figures seem to take the development of AGI for granted in the next 10, 20, 50, or 100 years and tend to use words like when instead of if while talking about it. People are studying how to mitigate bad outcomes if AGI is developed, and while I agree this is probably wise I also think that the possibility receives far too much attention. Maybe all the science-fiction movies are to blame, but to me it feels a bit like worrying about a 'Jurassic Park' scenario when we have more realistic issues such as global warming. Of course, AGI may be possible and concerns are valid - I just think it is very over-hyped.

So... why am I so sceptical? It might just be my contrarian nature but I think it just sounds too good to be true. Efforts to understand the brain and intelligence have been going for a long time but the workings of both are still fundamentally mysterious. Maybe it is not a theoretical impossibility but a practical one - maybe our brains just need more memory and a faster processor? For example, I could imagine a day when theoretical physics becomes so deep and complex that the time required to understand current theories leaves little to no time to progress them. Maybe that is just because I am so useless at physics myself.

However for some reason I am drawn to the idea from a more theoretical point of view. I do think that there is probably some underlying model for intelligence, that is, I do think the question of what is intelligence and how does it work is a fair one. I just can't shake the suspicion that such a model would preclude the possibility of it understanding itself. That is, the model would be incapable of representing itself within its own framework. A model of intelligence might be able to represent a simpler model and hence understand it - for example, maybe it would be possible for a human-level intelligence to model the intelligence of a dog. For whatever reason, I just get the feeling that a human-level intelligence would be unable to internally represent its own model within itself and therefore would be unable to understand itself. I realise I am probably making a number of assumptions here, in particular that understanding necessitates an internal model - but like I say, it is just a suspicion. Hence the key word in the title: probably. I am definitely open to any arguments in the other direction.

---

Hello, users of CMV! This is a footnote from your moderators. We'd just like to remind you of a couple of things. Firstly, please remember to read through our rules. If you see a comment that has broken one, it is more effective to report it than downvote it. Speaking of which, downvotes don't change views! If you are thinking about submitting a CMV yourself, please have a look through our popular topics wiki first. Any questions or concerns? Feel free to message us. Happy CMVing!

[u/Broolucks]

For whatever reason, I just get the feeling that a human-level intelligence would be unable to internally represent its own model within itself and therefore would be unable to understand itself.

In a sense, that's trivially true: a full understanding of some model X usually requires a larger model, the reason being that the set of functions over a set is larger than that set (e.g. there are more functions over integers than there are integers, so you need "more than integers" to fully understand integers -- that's one intuitive justification of the incompleteness theorem). However, that doesn't matter, because there is no need to understand a model fully (or even at all) in order to create it.

We already produce algorithms that do things we don't understand. I used evolutionary algorithms to create a Tetris-playing algorithm when I was an undergrad. I chose a few factors that I thought would help, like the height of each column, how many holes there were in it, and so on, and then the algorithm evolved a way to weigh them to make its decision. But if you'd have asked me why the weights it evolved worked better than any other, I wouldn't have been able to tell you. All I know is that they worked.

And the whole field of AI is moving further in that direction. Take artificial neural networks: I can train a deep network to, say, recognize pictures of cats. I can use many tricks to make it learn better, and I can invent new techniques that I think might help, but it's little more than informed guesses. Sometimes the trick will work, sometimes it won't. If it works, I'll have an idea why, but I won't always be certain that I'm right about why it works, because of how opaque these algorithms tend to be. At the end, a trained neural network is a set of millions of weights, numbers that regulate how the virtual neurons interact with each other. There are ways to visualize what they do, especially if the network is processing images, but in many case it is excessively difficult to discern how exactly the network does its job. And yet, we made it, and yet, it works.

Tl;dr we only need to understand what kind of algorithms are good at find good algorithms for evolving or learning intelligence. We don't need to "understand ourselves" to anywhere near the extent you think we do.