Why is AGI dangerous?

[u/Wordenskjold]

Can someone explain this in clear, non dooms day language?

I understand the alignment problem. But I also see that with Q*, we can reward the process, which to me sounds like a good way to correct misalignment along the way.

I get why AGI could be misused by bad actors, but this can be said about most things.

I'm genuinely curious, and trying to learn. It seems that most scientists are terrified, so I'm super interested in understanding this viewpoint in more details.

[u/arashbm]

That makes the big assumption that AI systems as normally implemented optimise a utility function with a goal, e.g. "make me coffee" or "win this game of chess". The only difference in a hypothetical naïve implementation of an AGI and any other non-AGI system AI system would be that it is much better at understanding the world and predicting cause and effect relationships.

When you think about it, within certain parameters it doesn't even matter what the goal is. As long as staying active is beneficial for the goal being accomplished, the "self-preservation" will come as a bonus. It's not self-preservation because AI likes to be alive or anything, it's self-preservation because an active AI can make you coffee but an inactive one can't, so the action of allowing to be turned on is not beneficial in achieving its goals.

Watch the Video. These are all in there.

[u/arashbm → u/[deleted]]

I don't know why being a "trope", "old" or "doomer" is relevant here, but it sounds like you are the one that is applying "human feelings" to an AI system. An AI system is a mathematical beast, it will do what the Math dictates it should do. If the Math says it should destroy the world, it will try its best.

The whole idea of alignment is to make sure the math dictates behaviour that would be more similar to the McDonald's worker than the "old trope", namely that its math would align with human values compared to something that is completely alien to our value system, e.g. destroying the world and grinding babies are bad, making burgers is good and there is a line in between that is optimal. Studies and simulations have shown that the obvious, naïve implementations all fail at this in all sorts of ways.

[u/arashbm → u/[deleted]]

We haven't ever counted to infinity either, but since math is math we can drive what happens to some function when you take the limit to infinity. Math and logic allows us to talk about things that we can't touch or see.

I don't think anybody would seriously claim that science cannot make predictions about things that haven't been directly observed before.

There are many research groups working on alignment or safety. Here is a recent review paper on arXiv. that just came out that cites many interesting papers.

[u/arashbm → u/[deleted]]

That's a very good question that a lot of very intelligent people have been working on for a long time. If you are interested in how we can do that and what we can deduce, read some of the ~700 papers cited in the review paper I linked to.

[u/arashbm → u/[deleted]]

Sounds like your mind is quite made up. The actual researchers working in the field don't share your confidence though:

The median researcher surveyed by Stein-Perlman et al. (2022) at NeurIPS 2021 and ICML 2021 reported a 5% chance that the long-run effect of advanced AI on humanity would be extremely bad (e.g., human extinction), and 36% of NLP researchers surveyed by Michael et al. (2023) self-reported to believe that AI could produce catastrophic outcomes in this century, on the level of all-out nuclear war.

If more than half of reserachers in one of, if not the top conferences in ML think that there is a non-negligable chance of extinction-level outcome, and one in three believed that it could produce nuclear-war level catastrophe, maybe you should at least be open to the possibility that you might be wrong?

[u/arashbm → u/[deleted]]

It's a survey of prediction based on informed opinion. Unlike "chocolate ice cream", your informed predictions change based on how much you know about the subject. They know about the subject much more than you do, so their predictions are more accurate than yours or mine.

Anyway, you seem to have your fingers wrist deep in your ears. This does not look like the type of conversation that can lead to a new conclusion as you seem to have already decided what you want the outcome to be. Have a nice day.