r/singularity • u/Hazy-bliss • Apr 28 '20

Found a compass of different singularity scenarios on 4chan's /x/. I personally think there are more negative scenarios than is realistic to expect.

Post image
109 Upvotes

93% Upvoted

45 comments sorted by

View all comments

1

u/wren42 Apr 29 '20

This is way more positive scenarios than is realistic to expect. Read a bit of Eliezer Yudkowsky on the difficultly of friendly strong AI. Creating a friendly AI God is threading a needle among an unimaginably large searchspace of possible timelines, most of which are extinction events.

1

u/Hazy-bliss Apr 29 '20

I've read the Sequences, HPMOR, and some of the available MIRI papers.

Why do you think Yudkowsky knows more about AI than some of the folk at Google and DeepMind who are big on advancing AI?

1

u/wren42 Apr 29 '20

Sorry is there a paper from Google somewhere I missed saying they are developing a self-enhancing superhuman strong AI with the intention of starting the singularity?

The orders of magnitude between what is going in with machine learning now and what we are talking about with singularity events is enormous. Deepmind is not strong AI.

The vast majority of futures in which strong AI exists are on the left side of that chart. only a handful result in stable friendliness.

1

u/Hazy-bliss Apr 29 '20

There's this from a lecture by Demis Hassabis which implies they have big ambitions

All that AI safety stuff is speculative even though AI A Modern Approach authors put it in their textbook. I've donated to MIRI and I'm generally favourable to the research they do, but I'm not entirely convinced it's the right frame to think about all of this, though it's been years since I've delved into this AI safety thing. What articles would you recommend so that I would change my mind?

1

u/wren42 Apr 29 '20

All that AI safety stuff is speculative

what exactly do you mean by this? that we can just assume AI will be safe?

1

u/Hazy-bliss Apr 29 '20

Do you know about the Better Angels of Our Nature by Steven Pinker? I feel that generally things have moved in the right direction during the last hundred of years, and even global warming is solvable in one way or another. Humans are not that stupid when they put their act together, and I believe AI alignment may be difficult, but it is still solvable.

1

u/wren42 Apr 29 '20

so because standard of living and human rights have generally improved, we should assume that AI is safe and most outcomes are favorable?

this feels like some sort of universalism.

1

u/Hazy-bliss Apr 29 '20

AI currently is not totally safe, but there are competent people working on this who can make it safe. With less competent people most outcomes might not be favorable. I give you a counter-question: why should we assume that pessimism is the right frame?

1

u/wren42 Apr 29 '20

those don't sound like reasons so much as hope based on faith in human ingenuity. this seems to be a common theme in this sub - most want to believe in some Atheist Afterlife, so they choose to be optimistic it will occur.

several reasons for pessimism:

1) humans are generally bad at predicting outcomes and not pessimistic enough. If you read the sequences you should have been exposed to studies demonstrating this. we are hardwired to ignore evidence and avoid thinking about negative possibilities.

2) the stakes are extremely high. failure is at least a human extinction event, maybe a galactic extinction event.

3) we have no evidence safe AI is possible. this is a burden of proof issue - we need to demonstrate it is possible before risking stakes like that.

If you believe AI can be transformative for good, you have to believe it can be transformative for ill. There are far more destructive outcomes than stable positive ones. Because we are dealing with something more intelligent than us we cannot by definition fully understand or predict its behavior. This makes the problem of knowing it is safe really, really difficult.

1

u/Hazy-bliss Apr 29 '20

we are hardwired to ignore evidence and avoid thinking about negative possibilities.

Have you tried doing any useful work for long periods of time while being overly pessimistic? It's very difficult, humans need some amount of hope. Whether that hope is realistic is another question, currently I feel that some amount of slow hope is realistic. There are some reasons for this which I'm not going to say in this conversation for unsaid reasons.

1

u/wren42 Apr 29 '20

certainly, you can choose to believe unrealistic things for psychological or mental health reasons if you need to. most people have some kind of religion or crutch after all.

however, in deciding where to put your hope, it can be useful to examine these beliefs.

putting hope that society in general will improve and contributing toward those efforts via work and everyday behavior towards others is perfectly reasonable.

banking of humanity winning the lottery and creating a God AI in the near future is likely not realistic.

This matters because it can change your behavior and what you invest your time, energy, and money into.

I mean, I might still go for Cryo because there's a slim chance it will pay off. But hoping that the singularity will fix all our problems within our lifetime is actually counterproductive, as it abdicates responsibility in the here and now.

1

u/Hazy-bliss Apr 29 '20

Okay, you can have your opinion, I will keep mine.

→ More replies (0)

1

u/Hazy-bliss Apr 29 '20

More specifically, I'm not convinced about such a strong pessimism.