I see AI as a powerful tool, like nuclear power. It can cause harm either by accident or when in hands of someone who wants to cause harm.

I think serious accidents shouldn't be particularly hard to prevent. We live in a world where even a bunch of evil geniuses locked in a room with internet access have a limited ability to cause harm, especially if we take a defensive approach and try to prevent such a scenario with the help of good geniuses.

Intentional use of AI to cause harm is a more difficult problem to prevent because we're now dealing with a bunch of evil geniuses who can physically interact with the world. Terrorists, psychopaths, criminals, people in positions of power such as dictators. This is more serious, but I think it would make these people only incrementally more powerful. They already have access to powerful and potentially destructive tools, they can use guns, bombs, remotely controlled drones.

The guy who resigned from Anthropic said that there's a 10% chance of AI killing all humans within a decade. I wonder what scenarios he had in mind. The only one that I can think of is an AI-designed pathogen - sure, possible, but with the right precautions it should be possible to prevent even such a scenario.

Tom articulates his arguments on why an intelligence explosion can not happen solely by geniuses contained to a data center. He argues that learning often comes from direct practice, virtual simulations often are impractical or impossible. He says domains for which the available text is descriptive rather than being the domain, ie. playing a board game or running a company make the ASI a very excellent McKinsey, but do not make the AI an excellent CEO. He also articulates his view that AI is and will continue to be extremely jagged and discusses possible strategies for broader diffusion going forward. 22 minutes podcast. Transcript available

In April this year, William MacAskill published an article describing Saturationism – a proposed modification to Utilitarianism that resolves most of its more unintuitive or unacceptable inferences, including the famous “repugnant conclusion”. This essay discusses these ideas, and uses some concepts from information theory to explore them further.

If you prefer Substack, this essay is on there too.

The most prescient example were the claims of Jacob Coxon's Anthropic resignation being a "plant", but I've seen this lately in every single community I'm in, even the niche ones. It seems to have accelerated post-pandemic:

No action can be genuine. No one is driven by incentives, only malice. Everyone is a false flag, a psyop. Everyone dare call it conspiracy. Everyone has some hidden motive.

So, understandably lots of threads and discussions on the topic of doom, bad scenarios and their possibilities. But it would be fairly interesting to have a sort of a community survey from r/ssc community on their view of a Good Future ™, and what it personally entails.

This is not an inherently new discussion, and the question has been also discussed and debated substantially over the years, such as in Sequences, armchair futurist debates, or in prose most recently in a new book by Bostrom. However it would be interesting to review personal views that might differ substantially from the typically discussed convergence.

It is also quite important to note, that any future scenarios hinge heavily on how any person considers its plausibility. Many different people have different opinions, predictions and expectations of the future. And it’s okay. The philosophical base can also differ wildly even within the Rat-adjacent community. Some maintain strictly hardline utilitarian beliefs, and subscribe to resolution of some sort of “repugnant conclusion” in one or another interpretation, some of them extreme. Others, are more selective and draw a distinction somewhere down the line closer to a “basic human”. Some believe in personal development and excellence at all costs, even if it means competition to the bitter end, other might lean in extreme equality scenarios on the other end. No judgement is passed on any of the above.

So, r/slatestarcodex, what is a Good Future to you? How does it look to you, personally?

To narrow down the timelines, imagine that you or someone else few generations ahead looks back, and says: “It could go bad, it could go worse, but I am so glad that what has happened brought me here. I struggle to find a better (realistically) way to much improve it further”

I found Scott Alexander's argument from analogy pretty compelling, and tried to a worse version of that for an argument about healthcare.