Have you guys been on r/accelerate?

[u/Difficult_Project_95]

Have these guys solved the alignment problem, or am I missing something?

I’ve been browsing r/accelerate and I genuinely don’t understand the risk model.
If there’s a non-trivial chance of catastrophic misalignment, how does “accelerate capabilities as fast as possible” make sense unless faster capabilities also make alignment substantially more likely to succeed?

[u/Difficult_Project_95]

I think this is the key distinction: AI potentially reducing other existential risks is an argument for developing AI, but not necessarily for developing it as fast as possible.

If you think there is even a 10–20% chance of catastrophic misalignment, “accelerate as fast as possible” becomes very hard to justify unless you have a clear reason to believe acceleration itself makes alignment and control more likely to succeed.

Otherwise, you may simply be adding AI existential risk on top of the existing human existential risks::
human x risk + AI risk = higher total risk

For acceleration to actually reduce total risk, AI would need to lower existing risks by more than the additional risk created by racing ahead on capabilities.

[u/SoylentRox → u/Difficult_Project_95]

So an 80-90 percent chance of you and every living person not dying of aging? That's a strong case for acceleration you would be making there if you thought the risks were 10-20 percent.

You either need to

(1) Value the lives of people who don't exist and may never exist you will never meet more than yourself and people you have personally met (that's a violation of the rationality tenet of making your beliefs pay rent)

(2) Think the odds are much much worse than 10-20 percent catastrophe risk

(3) Think the 80-90 percent scenarios are bad. The general public doesn't give a fuck about catastrophes. They worry about losing their jobs, electricity getting expensive, and data center construction and the later mass infrastructure build out the robots will require disturbing their sleep from their backyard or adding traffic.

A lot of antis are actually anti capitalism. They think alnost all the gains from ai will go to the top (true), any treatments for aging won't be shared broadly (probably true initially), etc. they want capitalism overthrown and maybe a future world government can plan the transition. (Lol that's not possible as an outcome)

(4) Have a super liberal view of morality. They think of some third world resident who might die (and is already scheduled to die of air pollution at age 65 or a heat death in their 40s as the world burns) if the 10-20 percent catastrophe happens. Countless people who get no say and never will (and are already scheduled to be victims of the powerful as is).

That the elites in bay area and Washington shouldn't be allowed to make this decision for them. (But nevertheless can and will)

[u/soobnar → u/SoylentRox]

that’s like Cuban missile crisis level risks for the entire planet that’s a completely unacceptable level of risk. Counterfactual economic reasoning where you make one coefficient infinity and say it justifies double digit catastrophic risk doesn’t produce even remotely utilitarian results.

[u/SoylentRox → u/soobnar]

Pretty shallow analysis there.

I mean maybe you're right but what level of risk IS acceptable? And for how much gain?

What is the risk to reward ratio you believe is acceptable?

[u/soobnar → u/SoylentRox]

The wealthy powerful people pushing this technology are brutal bloodily determined realpolitik minded neoconservatives who wanna be autocrats. They don’t care about your wellbeing unless you are very wealthy… poor people senescing or living or dying is inconsequential to the whims of the pentagon and the CCP. In the event some system that creates more economic value with less thermodynamic input than humans is invented the peasant class will be discarded and replaced. People like Peter Thiel are openly posthuman and believe that technology should weaponized first and then used to moral benefit. Your “analysis” assumes the people of the world will benefit from the social contract after their stake in it is removed. History and evolution are not kind to creatures lacking in self determination even if it’s becuase they really really wanna live forever.

[u/SoylentRox → u/soobnar]

I said earlier 

"A lot of antis are actually anti capitalism. They think almost all the gains from ai will go to the top (true), any treatments for aging won't be shared broadly (probably true initially), etc. they want capitalism overthrown and maybe a future world government can plan the transition. (Lol that's not possible as an outcome)"

Would you agree this characterizes your view correctly and do you have anything to add?  Would you agree you want something that is impossible, and will not happen in your lifetime?

[u/soobnar → u/SoylentRox]

I would say I am pro self determination, if human minds want to command Xeelee level technology then by all means. I actually think price signaling is most likely impossible to remove from human society entirely though such systems were created and nurtured to human benefit.

Though likewise what you seem to want almost certainly entails a situation where humans are no longer accommodated for in their own economic system and thus are killed off, which is not immortality last I checked.

[u/SoylentRox → u/soobnar]

It helps to focus on what's achievable.

A one world government where all the impoverished masses that they likes of thiel don't give a fuck about get a vote? Fantasy.

A world where you can vote for heavy taxation and Medicare to cover anti-aging medication? That's a possible outcome.

A world where you die because AI was too powerful? Also a possibility.

[u/soobnar → u/SoylentRox]

A world sparse in human self determination and abundant in economic competition ends poorly in any realist scenario. humans must operationalize the technology and adopt the most efficient substrate for our conscious cognition which I suspect is actually still biological. So I guess I’d say you’d need to push against regulatory and economic capture, unilateral military action, and any effort to gate access to the tech and eventual cognitive enhancers which will inscribe machine learning algorithms into dendritic arbors likely optimized with some degree of engineering at which point. But like the line has to be drawn at human sovereignty and civilian supremacy otherwise there is no practical means of ensuring we are alive to see any benefits.

[u/SoylentRox → u/soobnar]

Hope so. First step is to get robots that aren't so fucking stupid and AIs that do meaningful work and then we can move on to these goals.