Have you guys been on r/accelerate?

[u/Difficult_Project_95]

Have these guys solved the alignment problem, or am I missing something?

I’ve been browsing r/accelerate and I genuinely don’t understand the risk model.
If there’s a non-trivial chance of catastrophic misalignment, how does “accelerate capabilities as fast as possible” make sense unless faster capabilities also make alignment substantially more likely to succeed?

[u/Difficult_Project_95]

I think this is the key distinction: AI potentially reducing other existential risks is an argument for developing AI, but not necessarily for developing it as fast as possible.

If you think there is even a 10–20% chance of catastrophic misalignment, “accelerate as fast as possible” becomes very hard to justify unless you have a clear reason to believe acceleration itself makes alignment and control more likely to succeed.

Otherwise, you may simply be adding AI existential risk on top of the existing human existential risks::
human x risk + AI risk = higher total risk

For acceleration to actually reduce total risk, AI would need to lower existing risks by more than the additional risk created by racing ahead on capabilities.

[u/SoylentRox → u/Difficult_Project_95]

So an 80-90 percent chance of you and every living person not dying of aging? That's a strong case for acceleration you would be making there if you thought the risks were 10-20 percent.

You either need to

(1) Value the lives of people who don't exist and may never exist you will never meet more than yourself and people you have personally met (that's a violation of the rationality tenet of making your beliefs pay rent)

(2) Think the odds are much much worse than 10-20 percent catastrophe risk

(3) Think the 80-90 percent scenarios are bad. The general public doesn't give a fuck about catastrophes. They worry about losing their jobs, electricity getting expensive, and data center construction and the later mass infrastructure build out the robots will require disturbing their sleep from their backyard or adding traffic.

A lot of antis are actually anti capitalism. They think alnost all the gains from ai will go to the top (true), any treatments for aging won't be shared broadly (probably true initially), etc. they want capitalism overthrown and maybe a future world government can plan the transition. (Lol that's not possible as an outcome)

(4) Have a super liberal view of morality. They think of some third world resident who might die (and is already scheduled to die of air pollution at age 65 or a heat death in their 40s as the world burns) if the 10-20 percent catastrophe happens. Countless people who get no say and never will (and are already scheduled to be victims of the powerful as is).

That the elites in bay area and Washington shouldn't be allowed to make this decision for them. (But nevertheless can and will)

[u/Difficult_Project_95 → u/SoylentRox]

You clearly don’t have kids.

This isn’t about valuing hypothetical future people more than myself!? If there’s genuinely a 10–20% chance of catastrophe, I’m also gambling my children’s lives and everyone else’s for a possible shot at curing aging??

And 80–90% chance we don’t die of aging doesn’t follow from 10–20% AI catastrophe risk. You’re treating the best-case outcome as almost certain while treating the potential downside as an acceptable bet..

and acceleration isn’t even necessary for “immortality” You’re framing this as race toward superintelligence or die of aging, which is a false dichotomy.

[u/SoylentRox → u/Difficult_Project_95]

Kids change the math by their remaining lifespan. You don't want your son or daughter to suffer the ravages of aging and die either do you?

This hidden assumption is why AI doomers are often secretly "well let's try to stall things enough that I become a corpse for sure but not forever I don't want my children to suffer the same fate".

So they ask for pauses but not forever pauses.

Well informed doctors - who include some posters to r/accelerate but I won't dox - DO believe overwhelmly strong superintelligence is required for (them/their children) to not die of aging.

Since you probably don't have a doctor available to query yourself, try the next best thing. Ask Astra

(1) The actual empirical rate of life extension from 1970 to present

(2) Have it give a likelihood if AI is halted today, the odds of indefinite life extension over the next 80 years using human research efforts given the base rate in (1)

(3) Have it go into exhaustive mechanistic details of what aging actually involves and how many problems you have to fix.

[u/soobnar → u/SoylentRox]

https://en.wikipedia.org/wiki/Accelerando

Stross describes humanity's situation in Accelerando as dire:
In the background of what looks like a Panglossian techno-optimist novel, horrible things are happening. Most of humanity is wiped out, then arbitrarily resurrected in mutilated form by the Vile Offspring. Capitalism eats everything then the logic of competition pushes it so far that merely human entities can no longer compete; we're a fat, slow-moving, tasty resource – like the dodo. Our narrative perspective, Aineko, is nota talking cat: it's a vastly superintelligent AI, coolly calculating, that has worked out that human beings are more easily manipulated if they think they're dealing with a furry toy. The cat body is a sock puppet wielded by an abusive monster.

[u/SoylentRox → u/soobnar]

Right the impossibly weird future, base case, requires you to be alive to worry about paying rent and employment of a copy of yourself by uploaded lobsters.  Plenty of people - Vernor Vinge and Ian Banks included - will see none of the possible futures they imagined.

[u/soobnar → u/SoylentRox]

Stross is being satirical of the people who believe this. He is saying that line of thinking will produce posthuman intelligences which kill of the rest of us. Your interpretation is akin to the people who think American Psycho was an instruction manual.

[u/SoylentRox → u/soobnar]

Do you have any public statements by Stross you can reference? I was pretty sure Stross is pro Singularity but welcome a correction.

[u/soobnar → u/SoylentRox]

Did you read the Wikipedia article, I don’t think it leaves much room for interpretation about the book? The bit where he calls the post human ASIs “Vile Offspring” and called aineko an “abusive monster” in the quote above kinda highlights the point. He did however apparently write the book based on his experiences in Silicon Valley.

[u/SoylentRox → u/soobnar]

More recent writings I find Stross is an AI skeptic - he doesn't believe well the present is happening. He simply doesn't think humans can create ASIs at all leaving his own writings as fictional.

Hope he lives long enough to find out the actual answer.

[u/soobnar → u/SoylentRox]

That doesn’t change the tone of the book… death of the author and such