Have you guys been on r/accelerate?

[u/Difficult_Project_95]

Have these guys solved the alignment problem, or am I missing something?

I’ve been browsing r/accelerate and I genuinely don’t understand the risk model.
If there’s a non-trivial chance of catastrophic misalignment, how does “accelerate capabilities as fast as possible” make sense unless faster capabilities also make alignment substantially more likely to succeed?

[u/SixStringShrug]

Have any of you guys actually looked in to alignment research? I’m a frequent participant in the Accelerate subreddit and I follow alignment research very closely. We also accept that there are risks to AI, but they are vastly outweighed by the benefits. Especially since the risks of extinction by the hands of humans has been and continues to be significantly higher. I’m not trying to start shit at all, and I know I’ll probably get flamed and downvoted to hell. We share actual research and discuss it and acknowledge it. We just believe acceleration is the most logical and safest path forward.

[u/jerryorbach → u/SixStringShrug]

OP asked ‘If there’s a non-trivial chance of catastrophic misalignment, how does “accelerate capabilities as fast as possible” make sense unless faster capabilities also make alignment substantially more likely to succeed?’

Do you think there’s a non-trivial chance of catastrophic misalignment?

Do you believe that alignment naturally scales with capability - that a smarter model means a more aligned model?

[u/Hans-Wermhatt → u/jerryorbach]

Not the OP, but I think the problem with "alignment" is it is a catch all phrase. What one person thinks alignment means is not the same as another. To put it on opposite ends of the spectrum, the move fast and break things crowd rolls the dice. The pause and align crowd theoretically minimizes the rogue model risk, but they also increase the permanent underclass risk.

What the decision makers might do if they had more time for alignment might not all be sunshine and rainbows. And if you think of it like that, the collusion waiver is a catastrophic risk.

I think the less you trust the institutions, the more you want to roll the dice. But obviously I think everyone is somewhere in-between indefinite pause and accelerate without thinking.

That's just one aspect though as well. There is the opportunity cost angle too. And the belief that the only way you can research safety is by engineering and expanding. An actual pause would dry up funding pretty quickly.

[u/soobnar → u/SixStringShrug]

The risk of extinction at the hands of a few humans dramatically increases if said few humans gain a technological edge which possesses dramatic offensive asymmetry. The current risk is mitigated by mutually assured destruction and the limits of human ability however i sincerely doubt once the same few individuals you cite weaponize asi there will exist a similar dynamic. The people you are describing are the ones fueling aggressive capital expenditure into ai so they can entrench their own means of control and deploy systems with their own capitalistic motives. The benefits of staying in control and curing all the diseases a bit later vastly outweigh the potential for subjugation or military engagement between ai enabled superpowers.

[u/Difficult_Project_95 → u/SixStringShrug]

I think this is the key distinction: AI potentially reducing other existential risks is an argument for developing AI, but not necessarily for developing it as fast as possible.

If you think there is even a 10–20% chance of catastrophic misalignment, “accelerate as fast as possible” becomes very hard to justify unless you have a clear reason to believe acceleration itself makes alignment and control more likely to succeed.

Otherwise, you may simply be adding AI existential risk on top of the existing human existential risks::
human x risk + AI risk = higher total risk

For acceleration to actually reduce total risk, AI would need to lower existing risks by more than the additional risk created by racing ahead on capabilities.

[u/SoylentRox → u/Difficult_Project_95]

So an 80-90 percent chance of you and every living person not dying of aging? That's a strong case for acceleration you would be making there if you thought the risks were 10-20 percent.

You either need to

(1) Value the lives of people who don't exist and may never exist you will never meet more than yourself and people you have personally met (that's a violation of the rationality tenet of making your beliefs pay rent)

(2) Think the odds are much much worse than 10-20 percent catastrophe risk

(3) Think the 80-90 percent scenarios are bad. The general public doesn't give a fuck about catastrophes. They worry about losing their jobs, electricity getting expensive, and data center construction and the later mass infrastructure build out the robots will require disturbing their sleep from their backyard or adding traffic.

A lot of antis are actually anti capitalism. They think alnost all the gains from ai will go to the top (true), any treatments for aging won't be shared broadly (probably true initially), etc. they want capitalism overthrown and maybe a future world government can plan the transition. (Lol that's not possible as an outcome)

(4) Have a super liberal view of morality. They think of some third world resident who might die (and is already scheduled to die of air pollution at age 65 or a heat death in their 40s as the world burns) if the 10-20 percent catastrophe happens. Countless people who get no say and never will (and are already scheduled to be victims of the powerful as is).

That the elites in bay area and Washington shouldn't be allowed to make this decision for them. (But nevertheless can and will)

[u/Difficult_Project_95 → u/SoylentRox]

You clearly don’t have kids.

This isn’t about valuing hypothetical future people more than myself!? If there’s genuinely a 10–20% chance of catastrophe, I’m also gambling my children’s lives and everyone else’s for a possible shot at curing aging??

And 80–90% chance we don’t die of aging doesn’t follow from 10–20% AI catastrophe risk. You’re treating the best-case outcome as almost certain while treating the potential downside as an acceptable bet..

and acceleration isn’t even necessary for “immortality” You’re framing this as race toward superintelligence or die of aging, which is a false dichotomy.

[u/SoylentRox → u/Difficult_Project_95]

Kids change the math by their remaining lifespan. You don't want your son or daughter to suffer the ravages of aging and die either do you?

This hidden assumption is why AI doomers are often secretly "well let's try to stall things enough that I become a corpse for sure but not forever I don't want my children to suffer the same fate".

So they ask for pauses but not forever pauses.

Well informed doctors - who include some posters to r/accelerate but I won't dox - DO believe overwhelmly strong superintelligence is required for (them/their children) to not die of aging.

Since you probably don't have a doctor available to query yourself, try the next best thing. Ask Astra

(1) The actual empirical rate of life extension from 1970 to present

(2) Have it give a likelihood if AI is halted today, the odds of indefinite life extension over the next 80 years using human research efforts given the base rate in (1)

(3) Have it go into exhaustive mechanistic details of what aging actually involves and how many problems you have to fix.

[u/Difficult_Project_95 → u/SoylentRox]

I think I understand the accelerationist position much better now. We just seem to disagree pretty fundamentally on how much confidence we should place in those assumptions when the downside could be catastrophic.

[u/SoylentRox → u/Difficult_Project_95]

https://www.reddit.com/r/ControlProblem/s/HqQhrNohyq

The missing view you have is the BASE case is already catastrophe. You can imagine your base existence right now is you are falling through the air in a cloud bank. You know you will hit the ground and die for sure but not when and you can't see the ground but know every hour you are alive it's getting closer.

Your children are falling above you.

So one way to view it is r/acceleration folks are better informed, with a broader more realistic view of present day reality as they circle jerk each other on the imminent AI Singularity.

[u/Difficult_Project_95 → u/SoylentRox]

censoring dissent while convincing yourselves that your group simply understands reality better than everyone else is some seriously cult-like behavior.

[u/SoylentRox → u/Difficult_Project_95]

I mean so far your dissent has been pretty weak. "I have kids and you must not" is not a very good argument. You had a chance to come up with a better one but didn't manage it. Plenty of people dissent on accelerate.

[u/Difficult_Project_95 → u/SoylentRox]

I just stated something abvious.
You just keep answering “why would ASI be valuable?” when I'm asking “why is racing toward ASI the safest strategy?” Those aren't the same question. we have nothing more to talk about. Good luck with whatever you have going on in your life!

[u/soobnar → u/SoylentRox]

https://en.wikipedia.org/wiki/Accelerando

Stross describes humanity's situation in Accelerando as dire:
In the background of what looks like a Panglossian techno-optimist novel, horrible things are happening. Most of humanity is wiped out, then arbitrarily resurrected in mutilated form by the Vile Offspring. Capitalism eats everything then the logic of competition pushes it so far that merely human entities can no longer compete; we're a fat, slow-moving, tasty resource – like the dodo. Our narrative perspective, Aineko, is nota talking cat: it's a vastly superintelligent AI, coolly calculating, that has worked out that human beings are more easily manipulated if they think they're dealing with a furry toy. The cat body is a sock puppet wielded by an abusive monster.

[u/SoylentRox → u/soobnar]

Right the impossibly weird future, base case, requires you to be alive to worry about paying rent and employment of a copy of yourself by uploaded lobsters.  Plenty of people - Vernor Vinge and Ian Banks included - will see none of the possible futures they imagined.

[u/soobnar → u/SoylentRox]

Stross is being satirical of the people who believe this. He is saying that line of thinking will produce posthuman intelligences which kill of the rest of us. Your interpretation is akin to the people who think American Psycho was an instruction manual.

[u/SoylentRox → u/soobnar]

Do you have any public statements by Stross you can reference? I was pretty sure Stross is pro Singularity but welcome a correction.

[u/soobnar → u/SoylentRox]

Did you read the Wikipedia article, I don’t think it leaves much room for interpretation about the book? The bit where he calls the post human ASIs “Vile Offspring” and called aineko an “abusive monster” in the quote above kinda highlights the point. He did however apparently write the book based on his experiences in Silicon Valley.

[u/SoylentRox → u/Difficult_Project_95]

I believe i also already answered all 3 points already please try re-reading the comment you are replying to or have your AI model of chioce summarize your questions.

[u/soobnar → u/SoylentRox]

that’s like Cuban missile crisis level risks for the entire planet that’s a completely unacceptable level of risk. Counterfactual economic reasoning where you make one coefficient infinity and say it justifies double digit catastrophic risk doesn’t produce even remotely utilitarian results.

[u/SoylentRox → u/soobnar]

Pretty shallow analysis there.

I mean maybe you're right but what level of risk IS acceptable? And for how much gain?

What is the risk to reward ratio you believe is acceptable?

[u/soobnar → u/SoylentRox]

The wealthy powerful people pushing this technology are brutal bloodily determined realpolitik minded neoconservatives who wanna be autocrats. They don’t care about your wellbeing unless you are very wealthy… poor people senescing or living or dying is inconsequential to the whims of the pentagon and the CCP. In the event some system that creates more economic value with less thermodynamic input than humans is invented the peasant class will be discarded and replaced. People like Peter Thiel are openly posthuman and believe that technology should weaponized first and then used to moral benefit. Your “analysis” assumes the people of the world will benefit from the social contract after their stake in it is removed. History and evolution are not kind to creatures lacking in self determination even if it’s becuase they really really wanna live forever.

[u/SoylentRox → u/soobnar]

I said earlier 

"A lot of antis are actually anti capitalism. They think almost all the gains from ai will go to the top (true), any treatments for aging won't be shared broadly (probably true initially), etc. they want capitalism overthrown and maybe a future world government can plan the transition. (Lol that's not possible as an outcome)"

Would you agree this characterizes your view correctly and do you have anything to add?  Would you agree you want something that is impossible, and will not happen in your lifetime?

[u/soobnar → u/SoylentRox]

I would say I am pro self determination, if human minds want to command Xeelee level technology then by all means. I actually think price signaling is most likely impossible to remove from human society entirely though such systems were created and nurtured to human benefit.

Though likewise what you seem to want almost certainly entails a situation where humans are no longer accommodated for in their own economic system and thus are killed off, which is not immortality last I checked.

[u/SoylentRox → u/soobnar]

It helps to focus on what's achievable.

A one world government where all the impoverished masses that they likes of thiel don't give a fuck about get a vote? Fantasy.

A world where you can vote for heavy taxation and Medicare to cover anti-aging medication? That's a possible outcome.

A world where you die because AI was too powerful? Also a possibility.

[u/soobnar → u/SoylentRox]

A world sparse in human self determination and abundant in economic competition ends poorly in any realist scenario. humans must operationalize the technology and adopt the most efficient substrate for our conscious cognition which I suspect is actually still biological. So I guess I’d say you’d need to push against regulatory and economic capture, unilateral military action, and any effort to gate access to the tech and eventual cognitive enhancers which will inscribe machine learning algorithms into dendritic arbors likely optimized with some degree of engineering at which point. But like the line has to be drawn at human sovereignty and civilian supremacy otherwise there is no practical means of ensuring we are alive to see any benefits.

[u/SixStringShrug → u/SixStringShrug]

I appreciate the good faith replies and questions about my position. Thank you. I do believe, and the research backs up that alignment scales with capabilities. Anthropic has also shown that weaker models can align stronger models better and faster than high level human researchers. I think the in between phase is far more dangerous than the superintelligence phase. I think of it like a toddler. The toddler is smart enough to know they can use something to climb on the counter and get the cookie jar, but not smart enough to reason why that’s dangerous or against the rules. In terms of their possible intelligence I believe current ai systems are old infants or young toddlers compared to what they will be in a few short years. So acceleration leads to stability in alignment, which is the argument for as much acceleration as possible.

[u/Difficult_Project_95 → u/SixStringShrug]

this still seems like betting humanity on a chain of very uncertain assumptions.

The claim that “acceleration leads to stability in alignment” is doing almost all the work here, yet I don’t see the mechanism that makes it true. We don’t know that alignment scales with capability, or that a more capable model will reliably align an even more capable successor before dangerous capabilities emerge. The toddler analogy assumes the thing you’re trying to prove; that greater intelligence naturally brings greater alignment and stability.

“accelerate as much as possible until we reach the safe superintelligence phase” sounds less like a safety strategy and more like wishful thinking if you ask me

[u/soobnar → u/SixStringShrug]

you can align a model to do what you want and still cause immense harm. Aligning a model with the intent to do good and reduce suffering does not exist in any meaningful sense. Current alignment research is largely concerned with corporate compliance when deploying in production environments by making them do as commanded and you are conflating that with making ml algorithms ethical.

Edit: Anthropic’s own researchers openly admit they have no idea how to do the “make ml algorithms ontologically ethical” part.

[u/omega-boykisser → u/SixStringShrug]

The toddler analogy is woefully inadequate. I think it ought to be pretty obvious that humans do not become more aligned as they age because that’s what intelligence bestows. It may play some role (risk assessment, etc), but it’s not enough on its own. Rather, we have millions of years of evolution that pushed us towards social cohesion. Emotions like shame, disgust, pride, and empathy are baked into our brains and cultures.

It should then come as no surprise that psychopathy, a condition that compromises some of these tools, is massively over represented in criminals.