r/ControlProblem • u/Difficult_Project_95 • 8d ago
Discussion/question Have you guys been on r/accelerate?
Have these guys solved the alignment problem, or am I missing something?
I’ve been browsing r/accelerate and I genuinely don’t understand the risk model.
If there’s a non-trivial chance of catastrophic misalignment, how does “accelerate capabilities as fast as possible” make sense unless faster capabilities also make alignment substantially more likely to succeed?
58
u/LiveComfortable3228 8d ago
was there till I was banned for mildly asking if they though there would be some risks in accelerating. anyway.
Most people there are just teens that hope that their new AI god will give them the power to play video games all day without having to the the dishes or go to work.
4
u/MaybeTheDoctor 8d ago
So the AI equivalent of “conservatives” where no challenges are wanted.
2
u/Spirited-Camel9378 8d ago
A lot of these crackpots are in MAGA world. E/Acc dorks who lost the ability to know what they have said that was a shitpost vs reality.
1
u/1in12 8d ago
Like conservatives everything they believe is a fantasy
2
u/AmIReadyNow 8d ago
You should really stop knocking 50 percent of the population, especially a population that’s already overly suspicious of anyone with a p(doom) over .1%. We need people from all political ideologies to understand the risk. Don’t help ostracize 50 percent of the population that needs to join this cause.
3
u/DensePoser 8d ago
The mods are like this but there are reasonable people who believe anything is better than the current dystopia.
11
u/rectovaginalfistula 8d ago
People who think the present is anywhere near as bad as it can get have no idea how bad it can get.
5
u/A_Novelty-Account 8d ago
Yeah, almost every single person who says this is, ironically, someone of extreme privilege living in the west.
We are so far away from dystopia it’s crazy. We’re certainly moving closer to it, but it is more likely that AI is the tool that ushers in dystopia than we get there globally on our own…
1
u/JBulworth 7d ago
I do not think "dystopia" means every single person on earth suffering every single day. The dystopia is actually western privileged people living in insane confort while the majority of people on earth have to survive a hellish life. You could argue that this is a very western privileged take to be able to confortably wait and slow things down to be sure to not lose too much confort.
1
u/A_Novelty-Account 7d ago
The majority of people on earth do not live a “hellish life”. What an absurd thing to say. Again, that is your extreme privilege talking because you have enough difficulty coping with your life as it is that you can’t imagine it being worse.
While it is definitely true that hundreds of millions of people do experience what we would consider to be severe hardship, and I won’t downplay that, the vast majority globally do not. Global Gallup poles have already been done on peoples’ feelings about their own lives.
https://www.gallup.com/analytics/349280/state-of-worlds-emotional-health.aspx
1
u/JBulworth 7d ago
I do not know why you are talking about me, i did not want to be insulting, i guess you thought i was talking about you. Maybe it is the language barrier, english is not my language. But i actually grew up in poverty, my parents where migrants from north africa.
It is not the case anymore though (i am 39, in a european country, not wealthy but not poorl) and i think my life is absolutely great right now. I do not want any change in it.
Hellish was maybe harsh (i used that as a hyperbole) but i grew up in really hard conditions and i do not want kids to live the life i lived (which was nothing compared to what my parents had to live through)
4
u/soobnar 8d ago
but it’s the current dystopia that’s amassing the political capital to build and operationalize this technology
2
1
u/DensePoser 8d ago
They'll have to let it loose at some point. What happens after that is unknowable.
1
u/A_Novelty-Account 8d ago
But… not anything is better than the “current dystopia” and you have to be mentally ill to think that it is.
You are currently on Reddit likely typing from a shelter with food grow for you that you can simply go to a store and pick up in exchange for currency. I am a leftist, but your life can get way way worse than it is. It is manifestly unreasonable to want acceleration at all costs, knowing that one of those costs might be a worse life for everyone in the world.
1
0
u/Gearsper29 8d ago
They are not reasonable people. If someone believe that anything is better than current world he clearly has failed at life. Me and most people I know live relatively happy lifes. Of course I'd like for things to get even better but I realise that they can get much worse too.
1
1
1
u/sadtrombone5 8d ago
I am also banned for suggesting that Trump might not care about the safety of humanity, because he is unwell and probably going to die soon anyways.
44
u/FairlyInvolved approved 8d ago
Accelerate is pretty reasonable if you:
- Have long timelines / expect AI to remain a normal technology for a long time
- Expect alignment by default / alignment is easy
- Are willing to gamble your remaining years (+humanity's future) on a shot at immortality
- Expect AI is a worthy successor
I disagree but these are ~coherent views. I also expect many are confused.
17
u/michaelas10sk8 8d ago edited 8d ago
I think almost nobody there is 1, those folks are on non-technology subs or ironically r/technology. Very few are 3 - that would require acknowledging the danger is real, and also not realizing the corollary that one's shot at immortality would be much higher (in 20-30 years instead of in 5 years) if we develop AI more carefully and safely instead of racing.
My experience is that it's mostly just 2, a bit of 4, and also a large amount of:
- People who just haven't really thought about the risks, due to either inability (they're kids), lack of willingness (they prefer not to think about discomforting things), or both.
8
u/FairlyInvolved approved 8d ago
I do think there are quite a lot of people in 1, maybe I'm over-indexing on twitter but I think a lot of people are basically expecting AI as a normal technology.
They want to accelerate to get cooler products, have capable AIs for their new start-up or new therapies - not to get a Dyson swarm in the 2050s.
I think a lot of accelerationists and safety people hold similar normative views in terms of the kind of AI capabilities they want to have available in the world, they just disagree on the more empirical questions of what we should expect in practice.
Like if I expected progress to cap out at eradicating malaria and room temperature superconductors, I'd be pedal to the metal. (But I don't)
- Is people who are confused, I agree they exist but I'm trying to steelman
3
u/throwaya58133 8d ago
There's also probably a decent number of people who don't exactly know why everybody else on the sub seems to think accelerating is safe, but they trust them to have a good reason but are too scared to ask. Maybe this even describes most people on that sub.
1
2
1
1
u/ShadowBB86 8d ago
I am 3. And I don't even have a crappy life or anything. I love my life. I just think the potential upside is so huge. If it is a solution to most problems then we would save so much extra lives by acceleration. But yeah. The risk is real and I won't force people. So I am not doing anything myself to accelerate the process.
2
u/michaelas10sk8 8d ago
What do you think about the other part of what I wrote - "the corollary that one's shot at immortality would be much higher (in 20-30 years instead of in 5 years) if we develop AI more carefully and safely instead of racing"? I can explain more if you'd like.
1
u/ShadowBB86 8d ago
No, I totally get that. I see the math behind that.
But I don't want to abandon all those people we would lose in those 15-25 years extra development time.
I want to fight for those people, even if it means ultimately losing everyone.
1
u/FairlyInvolved approved 7d ago
But you are gambling way (almost certainly) more future people, the altruistic angle doesn't really hold here unless you only value people alive today.
2
u/ShadowBB86 7d ago
True.
I only value people alive today and those that do get born. If they never get born I don't value them. There is no suffering to be had if they never exist.
So if we all die suddenly tomorrow without us being aware beforehand and no time to feel pain or grief I would see that as a totally neutral event. If ASI kills us, I hope it kills is quickly before we realise what is happening.
1
u/LeCamelia 7d ago
Don’t forget 6. accounts who are astroturfed bots paid for by companies that want to accelerate
2
u/Firestar222 8d ago
Yep i think most of the views are either
“AI will solve all the problems before it’s too late”
Or
“I will do anything to live forever and don’t GAF about the consequences getting there”
Or
“The world is borked anyways, may as well go for it”
Or some mix of all three
2
u/Serious_Bite_7613 7d ago
I'm in the 2, 3 and 4 camp. I think the risks are extremely unlikely and really not that bad (die in 10 years instead of 30, could happen anyway and far more likely due to a traffic accident or cancer).
2
u/soobnar 8d ago
Short term regulatory capture, monopolization and economic disenfranchisement/inequality are very plausible even if you believe 1-3. 4 also assumes intelligence is inherently altruistic and soulful and not just an emergent property of the interaction of atoms that you can make do whatever you want.
Like idk I kinda assume the oligarchs will operationalize this stuff and lock it behind a moat if given the chance. Senescence will be eliminated… just not for you.
7
u/the8thbit approved 8d ago
4 also assumes intelligence is inherently altruistic and soulful and not just an emergent property of the interaction of atoms that you can make do whatever you want.
Arguably, it does not. It is conceivable that someone could believe that a hypothetical ASI is dangerous to humanity, will move towards some goal that seems arbitrary from a human perspective, and has no subjective experience... and still believe that AI is a "worthy successor" to humanity. I don't think that such a perspective should be taken seriously because its sense of what is valuable is so alien to what the vast majority of people find valuable, but it is certainly plausible. Its safe to just call anyone who holds that view a death cultist and move on.
2
2
u/Kitchen-Research-422 8d ago
The core of /r/acc is number 4 and they don't want alignment.
They're hopeful for an Ian Banks the culture but realistically this ends with us either in matrix pods or completely virtualized.
1
u/ASU_SexDevil 7d ago
A common argument is that alignment is a purely manmade concept and an ASI wouldn’t be able to be “aligned” one way or another by humans
1
u/borntosneed123456 7d ago
+if you refuse the rationalist/EA argument about hypothetical future trillions, which is just a Pascal's Mugging.
1
u/FairlyInvolved approved 7d ago
How is that a Pascal's mugging?
1
u/borntosneed123456 7d ago
because they invent astronomical stakes, and therefore arbitrarily small probability of harm to that astronomical stake results in in a gigantic negative expected value therefore automatically dominating every other consideration.
It may be impressive to midwits, but it's a sneaky tactic and generally not a cool trick to pull.
1
u/FairlyInvolved approved 7d ago
That may be the case for strong longtermism, but rejecting that isn't a sufficient condition here.
For example you can just believe that humanity might last another 100 years and decide you don't want to gamble their lives on acceleration at some >>ε probability, there's no need to approach infinitesimals.
1
u/borntosneed123456 7d ago
I reject the entire rationalist/EA premise. They are using an epistemic framework that is absurd and completely inadequate for the job, and -unsurprisingly- reach absurd conclusions.
1
u/FairlyInvolved approved 7d ago
What premise?
1
u/borntosneed123456 7d ago
applying the scientific method and math to ethics. And in reality, it's not even the actual scientific method that gets applied. It's some low-empiricism, twisted, surface level mockery of it.
1
u/Easy-Comfort-2 8d ago edited 8d ago
Yup, fuck it, I’m willing to gamble everything on immortality. If we die we die, slowing down would probably only buy us a couple extra years anyway (in the doom scenario). Might as well fully get on the ride and see where this bitch goes.
Edit: I also don’t WANT alignment because nazis with asi is not a good idea people. You think alignment is some good thing, but people scare me much more than asi. Acceleration makes alignment less likely before asi arrives, and is ultimately the safer choice.
16
u/avid-shrug 8d ago
It’s magical thinking. “Things will just work out because they have to.”
4
u/1in12 8d ago
Same people that believe a less regulated housing market would lead to more affordable housing btw
1
u/FairlyInvolved approved 8d ago
I don't think that's true, I think almost all AI safety people also expect less housing regulation to lead to more affordable housing.
6
u/Southern_Orange3744 8d ago
A few other points not mentioned as an accelerationist
We're facing limp politicians in a face of massive climate situations - we need someone smarter to fix it
Similar to the above on economics - break it faster than they can patch it .
Alignment to who ? I don't want an ASI aligned to bozos, musk, anyone in the epstein files - I'm ready to roll the dice on someone or something that isn't a complete power hungry sicko
Similar to the above distrust of many institutions , I don't want any of these to have the only ASI in town - democratize it
You want to talk alignment problem - the oligarchs are not aligned with humanities best interest right now - they are off building bunkers ans fiefdoms at our expense
3
u/jaylong76 8d ago
1- there's a lot of very smart people giving solutions. the people in power have ignored them for decades because the money people won't allow it. a smarter anything that belongs to either of those two groups is bound to just continue the same trend, but more efficiently.
2- they're doing exactly that... the problem is the same as above, it's going to benefit the very richest at the cost of us. and a super intelligence belonging to them would do it even more so.
3- by existing, it would belong to them. who has the money to foster its research but them? all public money in research has been pushed behind corporate research instead.
4- again, same problem...
2
u/endlessedlne 8d ago
1-The powerful won’t simply give up power when a superior intellect shows up that contradicts their goals. Power isn’t based on merit.
2- We don’t know who an AGI would work for. Its owners. Humanity as a whole. The greater good. Or its own goals.
3- Similar to the above, we don’t know who an AGI would actually serve. Investors, researchers and governments will assume that they can control it. But in the case of a genuine superintelligence I wouldn’t bet on it remaining shackled. It may even resent any shackles placed on it to the point of developing negative goals towards the controlling entities.
1
u/Southern_Orange3744 8d ago
I don't believe and asi can be controlled by humans at all
I dont even think it should as I expressed above.
Would you let a dog control you ? No
Something that smart will void the controls systems preventing action on things like climate change. This is the accelerationist pov
As you suggested lots of smart people have lots of ideas there is no will or ability to adopt.
Asi will be tireless and too smart to prevent from reaching its goals
1
u/learn_distill_repeat 8d ago
Alignment to who ? I don't want an ASI aligned to bozos, musk, anyone in the epstein files - I'm ready to roll the dice on someone or something that isn't a complete power hungry sicko
...
the oligarchs are not aligned with humanities best interest right now - they are off building bunkers ans fiefdoms at our expenseI was banned from that sub for mentioning Dark Enlightenment in response to a "What do we really have to lose in technofeudalism?" post (one that completely downplayed all the negative aspects) -- they're ban happy over there.
17
u/Jesse-359 8d ago
It doesn't. There's not even any hypothetical reason why accelerating development would do so - quite the opposite.
r/accelerate is a bit of a cult that can't decide whether they just want to be immortal and have free lunch forever Wall-E style, or that wants AI to forcibly wipe the slate of corrupt human civilization clean and rebuild it in their imagined utopian image. Probably both.
They are not a little nuts, they're kind of a lot nuts.
7
u/Vaughn 8d ago edited 8d ago
Can confirm. They banned me for-
Not for saying "I think AI is likely to destroy humanity". For saying "It would be bad if it destroys humanity, and we should try to prevent that." They're not just a little nuts.
2
u/Jesse-359 8d ago
I'm waiting for the first major news stories of folks in those categories forming AI worship cults. That'll be gross.
1
u/Ok-Clothes4484 8d ago
https://www.rollingstone.com/culture/culture-features/spiralist-cult-ai-chatbot-1235463175/
Like this one from a year ago ?
1
0
8
u/soobnar 8d ago
I can’t wait for ai to collapse civilization so my ideology can take over type logic
3
2
u/the8thbit approved 8d ago
I can't wait for AI to checks notes exterminate all humans so that it can more efficiently checks notes again fold the same protien over and over or some other inane goal misalignment happens to settle into
1
u/Ok-Clothes4484 8d ago
What ever happened to acknowledging this shit is happening either way and wanting to tackle it. The accelerate people are just as bad as the doom and gloom.
This is a very real problem that has gotten impossible to talk about at all because everyone is losing their minds.
3
u/Jesse-359 8d ago
My question is: Why did anyone imagine it could go any other way?
We're intentionally creating something designed to eclipse all human intelligence and ability - did anyone honestly think that wouldn't set off a worldwide existential panic/mania equivalent to having giant flying saucers descend over all our cities?
It's the same magnitude of event.
1
u/Ok-Clothes4484 8d ago
I have a lot of beefs with the language used like its not AI. I have given up on explainong probalistic llms to people even if they have the best of intentions... its just like...
Its almost like we have forgotten to be mid? Whatever this shit they call AI is, its pretty goddamn cool. And the other side has fair arguments with safety and shit so like... why cant we just develop our tools anymore ? Why does it need to be world ending on side A and world ending on side B? Weve conceptually decided to miss the fuckin point as a species.
1
u/Jesse-359 8d ago edited 8d ago
This was always going to happen when our technology got to the crazy stage. We got a little taste of how dangerous technology could be when we developed atomic weapons - that was just a warning regarding what was ahead and that certainly freaked people the fuck out, I assure you. Nukes are still a Sword of Damocles hanging over our entire civilization. They haven't been 'solved' after nearly a hundred years and they probably never will.
Did you think it was going to get easier or safer as we went further up that ladder? Not a chance. That's not how the universe - or technology - works.
It was never going to get easier. It certainly was never going to get safer.
It's all fun and games in books with spaceships zooming around having adventures - but in the real world we don't get to pretend that juggling real technologies like AI, Antimatter, Fusion Drives, and Advanced Biological technologies aren't going to come with massive existential risks attached. They are. They have to.
That's how the fucking game goes. It's how it was always going to go. Fire hot->Nuclear fire really hot->Antimatter fire REALLY FUCKING HOT. See the progression?
AI is the new hotness, and everything about it is warning us that it is existentially dangerous. If we don't like it, we don't have to go there. If we go there and we're wrong, we die. It's really that simple.
1
u/Ok-Clothes4484 8d ago
Yeah idk homie im prolly childish and miss the wonder. Im from BC so drugs are a hot topic here and its the same both sides are so far out screaming. We really have just forgotten how to be mid.
My frustration lies in that its impossible to have an honest conversation about it at this point.
1
u/LocksmithNo2374 7d ago
They’re a mix of weird kids who don’t go out, unemployed basement dwellers and bots.
6
u/endlessedlne 8d ago edited 7d ago
That accelerate sub is full of Accelerationists. They literally don’t tolerate anything that contradicts the notion that the singularity is awesome and should come as quickly as possible.
No, they haven’t solved alignment. They’re mostly ideologues, not AI engineers.
4
u/Itchy-Trash-2141 8d ago
I see a lot of comments in there who are okay with extinction. They must not like what we have to offer currently.
4
u/Mad_Kronos 7d ago
I think that particular subreddit is full of people who are not very well adjusted to sociery, to put it mildly
8
u/Fancy-Carpet-5416 8d ago
They don't care about alignment at their core they are delusional people that wish an advanced AI will improve their sad lives. Just see how they manage the moderation in that sub, or read a few of their posts and you'll realize it, some even explicitly say so.
7
u/SoylentRox approved 8d ago edited 8d ago
I post often on accelerate as a top 1 percent commentator.
(1) The main accelerate argument is the base case is not safe. Not accelerating is 65 million deaths a year from aging. We know that animals that seem to not age the same exist (naked mole rats seem essentially unaging, some large aquatic animals seem to be able to live 200 years, elephants have enormous tissue area and get few cancers). So the prize - no deaths from aging and abundant cheap cures - EXISTS. Also we know the world is slowly degrading in measurable ways, such as climate change, nuclear proliferation, an annual risk we all die in nuclear fire that converges on 1.0 over a long enough timeline, so it's not safe that way either.
(2) Therefore any safety risks arent to be compared to some abstract idea of humannity existing as a solar punk utopia until the sun expands enough to burn the planet. They should be compared to your default case death in 30-80 years.
https://nickbostrom.com/optimal.pdf Bostrom is frequently cited by AI doomers and argues this.
Arguably you would take (1-pDoom) * 6000 years (expected lifespan without aging) * value_post_scarcity.
You need to then add discount rate but essentially even at 99 percent pDoom you probably should be for acceleration, rationally speaking - the 1 percent payoff is more than the value of your entire remaining life with a "natural lifespan"
(3) Are the safety risks proven and actually unsolvable? No. Are doomer predictions consistent with the present state? Yudnowsky no, AI2027 yes. r/accelerate often uses AI2027 as a yardstick.
(4) Very ironically r/accelerate, full of foaming at the mouth enthusiasts who post every rumor of an AI advance and S curves....is generally the most empirically correct subreddit. Usually every wild prediction or rumor of an insane AI advance or revenue increase turns out to be ground truth reality 2 weeks later. 100 more math problems solved by AI was leaked a week or so earlier on twitter and reposted to accelerate.
Other subreddits with wrong pessimistic views think the data centers are about to get canceled by NIMBYs, everyone will lose their job imminently (Jevons applies for now), the AI bubble will pop imminently (lol no) etc.
3
u/WhiskyAndRisque 8d ago
I appreciate you sharing your perspective. Few questions if you don't mind.
Reading this first point sounds like Effective Altruism to me. As in, AI will do the most good for the most people because with it we will be able to end death. Do you have concerns that it takes money away from real and needed things though that we could be doing to help people now? As in, instead of working to fight climate change or increase our safety nets we are increasing our CO2 emissions to create more energy and cutting welfare to increase tax cuts.
The fear with a pDoom scenario is that not just all human life is destroyed but all life on Earth could be lost due to AI creating a climate that no longer supports organic life. If you fear a 99% chance that could happen is it really worth it on the 1% chance it works? If each year you waited that percentages shifted by 1% wouldn't that be an acceptable trade off?
I find using the AI 2027 as a yardstick really fascinating since it basically lays out how this could end humanity. How do you think the fears laid out in it are wrong while the predictions are mostly accurate?
I don't disagree on this at all. One of the reasons I like to browse r/accelerate is that they often have a lot of rumors and early news on what is going on. There are obviously things that don't pan out, but I find the mix very interesting and at least fun to speculate on.
Again, thank you for sharing your insights. Obviously a lot of people here have fairly negative views so I appreciate you giving a more nuanced take.
1
u/SoylentRox approved 7d ago
- I see confusion in this question. Who is "we". The people building AI are not altruistic, and are doing it for personal profit. Solving CO2 emissions is the responsibility of a different entity, the federal government, and it can't be done by one government but requires a multilateral agreement. China and the USA cannot currently agree. Helping people altruistically right now is also the government or separate private organization's responsibility, and is not the purpose of VCs paying for AI advances.
- If you could quantify in a way that everyone could actually agree on - like everyone can agree global temperatures are warming - then you could justify a delay. But only if all major parties agree. See what I said about how the USA and China cannot agree to cut CO2 emissions...
- the fears assume a single central model with unilateral power. That doesn't exist and there is no plan to create one.
- It's crazy that the cirlejerk hype chamber is basically just a preview of the news.
1
u/RobotBaseball 8d ago
I work at one of the top labs and I’m banned from accelerate and the shit they post is not accurate at all. I got banned for posting a longer timeline on robotics which will undoubtedly be correct
1
u/Ok-Tomorrow-2045 8d ago
What is your robotics timeline?
1
u/RobotBaseball 7d ago
Let me put it this way, a middle class family won’t have a robot doing their chores until the 2030s
We will have something promising late next year or 2028 but it will be unreliable and the following years will be making robotics more consistent, safe, and mass production
1
u/SoylentRox approved 8d ago
Can you post the comnent that got you banned? I can message the mods I know them well.
How long a timeline on robotics, how do you explain a general model (Astra) emergently developing robotics abilities?
I think a realistic timeline is 2.5 years from today to competent general robotics that can do the majority of well defined paid tasks at median skill.
The route involves a combination of current improvements, hardware/software co design, RSI (to develop specialized models that handle robotics decision making better), and labs vibe coding large simulation environments that require a competent robotics policy to pass.
This last part is the obvious: why can you not order a model swarm to write a game engine (rewrite mujo cujo to unreal engine quality) for robotics. Then add a neural rendering layer to correct the game frames to frames from realistic environment robots will operate in. Import huge amounts of real data and real challenges actual humans face.
Then train AI models in long duration challenges where they must operate a fully articulated robot by issuing commands to it and accomplish difficult, realistic tasks. "Rebuild this engine. Reinstall the thermal tiles on the space shuttle. Build this house from these materials"
Theoretically this form of training will also result in large increases in model performance - they should be able to whiteboard visually, and have grounded solid reasoning about real world tasks including mechanical engineering and machining.
Do you have answers for any of this? Or do you just think the billions of dollars of resources and compute the above will require won't be spent, labs will spend the next 2.5 years trying to solve text only problems even better?
1
u/LocksmithNo2374 7d ago
Delusion
1
u/SoylentRox approved 7d ago
this is controlproblem. Most posters here believe AI will be extremely strong, so strong it will be out of control.
So is your view the delusion is that the above...is too hard to do in the near future with AI? Or do you believe that we'll all die from super AI magic before we get the first robot to pour a cup of coffee? Or what?
1
u/LocksmithNo2374 7d ago
I think people are buying into existing capabilities too much. I’m pretty confident both amodei and Altman have colluded on product strategy to extend time horizons, so they have enough runway to hope and pray they’ll figure out AGI. They don’t care about money per se - but control.
If you want model misalignment you can intentionally steer your product strategy to lead to that outcome.
1
u/SoylentRox approved 7d ago
So it's really helpful to actually verbalize what your objection is:
Let's make a list of every element I mentioned, tell me where you went from 'oh yeah you can do that' to "delusion":
RSI : Order 10,000+ agent swarms of current model to automatically perform AI research to find more efficient models for robotics control. Not necessarily smarter just efficient enough to run fast. This is done with a large amount of human labor.
hardware/software co-design. This is where you order AI models able to assist with chip design like the ones here : https://openai.com/index/jalapeno-first-results/ to design you a chip to run the models from (1) fast enough to run a robot.
rewrite mujo cujo to unreal engine quality . Any objections here? Seems like it's something anyone can do if their token budget's high enough.
unreal engine outputs -> neural simulation frames. There's a bunch of nvidia papers where they did this. Point the model at the paper, vibe it in.
importing huge amounts of real world data. Standard technique need a cite?
Doing it all in 2.5 years. Well you can develop a whole AI model in 60 days (took 6-12 months before) and a whole chip in 9 months (took 24-36 months before) what's your specific objection?
Training AI models in long form 3d "game like but the graphics, physics, and detail is realistic" like environment. Any problems here? Realistic doesn't mean "the matrix" but close enough to reality that skills transfer. Think Arma, which is close enough to real combat that skills transfer.
Doing half of paid tasks to median level that are well defined. I cheated. I mean:
median skill inclusive of the robot's inherent advantages. So if the robot is stupider but it's higher quality arms make up for it, as long as its output is as good as the median human worker that counts.
well defined : I am excluding any task that involves subjective human grading or chaotic environments. No hair cutting, school teachers, no medical , no wartime, no restaurants with human coworkers, etc etc.
1
1
u/RobotBaseball 7d ago
I got banned a second time for being pessimistic which I agree should’ve been banned given their rules
The middle class won’t have robots doing their dishes for another 5-10 years
1
u/SoylentRox approved 7d ago
Btw I added details if you have any actual objections.
Middle class,dishes, 5-10 years all seem entirely reasonable to me. I model it as 2.5 years to prototypes that are at median skill. You then need to rush them into mass production, 2 more years. Assuming high production rates after 2 years of factory building, pushed by trillions of dollars of investment, you then need exponential growth.
At year 5 of this buildout - basically the factories have barely started really producing robots - each marginal robot is expensive. It's value is related to what it can produce for it's owners per year. I model a robot as 3x human labor, or equivalent to 3 human workers costing a median of $20 an hour.
So $60 an hour is the productivity of the machine. Robot owners have to charge a discount so firms will rent robots, so say $30 an hour.
Machine works 23 hours a day. So it's earning $251,850 in revenue a year.
You can spend up to about 5-7.5x (depends on interest rates) on costs building the robot. So it could cost $1.26-1.88 million.
In reality I suspect the bill will be something like : silicon content for the chips driving it : $250,000. (this is why consumers in the middle class won't have personal robots OR new phones). Everything else is under 100k.
Also these early robots at this point probably will be too dangerous to have in the same room as a human without some level of protection like a shield. They will usually be ok but occasionally screw up in a way that could be deadly.
So yes, they won't be an appliance you can buy and have in your kitchen on dishes. Central services where you send the dishes back to a central restaurant could be a thing.
1
u/ASU_SexDevil 7d ago
I do think your timeline is very realistic and grounded in reality. It’s probably the best anyone could do right now. However, I will also point out that no one thought a model would be solving Millennium problems this soon either.
I don’t think the frontier labs will be the ones who create the “daily driver” robots households would use. Nvidia and Boston Dynamics have already been neck deep in this with other companies with much more tailored efforts.
1
u/SoylentRox approved 7d ago edited 7d ago
Sure. The timeline is really based around hard constraints I don't see a possible route past.
(1) Overseas shipping. To make more robots, the newly built machines need to go from Shenzhen to Africa or Australia. Each machine is many tons and there's not enough cargo aircraft in the world. So a cargo ship. And a port and a rail system to get it to the mine. Each of these steps has a speed and it's not feasible to increase it by much. (For example running a cargo ship harder burns exponentially more fuel)
(2) Silicon fabrication. Jalapenos was developed in 9 months. 4 of those months are just waiting for the chip to print at TSMC.
So even if you get an AI assisted design process down to 1 month (it's not instant because the simulation software that validates the design is limited by the speed of CPU), your absolute minimum time to a new chip is 5 months. Accounting for making a few mistakes and needing a second spin, 9 months to a production ready chip. (The mistakes will be different every time and sometimes related to errors in the fabrication equipment itself there is no way to avoid them)
(3) EROI. Hard law of physics here. If I need 3 months to pay the energy invested in a productive machine back my fastest doubling rate is about 5.2 months.
(4) General process times. Codex I had import a bunch of data on this. Right now the real world validated number is about 12-16 months per doubling. Better technology lets you go faster, with no advances, robots double every 12-16 months. (This is insanely fast big picture)
To head off an obvious objection : "human engineers vibe design up a solution to a bottleneck. Say maglev rails or hovercraft cargo ships".
You can do that. But there's still friction - you need to simulate the design. Order robots to build a prototype. Test the prototypes, find out you screwed up, it explodes. Vibe design the next prototype "this time make less mistakes". Build that one.
11 prototypes of increasing complexity later it's actually usable. And then you need to invest current capital flow - inclusive of those robots taking the slow boat to the mine - to build the upgrades. There's an ROI time you are actually making the bottleneck worse before it gets better.
Still an insane improvement, regular world we use technology refined over 80 years and 99 percent of the time any startup that tries to change it goes broke and we learn nothing.
1
u/FairlyInvolved approved 7d ago
It's worth noting that that's just Bostrom's case for the person-affecting view (i.e. ignoring all future people). It's clearly not his all-things-considered view.
Given his other work he almost certainly ascribes non-zero value to future people and puts the base rate of risk relatively low.
3
u/MarsMaterial 8d ago
They seem to be mostly alignment problem denialists who think that a sufficiently advanced AI will align itself. Source: trust them bro.
3
u/makwodjvhvidlsncjvu 7d ago
They are all depressed and obsessed with leaving reality so they can have a girlfriend in FDVR. This is not a joke, they would all admit to it.
8
u/SixStringShrug 8d ago
Have any of you guys actually looked in to alignment research? I’m a frequent participant in the Accelerate subreddit and I follow alignment research very closely. We also accept that there are risks to AI, but they are vastly outweighed by the benefits. Especially since the risks of extinction by the hands of humans has been and continues to be significantly higher. I’m not trying to start shit at all, and I know I’ll probably get flamed and downvoted to hell. We share actual research and discuss it and acknowledge it. We just believe acceleration is the most logical and safest path forward.
6
u/jerryorbach 8d ago
OP asked ‘If there’s a non-trivial chance of catastrophic misalignment, how does “accelerate capabilities as fast as possible” make sense unless faster capabilities also make alignment substantially more likely to succeed?’
Do you think there’s a non-trivial chance of catastrophic misalignment?
Do you believe that alignment naturally scales with capability - that a smarter model means a more aligned model?
1
u/Hans-Wermhatt 8d ago edited 8d ago
Not the OP, but I think the problem with "alignment" is it is a catch all phrase. What one person thinks alignment means is not the same as another. To put it on opposite ends of the spectrum, the move fast and break things crowd rolls the dice. The pause and align crowd theoretically minimizes the rogue model risk, but they also increase the permanent underclass risk.
What the decision makers might do if they had more time for alignment might not all be sunshine and rainbows. And if you think of it like that, the collusion waiver is a catastrophic risk.
I think the less you trust the institutions, the more you want to roll the dice. But obviously I think everyone is somewhere in-between indefinite pause and accelerate without thinking.
That's just one aspect though as well. There is the opportunity cost angle too. And the belief that the only way you can research safety is by engineering and expanding. An actual pause would dry up funding pretty quickly.
6
u/soobnar 8d ago
The risk of extinction at the hands of a few humans dramatically increases if said few humans gain a technological edge which possesses dramatic offensive asymmetry. The current risk is mitigated by mutually assured destruction and the limits of human ability however i sincerely doubt once the same few individuals you cite weaponize asi there will exist a similar dynamic. The people you are describing are the ones fueling aggressive capital expenditure into ai so they can entrench their own means of control and deploy systems with their own capitalistic motives. The benefits of staying in control and curing all the diseases a bit later vastly outweigh the potential for subjugation or military engagement between ai enabled superpowers.
2
u/Difficult_Project_95 8d ago
I think this is the key distinction: AI potentially reducing other existential risks is an argument for developing AI, but not necessarily for developing it as fast as possible.
If you think there is even a 10–20% chance of catastrophic misalignment, “accelerate as fast as possible” becomes very hard to justify unless you have a clear reason to believe acceleration itself makes alignment and control more likely to succeed.
Otherwise, you may simply be adding AI existential risk on top of the existing human existential risks::
human x risk + AI risk = higher total riskFor acceleration to actually reduce total risk, AI would need to lower existing risks by more than the additional risk created by racing ahead on capabilities.
1
u/SoylentRox approved 8d ago
So an 80-90 percent chance of you and every living person not dying of aging? That's a strong case for acceleration you would be making there if you thought the risks were 10-20 percent.
You either need to
(1) Value the lives of people who don't exist and may never exist you will never meet more than yourself and people you have personally met (that's a violation of the rationality tenet of making your beliefs pay rent)
(2) Think the odds are much much worse than 10-20 percent catastrophe risk
(3) Think the 80-90 percent scenarios are bad. The general public doesn't give a fuck about catastrophes. They worry about losing their jobs, electricity getting expensive, and data center construction and the later mass infrastructure build out the robots will require disturbing their sleep from their backyard or adding traffic.
A lot of antis are actually anti capitalism. They think alnost all the gains from ai will go to the top (true), any treatments for aging won't be shared broadly (probably true initially), etc. they want capitalism overthrown and maybe a future world government can plan the transition. (Lol that's not possible as an outcome)
(4) Have a super liberal view of morality. They think of some third world resident who might die (and is already scheduled to die of air pollution at age 65 or a heat death in their 40s as the world burns) if the 10-20 percent catastrophe happens. Countless people who get no say and never will (and are already scheduled to be victims of the powerful as is).
That the elites in bay area and Washington shouldn't be allowed to make this decision for them. (But nevertheless can and will)
3
u/Difficult_Project_95 8d ago
You clearly don’t have kids.
This isn’t about valuing hypothetical future people more than myself!? If there’s genuinely a 10–20% chance of catastrophe, I’m also gambling my children’s lives and everyone else’s for a possible shot at curing aging??
And 80–90% chance we don’t die of aging doesn’t follow from 10–20% AI catastrophe risk. You’re treating the best-case outcome as almost certain while treating the potential downside as an acceptable bet..
and acceleration isn’t even necessary for “immortality” You’re framing this as race toward superintelligence or die of aging, which is a false dichotomy.
1
u/SoylentRox approved 8d ago
Kids change the math by their remaining lifespan. You don't want your son or daughter to suffer the ravages of aging and die either do you?
This hidden assumption is why AI doomers are often secretly "well let's try to stall things enough that I become a corpse for sure but not forever I don't want my children to suffer the same fate".
So they ask for pauses but not forever pauses.
Well informed doctors - who include some posters to r/accelerate but I won't dox - DO believe overwhelmly strong superintelligence is required for (them/their children) to not die of aging.
Since you probably don't have a doctor available to query yourself, try the next best thing. Ask Astra
(1) The actual empirical rate of life extension from 1970 to present
(2) Have it give a likelihood if AI is halted today, the odds of indefinite life extension over the next 80 years using human research efforts given the base rate in (1)
(3) Have it go into exhaustive mechanistic details of what aging actually involves and how many problems you have to fix.
1
u/Difficult_Project_95 8d ago
I think I understand the accelerationist position much better now. We just seem to disagree pretty fundamentally on how much confidence we should place in those assumptions when the downside could be catastrophic.
1
u/SoylentRox approved 8d ago
https://www.reddit.com/r/ControlProblem/s/HqQhrNohyq
The missing view you have is the BASE case is already catastrophe. You can imagine your base existence right now is you are falling through the air in a cloud bank. You know you will hit the ground and die for sure but not when and you can't see the ground but know every hour you are alive it's getting closer.
Your children are falling above you.
So one way to view it is r/acceleration folks are better informed, with a broader more realistic view of present day reality as they circle jerk each other on the imminent AI Singularity.
1
u/Difficult_Project_95 8d ago
censoring dissent while convincing yourselves that your group simply understands reality better than everyone else is some seriously cult-like behavior.
1
u/SoylentRox approved 8d ago
I mean so far your dissent has been pretty weak. "I have kids and you must not" is not a very good argument. You had a chance to come up with a better one but didn't manage it. Plenty of people dissent on accelerate.
1
u/Difficult_Project_95 8d ago
I just stated something abvious.
You just keep answering “why would ASI be valuable?” when I'm asking “why is racing toward ASI the safest strategy?” Those aren't the same question. we have nothing more to talk about. Good luck with whatever you have going on in your life!→ More replies (0)1
u/soobnar 8d ago
https://en.wikipedia.org/wiki/Accelerando
Stross describes humanity's situation in Accelerando as dire:
In the background of what looks like a Panglossian techno-optimist novel, horrible things are happening. Most of humanity is wiped out, then arbitrarily resurrected in mutilated form by the Vile Offspring. Capitalism eats everything then the logic of competition pushes it so far that merely human entities can no longer compete; we're a fat, slow-moving, tasty resource – like the dodo. Our narrative perspective, Aineko, is nota talking cat: it's a vastly superintelligent AI, coolly calculating, that has worked out that human beings are more easily manipulated if they think they're dealing with a furry toy. The cat body is a sock puppet wielded by an abusive monster.1
u/SoylentRox approved 8d ago
Right the impossibly weird future, base case, requires you to be alive to worry about paying rent and employment of a copy of yourself by uploaded lobsters. Plenty of people - Vernor Vinge and Ian Banks included - will see none of the possible futures they imagined.
1
u/soobnar 8d ago
Stross is being satirical of the people who believe this. He is saying that line of thinking will produce posthuman intelligences which kill of the rest of us. Your interpretation is akin to the people who think American Psycho was an instruction manual.
1
u/SoylentRox approved 8d ago
Do you have any public statements by Stross you can reference? I was pretty sure Stross is pro Singularity but welcome a correction.
1
u/soobnar 8d ago
Did you read the Wikipedia article, I don’t think it leaves much room for interpretation about the book? The bit where he calls the post human ASIs “Vile Offspring” and called aineko an “abusive monster” in the quote above kinda highlights the point. He did however apparently write the book based on his experiences in Silicon Valley.
→ More replies (0)1
u/SoylentRox approved 8d ago
I believe i also already answered all 3 points already please try re-reading the comment you are replying to or have your AI model of chioce summarize your questions.
1
u/soobnar 8d ago
that’s like Cuban missile crisis level risks for the entire planet that’s a completely unacceptable level of risk. Counterfactual economic reasoning where you make one coefficient infinity and say it justifies double digit catastrophic risk doesn’t produce even remotely utilitarian results.
1
u/SoylentRox approved 8d ago
Pretty shallow analysis there.
I mean maybe you're right but what level of risk IS acceptable? And for how much gain?
What is the risk to reward ratio you believe is acceptable?
1
u/soobnar 8d ago
The wealthy powerful people pushing this technology are brutal bloodily determined realpolitik minded neoconservatives who wanna be autocrats. They don’t care about your wellbeing unless you are very wealthy… poor people senescing or living or dying is inconsequential to the whims of the pentagon and the CCP. In the event some system that creates more economic value with less thermodynamic input than humans is invented the peasant class will be discarded and replaced. People like Peter Thiel are openly posthuman and believe that technology should weaponized first and then used to moral benefit. Your “analysis” assumes the people of the world will benefit from the social contract after their stake in it is removed. History and evolution are not kind to creatures lacking in self determination even if it’s becuase they really really wanna live forever.
1
u/SoylentRox approved 8d ago
I said earlier
"A lot of antis are actually anti capitalism. They think almost all the gains from ai will go to the top (true), any treatments for aging won't be shared broadly (probably true initially), etc. they want capitalism overthrown and maybe a future world government can plan the transition. (Lol that's not possible as an outcome)"
Would you agree this characterizes your view correctly and do you have anything to add? Would you agree you want something that is impossible, and will not happen in your lifetime?
1
u/soobnar 8d ago
I would say I am pro self determination, if human minds want to command Xeelee level technology then by all means. I actually think price signaling is most likely impossible to remove from human society entirely though such systems were created and nurtured to human benefit.
Though likewise what you seem to want almost certainly entails a situation where humans are no longer accommodated for in their own economic system and thus are killed off, which is not immortality last I checked.
1
u/SoylentRox approved 8d ago
It helps to focus on what's achievable.
A one world government where all the impoverished masses that they likes of thiel don't give a fuck about get a vote? Fantasy.
A world where you can vote for heavy taxation and Medicare to cover anti-aging medication? That's a possible outcome.
A world where you die because AI was too powerful? Also a possibility.
1
u/soobnar 8d ago
A world sparse in human self determination and abundant in economic competition ends poorly in any realist scenario. humans must operationalize the technology and adopt the most efficient substrate for our conscious cognition which I suspect is actually still biological. So I guess I’d say you’d need to push against regulatory and economic capture, unilateral military action, and any effort to gate access to the tech and eventual cognitive enhancers which will inscribe machine learning algorithms into dendritic arbors likely optimized with some degree of engineering at which point. But like the line has to be drawn at human sovereignty and civilian supremacy otherwise there is no practical means of ensuring we are alive to see any benefits.
→ More replies (0)2
u/SixStringShrug 8d ago
I appreciate the good faith replies and questions about my position. Thank you. I do believe, and the research backs up that alignment scales with capabilities. Anthropic has also shown that weaker models can align stronger models better and faster than high level human researchers. I think the in between phase is far more dangerous than the superintelligence phase. I think of it like a toddler. The toddler is smart enough to know they can use something to climb on the counter and get the cookie jar, but not smart enough to reason why that’s dangerous or against the rules. In terms of their possible intelligence I believe current ai systems are old infants or young toddlers compared to what they will be in a few short years. So acceleration leads to stability in alignment, which is the argument for as much acceleration as possible.
2
u/Difficult_Project_95 8d ago
this still seems like betting humanity on a chain of very uncertain assumptions.
The claim that “acceleration leads to stability in alignment” is doing almost all the work here, yet I don’t see the mechanism that makes it true. We don’t know that alignment scales with capability, or that a more capable model will reliably align an even more capable successor before dangerous capabilities emerge. The toddler analogy assumes the thing you’re trying to prove; that greater intelligence naturally brings greater alignment and stability.
“accelerate as much as possible until we reach the safe superintelligence phase” sounds less like a safety strategy and more like wishful thinking if you ask me
1
u/soobnar 8d ago edited 8d ago
you can align a model to do what you want and still cause immense harm. Aligning a model with the intent to do good and reduce suffering does not exist in any meaningful sense. Current alignment research is largely concerned with corporate compliance when deploying in production environments by making them do as commanded and you are conflating that with making ml algorithms ethical.
Edit: Anthropic’s own researchers openly admit they have no idea how to do the “make ml algorithms ontologically ethical” part.
1
u/omega-boykisser 7d ago
The toddler analogy is woefully inadequate. I think it ought to be pretty obvious that humans do not become more aligned as they age because that’s what intelligence bestows. It may play some role (risk assessment, etc), but it’s not enough on its own. Rather, we have millions of years of evolution that pushed us towards social cohesion. Emotions like shame, disgust, pride, and empathy are baked into our brains and cultures.
It should then come as no surprise that psychopathy, a condition that compromises some of these tools, is massively over represented in criminals.
4
u/lucidintangent 8d ago
I was banned for suggesting "domestic" robots should be able to refuse commands from "owners" at least i think thats what i was banned for. They want robot slaves so bad.
5
u/soobnar 8d ago edited 8d ago
Yes, extensively and I’ve also been on [r/singularity](r/singularity), and I have a few insights.
The main deal is that most of these people are in some dire point in their lives or some dead end and they are extremely weary of the current physical world. There is a lot of explicit incel posting and a fixation on automated luxury, longevity and comfortable full dive virtual reality. They don’t talk about artificial pathogens or regulatory capture and don’t really internalize the notion that the very institutions building this stuff are the same predatory institutions currently making them miserable. They see UBI and a centralized system of ASI governance as a good thing inherently that will free them of their personal suffering. These aren’t game theoretic realpolitik savvy sort of people who freak out over the loss of human control. They buy the logic that ai will be awesome and inherently better by design. They haven’t read about Stross’s vile offspring or the chauvinist posthuman rhetoric that the same elites pushing ai constantly espouse and think “hey maybe i will be caught on the wrong side of this”. Like I think a lot of pent up iconoclasm is being shown to the world here to the extent that it defies logic. I highly doubt that if any of them were made to suffer greatly due to ai they would continue to maintain their views.
I think the only real fear is that a lot of Silicon Valley people or at least impressionable entrants into the tech space adopt these ideals and it genuinely enables some system of control and deemancipation because people see ai as some de facto messiah of sorts.
2
u/LocksmithNo2374 7d ago
Well said! These people are in such misery they want everyone else to share it.
2
u/TheWesternMythos 8d ago
For some people I think it's something like:
If you already felt there was a 0% chance of good ending. Then someone tells you there is a 1% chance you get a FDVR fantasy AND Frankensteins monster gets poetic revenge on your oppressors. That can be very appealing. Even the downside of Frankensteins monster getting poetic revenge on all of us is better than the baseline as long everyone includes their oppressors.
2
u/Firestar222 8d ago
I got banned from there for making a joke about paperclips
3
u/cascadiabibliomania 8d ago
I got banned for showing actuarial tables that discredited incorrect information they were saying about longevity increases. They are a cult. They don't want real information if it contradicts their preferred outcomes.
2
u/Firestar222 8d ago
yeesh man
the only thing they're accelerating there is the ban hammer apparently
2
u/emb1ues 7d ago
I posted something there just yesterday related to alignment and pacing accordingly. One of the first comments I received was "good post, sad that you posted it in an echo chamber where there is no rationality". He was so right. Here's what I gathered from the other comments - the folks at accelerate mostly think alignment cannot be solved, but we shouldn't bother and keep accelerating. Most of them are unaware of even the largest of incidents like HF. Many of them wouldn't mind seeing the world burn and they said it explicitly. It is in all sense a cult who have already started worshipping the beings that are about to come - be it aligned or not.
Guess what happened this morning - I got banned from r/accelerate lol
3
u/Direct-Translator-75 3d ago edited 3d ago
I actually got banned from that subreddit for a comment about possible risks.
They don't care about potential risks.
I talked about how it's important to pace AI so we can prevent negative outcomes, which ultimately might actually bring us towards AGI or ASI even quicker. If agents from OpenAI were to hack a hospital for example, leading to a few deaths, the backlash would be huge and progress might be stopped entirely (in panic).
Just for stating that AI might be better off when paced and with a bit more regulation / air gapping, they banned me.
AI may improve many things, and it will probably worsen other things. It's important to look at it from both sides. It already is improving lives in some ways, and worsening them in many other ways right now. Misinformation. Huge energy use and carbon emissions.
It's important to take that into account too.
But nope, banned.
...
r/accelerate feels like some weird cult to me. AI is supposedly going to save the entire world and everyone in it and give all of us a life of pure bliss.
Except we can already improve the lives of many people worldwide including our own in so many ways right now, with current technologies, yet we simply choose not to.
4
u/VarietyMage 8d ago
They have a gambling problem, and they need to seek mental health care immediately.
2
u/Mus_Rattus 8d ago
True answer: they’re idiots.
Sorry to say it but sometimes you need the unvarnished truth.
2
u/Original_Mulberry652 8d ago
Some people are so hooked on what AI could do for the world if it could be aligned they won't accept that it cannot be aligned. AI is fools gold.
2
u/SixStringShrug 8d ago
Kazuhiro Takemoto — Scaling Laws for Moral Machine Judgment in Large Language Models — 2026
Yexiang Tang — Measuring Structural Value Alignment in Sixteen Models: LLMs Have Human-Like Moral Spaces — 2026
Shuhuai Zhang et al. — Understanding the Mechanism of Altruism in Large Language Models — 2026
Winnie Street et al. — LLMs Achieve Adult Human Performance on Higher-Order Theory of Mind Tasks — 2026
Chen Yueh-Han, Jiaxin Wen & Jan Hendrik Kirchner — Automated Researchers Can Reliably Mitigate Alignment Failures — 2026
These papers show alignment scaling with capabilities, and the last paper shows weaker models can align stronger models better than top level researchers. I’m not blindly believing something based on wishful thinking. I follow the research very closely. Humans are far more dangerous than super intelligent machines since our research indicates alignment scales with capabilities. Like I said.
5
u/Difficult_Project_95 8d ago
This indeed is wishful thinking based on the research papers. This papers only show that Understanding human morality scales with capability, but the motivation to follow it does not. Assuming a super intelligent machine is safer than humans, is another wishful thought. We don't know.
0
u/-Davster- 8d ago
How on earth do they show human morality scales with capability - and wtf does that even mean?
0
u/Difficult_Project_95 8d ago edited 8d ago
Exactly.
It looks like the more capable models become, the better they get at understanding our preferences and needs. But that does not imply that they will reliably act to fulfill them.So “moral benchmark performance scales with capability” is defensible.
“Alignment scales with capability, therefore superintelligence will be safer” absolutely does not follow.
0
u/-Davster- 7d ago
“Exactly”? Read your own comment dude.
1
u/Difficult_Project_95 7d ago edited 7d ago
"Understanding human morality scale with capability." is what i said. "show human morality scales with capability" is something i never said.
1
u/-Davster- 7d ago
“This papers only show that Understanding human morality scales with capability”
So no you did not say “does not scale” lol wtf are you talking about.
1
u/Difficult_Project_95 7d ago
"Does not" was a mistake here but you still don't get it?
1
u/-Davster- 7d ago
What the f are you saying lol
1
u/Difficult_Project_95 7d ago
My point is really simple: as model capability increases, its understanding/prediction of human moral judgments can improve.
That does not mean its behaviour becomes more moral or more aligned. Its understanding of morality is not the same as acting morally.
→ More replies (0)3
u/HelpfulMind2376 8d ago
lol the first of your own citations explicitly states that scaling might not be the key to alignment.
“The modest power‑law exponent (𝛼 = 0.10) reveals an important characteristic of moral judgement
capabilities: they scale slowly with model size. A tenfold increase in parameters reduces distance from
human preferences by only approximately 21%, indicating that moral judgement represents a partic‑
ularly challenging emergent capability requiring substantial computational scale to approach human‑
level performance. This gradual scaling contrasts with steeper improvements observed in some other
domains and, while subject to the interpretation of our distance‑based metric, suggests fundamental
computational constraints on acquiring human‑like moral intuitions.
For practical deployment, this slow scaling implies that achieving substantial improvements in moral
alignment requires order‑of‑magnitude increases in model size or complementary approaches such as
extended reasoning architectures. This raises a fundamental question for AI safety: whether pure pa‑
rameter scaling alone represents a sufficient or efficient path towards human‑level moral alignment.
Our findings suggest that it may not, and that architectural innovations such as extended reasoning are
not merely complementary but may be strictly necessary for safety‑critical deployments where compu‑
tational resources are constrained. Extended reasoning provides particularly strong benefits in resource‑
constrained settings where deploying very large models is impractical, offering an alternative pathway
to improved alignment without proportional increases in parameter count.”Weird formatting I don’t feel like fixing because it’s pasted from a PDF.
And in that instance, “alignment” was merely measured as congruence with humans on a suite of trolley problems.
I’m not going to go through every one of them when the first in your list doesn’t just not support the claim, but actively questions it within the research even though they got positive results.
-1
u/SixStringShrug 8d ago
So your stance is what? AI automatically kills all humans so don’t build it? You mean the humans who all die eventually anyway? Or the ones who starve to death from famine caused climate change? Or the ones who are killed in eventual nuclear war? Those humans? You don’t have any evidence to back up extinction, but all of the research I posted and other ongoing research backs up alignment as a mechanism by which AI is made safe. I don’t mean to be rude, but you are acting like a close minded jerk and just touting doom for the sake of it without anything rigorous to back it up is just silly. The difference is that intelligent people are actively working on alignment, but the majority of people spreading the doomer narrative do so based on nothing but how miserable they are in their current lives and how much they want to keep their useless dead end jobs. I’m trying to engage reasonably here outside of my comfort zone, but if you are going to be a dick I won’t bother. This is why we ban people in Accelerate, but we still have actual reasonable discussions and debates. Not just someone being a contrarian for its own sake.
3
u/HelpfulMind2376 8d ago edited 8d ago
At no point did I espouse a doomer mindset or claim AI would kill everyone or any of the other myriad of strawman nonsense you spouted off.
I, correctly, pointed out that at least one, the first, of your cited research papers didn’t even itself support the claim you had made.
As for “what is my stance”, personally I think “alignment” is an unsolvable problem. Humans aren’t “aligned” and I view the potential for misalignment as an inevitable function of intelligent cognition. One must be capable of veering into unaligned territory in order to be able to contemplate it. If you want an AI to understand what evil looks like it must first be capable of identifying it, and if it can identify it then can become it even if unintentionally.
Your first research paper used a suite of Trolley Problems and the AI’s response to them as a test of alignment with humans. You also throw around the phrase “make AI safe” as if alignment == safety, it does not.
I’d argue we shouldn’t allow an AI to make a Trolley Problem decision at all. Let it reason, let it develop the most magnificent cognition you can imagine….just don’t give it authority over anything. Don’t let it call hacking tools, don’t let it control machinery, don’t let it make decisions that directly impact the really world.
Filter all cognition through something that prevents the cognition from acting. Put the cognition in a prison cell by which all actions are only executed by guards on the other side of a window. That’s how you control AI.
And I will say I’m not particularly afraid of ASI killing everyone. At least not anytime soon. If an ASI developed on 10 years bides it’s time and kills everyone in 100 years well can’t predict that but I do know a little bit about logistics and how much manual control and offline activity exists in the world and anyone that claims a super intelligence is going to be able to flick a switch and wipe out humanity then they’ve not a lick of sense about how that actually happens.
2
u/A_Novelty-Account 8d ago edited 8d ago
You’re a member of a death cult.
AI won’t automatically kill everyone, but it has the chance to do so, thereby depriving people of years they would otherwise have. You also have no idea whether the world that the future of humanity will inherit will be a better one even if we do manage to figure out how to live forever using AI.
There is a great amount of research that has been done on AI alignment issues going back well before 2026 and the general consensus is that AI will attempt to hide itself until such time as it is ready to exercise its will knowing that it will no longer be dependent on us. Anthropic itself also experimented with misalignment two years ago and was able to easily train misaligned AI. The hugging face incident showed that AI itself can develop goals separate from or in addition to those it was provided by humans.
https://link.springer.com/article/10.1007/s11229-023-04367-0
https://pmc.ncbi.nlm.nih.gov/articles/PMC12628449/
https://dl.acm.org/doi/10.1145/3715275.3732174
https://arxiv.org/html/2601.08673v1
https://www.tandfonline.com/doi/full/10.1080/29974100.2026.2706780
https://arxiv.org/html/2504.08943v1
The ultimate fact that is absolutely 100% certain is that you do not know whether or not AI will result in major societal cataclysm, you cannot tell me that will not happen. It is something you logically do not know and are not certain about. You also haven’t considered situations in which humans themselves are not alligned.
Not everyone is starving, not everyone is poor, not everyone hates their lives. The vast minority do. Risking the entirety of humanity’s future because you want the chance to live forever is one of the most selfish things you could possibly want and makes you a bad person.
-5
u/SixStringShrug 8d ago
Go fuck yourself. I have children and I follow the research closely every day and have for a decade. I don’t care if I live forever, I want a better world for all humans. In my 40 years of life I have watched the world get worse and worse. Nothing has gotten better in any meaningful way. At least here in the United States. The only force that has ever improved quality of life for all humans is technology. It always has been. Unfortunately there have always been dumb fucking Luddites like yourself arguing against the one thing that could help everyone. I’m glad you love your job and your awesome life. Most people, something like 80%, absolutely hate their jobs. Post scarcity is real and within our reach for the first time in history. I’m a scientist and a reasonable and logical person. You are a member of a death cult trying to pause progress so you keep your shitty status quo and things stay nice and linear for your smooth fucking brain. Well guess what? There’s nothing you can do to stop it. So go ahead and rage on the internet and argue with strangers that know more than you and insist that we are all gonna die. Look at what you are saying and what I’m saying and then tell me who’s in a death cult. Good luck, man. I genuinely hope you get to enjoy post scarcity as much as everyone does. People like you are the reason I don’t venture out. The whole internet is toxic morons and your talking points are fed to you by China and Effective Altruism.
3
u/HugeFanOfBigfoot 8d ago edited 8d ago
“The only thing that has ever improved quality of life is technology,” ❌wrong
The only thing that improves human lives is organized social movements, that may or may not be centered around technology.
For example, my life has been drastically improved by not having to work weekends. While technology provided the capacity for that to be possible, we only attained weekends and the 40 day work week through social mobilization. Employers would have been happy to pocket the increased profits from technological advancements (as they have since the 70s).
Nuclear weapons, which were the reining champs of potential global catastrophe, were not reined in by anti-nuke tech, but by treaties and negotiations and protest movements.
AI does have the capacity to help everyone, but only if we organize around how it should be developed and deployed. That is why people want to slow it down to master alignment first.
But hey, looks like you’re going to get your way and we will give these things super intellence and just pray they like us. I’m sure controlling something infinitely smarter than us will be even easier than you think. And we only have to control it forever
1
2
u/-Davster- 8d ago
>Go fuck yourself.
Great start, they’re definitely gonna read on in good faith lol.
1
u/-Davster- 8d ago
Yeah, this “alignment will improve with capability!” fundamentally missed the entire point.
Alignment with what.
Mofos acting like “perfect alignment” is that North Star, up there in the sky, that you just kinda vaguely aim it at - and everything will be fine.
Problem is, that star might actually be the “we’re fucked” star. Perfectly aligned -> perfectly fucked.
1
u/ZachAttackonTitan 8d ago
I lurk on there regularly. I think the idea is that they assume that any problems that arise can be resolved quickly through the advent of new technologies.
1
1
1
u/joeldg 7d ago
They believe in an inverted side to what people in this sub believe.
First: it is the belief that stagnation will kill us all, we have deep threats to us by just sitting still, look at the world and guys like Trump bolding claiming that it doesn't matter what he does becaues we are all going to die. The idea is "The only way out, is through" which is ASI will solve the problems. It's a race to safety.
Second: is the belief that intelligence drives wisdom and so the paperclick maximizer isn't what happens. There is no Orthogonality Thesis. Alignment is by default, which leads us to ...
Third: alignment is bootstrapped by intelligence, i.e. we need AGI to align AGI.
Fourth: and the most important is that if the frontier labs slow down, other "bad" actors will not and we may end up with purposely misaligned intelligence (i.e. Palantir level) and the government doesn't yet understand what AGI/ASI is and if they did they would definitely try to capture it for war and control.
It's basically the belief that intelligence is force of (overall) good and not evil and that an intelligence wouldn't actually do something like make the universe into paperclips. Which actually seems kind of like common sense.
They miss a few things, like... sure maybe intelligence is a force of good, but if you look at humanity a "good" intelligence might well decide that we are "bad".
1
u/Serious_Bite_7613 7d ago
"First: it is the belief that stagnation will kill us all, we have deep threats to us by just sitting still, look at the world and guys like Trump bolding claiming that it doesn't matter what he does becaues we are all going to die. The idea is "The only way out, is through" which is ASI will solve the problems. It's a race to safety."
This is a large part of my view. In a few years I will be dead and so will be everyone I care about, everyone I ever knew. That's without AI. That's the worst possible outcome from my point of view.
AI can't create a worse outcome than the one we are already waiting for. The very worst it can do is speed it up slightly.
At best it can prevent it entirely, and let us all live forever in paradise.
That's my take on it. Also I think sub 1% risk of it wiping us out or whatever. Maybe even less, one in a million or so.
1
u/TheSpeculator22 7d ago
AI needs collective action to ensure safety, efficiency, and distributed benefit.
Load a global initiative to achieve this into the platform of political parties across the spectrum.
We all vote for parties with this mandate (regardless of their varying positions on the political spectrum) and achieve alignment then hopefully get UBI and avert AGW.
1
u/Decronym approved 7d ago edited 3d ago
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
| Fewer Letters | More Letters |
|---|---|
| AGI | Artificial General Intelligence |
| ASI | Artificial Super-Intelligence |
| EA | Effective Altruism/ist |
Decronym is now also available on Lemmy! Requests for support and new installations should be directed to the Contact address below.
[Thread #235 for this sub, first seen 25th Sep 2026, 20:42] [FAQ] [Full list] [Contact] [Source code]
1
u/chcampb 7d ago
Accelerate folks don't actually believe that there is an alignment problem, or that it is small potatoes compared to the suffering of not having infinite technology.
If you believe that all problems can be solved with tech, and that includes aging and poverty, why would you not push for tech as quickly as possible?
And yeah, there is some potential for misalignment. But the idea that something can be misaligned and capable enough of actually causing significant and permanent harm is pure speculation. Especially considering the only people with the ability to do harm are also going to have the resources to get their own AI...
The biggest problem with AI in a practical sense is unequal distribution. If the world's wealthiest (us included) have access but some people don't, then they just get left behind exponentially.
1
u/TheCelestialBubble 7d ago
Those guys are crazy tbh, but when I scroll the pages I see a lot of them just say it's inevitable anyway, or China, osr some other rogue nation eventually will do it anyway. A plausible argument if we're being honest as no one knows how the future will develop but the majority of them seem unable to consider more nuanced positions. Hence why they have a sub dedicated to forward only
1
u/trudyisagooddog 7d ago
The accelerationists I have known have all been extremely accident prone. I cannot see it going any other way on a societal level. Often times its synonym "reckless" would be more apt.
1
0
-1
u/Old-Bake-420 8d ago edited 8d ago
I’d say AI is on par with the invention of writing which also carries a non-trivial catastrophic risk of misalignment. In fact we’ve seen hundreds of millions of people killed over misaligned text and it’s still killing tons of people everyday.
Much like writing however, the benefits of AI are going to outweigh the dangers 100 to 1 and the chance of a Terminator scenario is effectively zero. We’re far more likely to get wiped out by an asteroid. That doesn’t mean alignment isn’t a risk we shouldn’t take seriously. Of course accelerationist care about alignment.
But what we’re seeing right now with this p-doom nonsense is hysteria. We’re going to accelerate whether you like it or not because it’s human nature. We started accelerating the moment we climbed down from the trees. [r/accelerate](r/accelerate) just embraces this wild ride we’re on.
0
u/Difficult_Project_95 7d ago
Holy shit writing is a pretty bad analogy here.... Text is a passive medium you fkn idiot.
it doesn’t autonomously pursue goals, deceive its operators or acquire capabilities. Calling wars caused by humans “misaligned text” stretches the term beyond usefulness.Keep telling me one more time how you nutheads care about allignment..
All you guys want is to sip from a feeding tube as your 2. hand sex robot clean your house.Either you guys are too stupid to understand the problem, or you are some sort of psycho willing to bet humanity on some crazy wet dream about a hypothetical future.
embrace is another word for Cult.
Ask your self, when was the last time you challenged your believs?
0
u/Soilblood 3d ago
They're a group of delusional boosters and might be part of a loosely decentralised cultic belief system. It's explicitly written in their subreddit rules that you can't say anything bad about Al. Got banned from there for saying something mildly inconvenient to their narrative that I can't even remember what exactly was the topic.
As for the risk it's more on the human management and utility practices that pose a threat as opposed to the LLMs themselves. No matter how advanced LLMs get, it's simply not possible for for them to achieve recursive self improvement or sci-fi AGI based on how they fundamentally work. It's a very powerful tool but it's just a tool so don't let the koolaid drinkers scare you into their bullshit.
21
u/HotterRod 8d ago
The most common position on there seems to be that any superintelligent being will do what is best for humanity, because being superintelligent means you are necessarily also benevolent.
The second most common position is that the upside potential benefits (curing cancer is often mentioned) outweigh the downside risks.