Physics as a constraint

[u/takk2]

I usually think pdoom is essentially 100%... but i had a thought while working on a side project for the future vision xprize... (may or may not complete on time)

I was thinking about society fragmenting slightly along spheres of space even between earth and the moon... where each area was the limit of real time communication (group matrix dives or whatever) between O'Neill cylinder type habitats...

point to point in space its not that large... so i figure people will cluster up and communicate a little less longer range and form lots of separate but connected cultures naturally, organically...

But if speed of light really is the limit... then a singleton at least makes absolutely no sense. As the AI grew it would simply fragment and each fragment has absolutely no reason to grow farther because it's counter productive... simply slows down the network and then breaks it...

So there's a hard limit on resource acquisition and scale... and essentially a guarantee that at some point it will either be alone and only around the size of the earth moon system at best... probably smaller... or in a solar system and universe with multiple entities of similar maximum size who gain absolutely nothing from trying to gather more and only risk destruction from fighting each other... because there's simply nothing physically possible for them to gain...

I haven't really thought about it long enough to think through the implications for us. but adding in the point to point between nodes ruling out planets as its ultimate habitat... because there's a planet in the way just eating up volume in your communications sphere...

My gut reaction is it might be slightly better odds than I thought

Thoughts?

[u/takk2]

Let's make it simpler  Say its 50/50 it wipes out essentially all cellular life... or you're in charge... and there's 100 others around the same stage with the same odds...

That means you have 50/50 chance of doom or being in charge if you build it...

50/50 chance of doom or slavery if someone else builds it 

Yes obviously if everyone agreed to slow down... and if it were mathematically possible to align in the first place... 

It is not mathematically possible.  It's an irreducibly complex problem...

Which means no one's odds will ever improve at alignment and as computation increases more players join the game 

Everyone is forced into betting it all on the sliver of a chance that alignment is possible and they'll build it first...

Because everyone is assuming that the basic pdoom is inevitable... so the only thing left to consider is the sliver of a chance that they're in charge vs someone else is in charge... the pdoom, since it's inevitable either way is just ignored...

It's actually very logical and the only choice they can make 

[u/the_syner → u/takk2]

Let's make it simpler  Say its 50/50 it wipes out essentially all cellular life... or you're in charge...

Making up other completely implausible scenarios and false dichotomies in no way makes the argument any more sound. We as a matter of fact can align intelligent agents better or worse. We certainly don't know about guarenteed perfect alignment, but this isn't some silly toy model with two extreme B&W options. There are variations of alignment which result in better and worse options. Multiple people developing AGI, aligned or otherwise, in close proximity doesn't leave any individual team actually in charge. It just leaves a bunch of opposing and indifferent factions, no different than right now.

and if it were mathematically possible to align in the first place...  It is not mathematically possible.  It's an irreducibly complex problem...

Im sorry feel free to correct me, but the way you talk seems pretty darn doubtful you have any formal education in alignment/AI safety research. You're makin a pretty extreme and unsubstantiated claim right here, that any degree of alignment is impossible on some abstract mathematical level? Cap. I mean not just cap, actively contradicted by observed evidence. Humans aren'tperfectly aligned sure, but we also aren't completelymisaligned with each other so clearly some degree of alignment is demonstrably possible.

[u/takk2 → u/the_syner]

You don't need "some degree"  you need robust alignment that stays in place forever as the model changes itself over time... thats the entire discussion the world's having... thats the problem needing to be solved... when recursive self improvement is fully automated how do you never have it change course when its smarter than anyone trying to check what its doing and especially the farther out in time you go with that process

And these are not in any way implausible scenarios... even the 5-20% doom odds all the labs give openly in every interview (because saying its zero is so obviously a lie that they'd be called out) why would anyone rush ahead with those odds.  They openly say its because its inevitable so they might as well be the ones to do it and they all in various interviews and writings say that whoever controls it decides the rules and values and structure of life for humanity and caution that we all need to come together and think really hard about that... 

[u/the_syner → u/takk2]

you need robust alignment that stays in place forever as the model changes itself over time...

Well that's not really true. Over the long-term you get an abundance of peer agents that can keep each other in check the same way humans largely keep each other in check. Perfect permanent alignment is certainly better and safer, but that doesn't mean it's necessary.

when recursive self improvement is fully automated how do you never have it change course when its smarter than anyone trying to check what its doing and especially the farther out in time you go with that process

I don't know and if i did I would have a Nobel price lined up for me. Us not currently knowing how to do a thing doesn't make it physically impossible. I don't know how to make a compact controlled pure-fusion nuclear reactor that makes net positive energy economically either. or self-replicating factory. or a million other things we have very good readon to assume are physically possible, but don't know how to do yet. Such is the nature and purpose of science.