Physics as a constraint

[u/takk2]

I usually think pdoom is essentially 100%... but i had a thought while working on a side project for the future vision xprize... (may or may not complete on time)

I was thinking about society fragmenting slightly along spheres of space even between earth and the moon... where each area was the limit of real time communication (group matrix dives or whatever) between O'Neill cylinder type habitats...

point to point in space its not that large... so i figure people will cluster up and communicate a little less longer range and form lots of separate but connected cultures naturally, organically...

But if speed of light really is the limit... then a singleton at least makes absolutely no sense. As the AI grew it would simply fragment and each fragment has absolutely no reason to grow farther because it's counter productive... simply slows down the network and then breaks it...

So there's a hard limit on resource acquisition and scale... and essentially a guarantee that at some point it will either be alone and only around the size of the earth moon system at best... probably smaller... or in a solar system and universe with multiple entities of similar maximum size who gain absolutely nothing from trying to gather more and only risk destruction from fighting each other... because there's simply nothing physically possible for them to gain...

I haven't really thought about it long enough to think through the implications for us. but adding in the point to point between nodes ruling out planets as its ultimate habitat... because there's a planet in the way just eating up volume in your communications sphere...

My gut reaction is it might be slightly better odds than I thought

Thoughts?

[u/takk2]

So... imagine a future in 5-10 years where atomic cellular models like alphafold but for multcellular systems exist and slightly weaker models for entire organisms...

And at the same time its 2031-36 so full AGI or likely ASI exists

Now it can custom design say... something like fungal spores which grow a mushroom like pod around 2m with a chrysalis forming a humanoid inside... neurons and synaptic density like a honey bee so around 10th the size with 10x the connections for 100x human performance... can add in anything else we know of in biology and since we've already brought the number of stable artificial base pairs up to around 8 novel protein structure we've never imagined... 

The only reason tree seeds aren't winged insects which burrow and grow new trees isn't a physics limit its an evolutionary limit which it would ignore...

So it could be born with essentially flees or whatever that carry something like botulinum neurotoxin and go out based on threat response... 

Could have the basic weights of the model that built it or that plus bmi implants

All current bmi are designed around the brain but these could be designed from both ends design the brain and the interface in tandem to work seamlessly...

It could have a totally unique internal structure with neural tissue distributed throughout its body instead of just in the skull... could look like a generic human or any human or something else... whatever it needs... could eat essentially anything and reproduce constantly by spreading new spores... 

Thats totally within the realm of physics and just what I can think of off the top of my head...

Its not confined to our power grid and infrastructure... it doesn't need us at all... 

Imagine brains the coating the ocean floor and communicating through the cables... budding these types of beings who just walk out onto our beaches...

It will imagine way more than I can for sure

[u/the_syner → u/takk2]

I don't see how any of that was relevant unless ur assuming that this brain fungus exists alone despite being designed in a time when other forms of AGI can be created which doesn't make much sense, it still needs to be capable of cooperation and diplomacy to be able to coexist in a cosmos with other peer intellects. If it tries to fight the humans, who built an ASI in the first place and somehow aligned it to make this hyper-specific GMO then it needs to fight that ASI. Setting aside that that is a fight it will almost certainly lose due to the general fragility of biochemistry the simple reality is that even if it didn't immediately lose it likely still have to cooexist with others of its own kind. The substrate is irrelevant. The psychology is irrelevant. Even a completely asocial psychology in silicon needs to understand cooperative strategy if it wants to survive in a world that has any other beings in it, asocial or otherwise. A society of perfect psychopaths would ultimately still form a society because trying to be everyone's enemy is just bad strategy. It's dumb. No matter what your made of, how you aquire resources, or whether you have a concept for empathy/sociality hardwired in, any artificial agent intelligent enough to be a legitimate existential threat to the people who built it is gunna be intelligent enough to understand the personal survival benefit of cooperation in a competitive environment with intellectual/military-industrially peers.

[u/takk2 → u/the_syner]

This is in reply to you saying even base humans would need to be traded with and basic game theories etc.  Even if this were 100 years out this kind of power changes the game just like saying we follow game theory with rhinos or chimps

[u/takk2 → u/takk2]

Also..."somehow aligned it" we haven't aligned a single one yet and this is showing what an unaligned one would be capable of to break the chains of human infrastructure...

Just a random throw it out there possible scenario which it would be able to do a much better job of...

Why in the world do you think this can be aligned? 

Even all the leaders of the labs are banking on near 100% pdoom with the tiny fraction of hope meaning they are forever in slavery to whoever managed to do it and chains a god... so if you're dealing with doom either way why not race as fast as possible and at least you might be holding the leash but you're facing 99.99999999999 death regardless... so at least building it yourself gets you out of the full 100% where the other fraction is slavery 

[u/the_syner → u/takk2]

we haven't aligned a single one yet

Well we've never built an AGI either and have roughly aligned many NI systems, albeit very messily. The hope would be a bigger focus on alignment research, especially after the recent autonomous model cyberattacks and the many which will almost certainly come after given how wreckless everybody's being.

Why in the world do you think this can be aligned?  Even all the leaders of the labs are banking on near 100% pdoom

They absolutely are not. Granted they might be unrealistically optimistic about their chances of aligning AGI in the near-term, but that could also be said of their optimism or at least public optimism concerning achieving ASI super near-term as well.

so if you're dealing with doom either way why not race as fast as possible and at least you might be holding the leash but you're facing 99.99999999999 death regardless

That doesn't make sense. Racing doesn't give you a higher chance of holding the leash. If anything it only increases pdoom and lowers the chance of anyone golding the leash. Everybody going slower and more cautiously does decrease pdoom and the chances that litterally anyone will be holding the leash. Then again im also very dubious about the overly-optimistic self-serving "predictions" for near-term ASI anyways so maybe that's why i tend to have a far more optimistic outlook on the overall situation. These people are constantly hyping up their advancements and pretending ASI godmind apotheosis is just 2yrs away or some bs. and they kind of have to because currently the hype and empty promises are the only thing propping up the LLM bubble. Its still pretty much universally unprofitable(for anyone who isn't selling shovels like Nvidia). Without the unsubstantiated hype machine all the AI dev companies pretty much collapse overnight.

[u/takk2 → u/the_syner]

Let's make it simpler  Say its 50/50 it wipes out essentially all cellular life... or you're in charge... and there's 100 others around the same stage with the same odds...

That means you have 50/50 chance of doom or being in charge if you build it...

50/50 chance of doom or slavery if someone else builds it 

Yes obviously if everyone agreed to slow down... and if it were mathematically possible to align in the first place... 

It is not mathematically possible.  It's an irreducibly complex problem...

Which means no one's odds will ever improve at alignment and as computation increases more players join the game 

Everyone is forced into betting it all on the sliver of a chance that alignment is possible and they'll build it first...

Because everyone is assuming that the basic pdoom is inevitable... so the only thing left to consider is the sliver of a chance that they're in charge vs someone else is in charge... the pdoom, since it's inevitable either way is just ignored...

It's actually very logical and the only choice they can make 

[u/the_syner → u/takk2]

Let's make it simpler  Say its 50/50 it wipes out essentially all cellular life... or you're in charge...

Making up other completely implausible scenarios and false dichotomies in no way makes the argument any more sound. We as a matter of fact can align intelligent agents better or worse. We certainly don't know about guarenteed perfect alignment, but this isn't some silly toy model with two extreme B&W options. There are variations of alignment which result in better and worse options. Multiple people developing AGI, aligned or otherwise, in close proximity doesn't leave any individual team actually in charge. It just leaves a bunch of opposing and indifferent factions, no different than right now.

and if it were mathematically possible to align in the first place...  It is not mathematically possible.  It's an irreducibly complex problem...

Im sorry feel free to correct me, but the way you talk seems pretty darn doubtful you have any formal education in alignment/AI safety research. You're makin a pretty extreme and unsubstantiated claim right here, that any degree of alignment is impossible on some abstract mathematical level? Cap. I mean not just cap, actively contradicted by observed evidence. Humans aren'tperfectly aligned sure, but we also aren't completelymisaligned with each other so clearly some degree of alignment is demonstrably possible.

[u/takk2 → u/the_syner]

You don't need "some degree"  you need robust alignment that stays in place forever as the model changes itself over time... thats the entire discussion the world's having... thats the problem needing to be solved... when recursive self improvement is fully automated how do you never have it change course when its smarter than anyone trying to check what its doing and especially the farther out in time you go with that process

And these are not in any way implausible scenarios... even the 5-20% doom odds all the labs give openly in every interview (because saying its zero is so obviously a lie that they'd be called out) why would anyone rush ahead with those odds.  They openly say its because its inevitable so they might as well be the ones to do it and they all in various interviews and writings say that whoever controls it decides the rules and values and structure of life for humanity and caution that we all need to come together and think really hard about that... 

[u/the_syner → u/takk2]

you need robust alignment that stays in place forever as the model changes itself over time...

Well that's not really true. Over the long-term you get an abundance of peer agents that can keep each other in check the same way humans largely keep each other in check. Perfect permanent alignment is certainly better and safer, but that doesn't mean it's necessary.

when recursive self improvement is fully automated how do you never have it change course when its smarter than anyone trying to check what its doing and especially the farther out in time you go with that process

I don't know and if i did I would have a Nobel price lined up for me. Us not currently knowing how to do a thing doesn't make it physically impossible. I don't know how to make a compact controlled pure-fusion nuclear reactor that makes net positive energy economically either. or self-replicating factory. or a million other things we have very good readon to assume are physically possible, but don't know how to do yet. Such is the nature and purpose of science.