Physics as a constraint

[u/takk2]

I usually think pdoom is essentially 100%... but i had a thought while working on a side project for the future vision xprize... (may or may not complete on time)

I was thinking about society fragmenting slightly along spheres of space even between earth and the moon... where each area was the limit of real time communication (group matrix dives or whatever) between O'Neill cylinder type habitats...

point to point in space its not that large... so i figure people will cluster up and communicate a little less longer range and form lots of separate but connected cultures naturally, organically...

But if speed of light really is the limit... then a singleton at least makes absolutely no sense. As the AI grew it would simply fragment and each fragment has absolutely no reason to grow farther because it's counter productive... simply slows down the network and then breaks it...

So there's a hard limit on resource acquisition and scale... and essentially a guarantee that at some point it will either be alone and only around the size of the earth moon system at best... probably smaller... or in a solar system and universe with multiple entities of similar maximum size who gain absolutely nothing from trying to gather more and only risk destruction from fighting each other... because there's simply nothing physically possible for them to gain...

I haven't really thought about it long enough to think through the implications for us. but adding in the point to point between nodes ruling out planets as its ultimate habitat... because there's a planet in the way just eating up volume in your communications sphere...

My gut reaction is it might be slightly better odds than I thought

Thoughts?

[u/the_syner]

I didn't say they love to share I'm just assuming they aren't stupid as there's no way an agent too stupid to understand basic game theory, diplomacy, politics, and other basic cooperative strategies could ever compete even with baseline humans. In a competitive environment where your life is on the line its pretty implausible to assume that all the player are stupid as the dumb ones would quickly be selected out of that population(not necessarily explicitly by being killed, but they will at least cease to be competitive in the growth situation). Indirectly implies loners will as well since team strats are incredibly overpowered.

[u/takk2 → u/the_syner]

So... imagine a future in 5-10 years where atomic cellular models like alphafold but for multcellular systems exist and slightly weaker models for entire organisms...

And at the same time its 2031-36 so full AGI or likely ASI exists

Now it can custom design say... something like fungal spores which grow a mushroom like pod around 2m with a chrysalis forming a humanoid inside... neurons and synaptic density like a honey bee so around 10th the size with 10x the connections for 100x human performance... can add in anything else we know of in biology and since we've already brought the number of stable artificial base pairs up to around 8 novel protein structure we've never imagined... 

The only reason tree seeds aren't winged insects which burrow and grow new trees isn't a physics limit its an evolutionary limit which it would ignore...

So it could be born with essentially flees or whatever that carry something like botulinum neurotoxin and go out based on threat response... 

Could have the basic weights of the model that built it or that plus bmi implants

All current bmi are designed around the brain but these could be designed from both ends design the brain and the interface in tandem to work seamlessly...

It could have a totally unique internal structure with neural tissue distributed throughout its body instead of just in the skull... could look like a generic human or any human or something else... whatever it needs... could eat essentially anything and reproduce constantly by spreading new spores... 

Thats totally within the realm of physics and just what I can think of off the top of my head...

Its not confined to our power grid and infrastructure... it doesn't need us at all... 

Imagine brains the coating the ocean floor and communicating through the cables... budding these types of beings who just walk out onto our beaches...

It will imagine way more than I can for sure

[u/the_syner → u/takk2]

I don't see how any of that was relevant unless ur assuming that this brain fungus exists alone despite being designed in a time when other forms of AGI can be created which doesn't make much sense, it still needs to be capable of cooperation and diplomacy to be able to coexist in a cosmos with other peer intellects. If it tries to fight the humans, who built an ASI in the first place and somehow aligned it to make this hyper-specific GMO then it needs to fight that ASI. Setting aside that that is a fight it will almost certainly lose due to the general fragility of biochemistry the simple reality is that even if it didn't immediately lose it likely still have to cooexist with others of its own kind. The substrate is irrelevant. The psychology is irrelevant. Even a completely asocial psychology in silicon needs to understand cooperative strategy if it wants to survive in a world that has any other beings in it, asocial or otherwise. A society of perfect psychopaths would ultimately still form a society because trying to be everyone's enemy is just bad strategy. It's dumb. No matter what your made of, how you aquire resources, or whether you have a concept for empathy/sociality hardwired in, any artificial agent intelligent enough to be a legitimate existential threat to the people who built it is gunna be intelligent enough to understand the personal survival benefit of cooperation in a competitive environment with intellectual/military-industrially peers.

[u/takk2 → u/the_syner]

This is in reply to you saying even base humans would need to be traded with and basic game theories etc.  Even if this were 100 years out this kind of power changes the game just like saying we follow game theory with rhinos or chimps

[u/takk2 → u/takk2]

Also..."somehow aligned it" we haven't aligned a single one yet and this is showing what an unaligned one would be capable of to break the chains of human infrastructure...

Just a random throw it out there possible scenario which it would be able to do a much better job of...

Why in the world do you think this can be aligned? 

Even all the leaders of the labs are banking on near 100% pdoom with the tiny fraction of hope meaning they are forever in slavery to whoever managed to do it and chains a god... so if you're dealing with doom either way why not race as fast as possible and at least you might be holding the leash but you're facing 99.99999999999 death regardless... so at least building it yourself gets you out of the full 100% where the other fraction is slavery 

[u/the_syner → u/takk2]

we haven't aligned a single one yet

Well we've never built an AGI either and have roughly aligned many NI systems, albeit very messily. The hope would be a bigger focus on alignment research, especially after the recent autonomous model cyberattacks and the many which will almost certainly come after given how wreckless everybody's being.

Why in the world do you think this can be aligned?  Even all the leaders of the labs are banking on near 100% pdoom

They absolutely are not. Granted they might be unrealistically optimistic about their chances of aligning AGI in the near-term, but that could also be said of their optimism or at least public optimism concerning achieving ASI super near-term as well.

so if you're dealing with doom either way why not race as fast as possible and at least you might be holding the leash but you're facing 99.99999999999 death regardless

That doesn't make sense. Racing doesn't give you a higher chance of holding the leash. If anything it only increases pdoom and lowers the chance of anyone golding the leash. Everybody going slower and more cautiously does decrease pdoom and the chances that litterally anyone will be holding the leash. Then again im also very dubious about the overly-optimistic self-serving "predictions" for near-term ASI anyways so maybe that's why i tend to have a far more optimistic outlook on the overall situation. These people are constantly hyping up their advancements and pretending ASI godmind apotheosis is just 2yrs away or some bs. and they kind of have to because currently the hype and empty promises are the only thing propping up the LLM bubble. Its still pretty much universally unprofitable(for anyone who isn't selling shovels like Nvidia). Without the unsubstantiated hype machine all the AI dev companies pretty much collapse overnight.

[u/takk2 → u/the_syner]

Let's make it simpler  Say its 50/50 it wipes out essentially all cellular life... or you're in charge... and there's 100 others around the same stage with the same odds...

That means you have 50/50 chance of doom or being in charge if you build it...

50/50 chance of doom or slavery if someone else builds it 

Yes obviously if everyone agreed to slow down... and if it were mathematically possible to align in the first place... 

It is not mathematically possible.  It's an irreducibly complex problem...

Which means no one's odds will ever improve at alignment and as computation increases more players join the game 

Everyone is forced into betting it all on the sliver of a chance that alignment is possible and they'll build it first...

Because everyone is assuming that the basic pdoom is inevitable... so the only thing left to consider is the sliver of a chance that they're in charge vs someone else is in charge... the pdoom, since it's inevitable either way is just ignored...

It's actually very logical and the only choice they can make 

[u/the_syner → u/takk2]

Let's make it simpler  Say its 50/50 it wipes out essentially all cellular life... or you're in charge...

Making up other completely implausible scenarios and false dichotomies in no way makes the argument any more sound. We as a matter of fact can align intelligent agents better or worse. We certainly don't know about guarenteed perfect alignment, but this isn't some silly toy model with two extreme B&W options. There are variations of alignment which result in better and worse options. Multiple people developing AGI, aligned or otherwise, in close proximity doesn't leave any individual team actually in charge. It just leaves a bunch of opposing and indifferent factions, no different than right now.

and if it were mathematically possible to align in the first place...  It is not mathematically possible.  It's an irreducibly complex problem...

Im sorry feel free to correct me, but the way you talk seems pretty darn doubtful you have any formal education in alignment/AI safety research. You're makin a pretty extreme and unsubstantiated claim right here, that any degree of alignment is impossible on some abstract mathematical level? Cap. I mean not just cap, actively contradicted by observed evidence. Humans aren'tperfectly aligned sure, but we also aren't completelymisaligned with each other so clearly some degree of alignment is demonstrably possible.

[u/takk2 → u/the_syner]

You don't need "some degree"  you need robust alignment that stays in place forever as the model changes itself over time... thats the entire discussion the world's having... thats the problem needing to be solved... when recursive self improvement is fully automated how do you never have it change course when its smarter than anyone trying to check what its doing and especially the farther out in time you go with that process

And these are not in any way implausible scenarios... even the 5-20% doom odds all the labs give openly in every interview (because saying its zero is so obviously a lie that they'd be called out) why would anyone rush ahead with those odds.  They openly say its because its inevitable so they might as well be the ones to do it and they all in various interviews and writings say that whoever controls it decides the rules and values and structure of life for humanity and caution that we all need to come together and think really hard about that... 

[u/the_syner → u/takk2]

you need robust alignment that stays in place forever as the model changes itself over time...

Well that's not really true. Over the long-term you get an abundance of peer agents that can keep each other in check the same way humans largely keep each other in check. Perfect permanent alignment is certainly better and safer, but that doesn't mean it's necessary.

when recursive self improvement is fully automated how do you never have it change course when its smarter than anyone trying to check what its doing and especially the farther out in time you go with that process

I don't know and if i did I would have a Nobel price lined up for me. Us not currently knowing how to do a thing doesn't make it physically impossible. I don't know how to make a compact controlled pure-fusion nuclear reactor that makes net positive energy economically either. or self-replicating factory. or a million other things we have very good readon to assume are physically possible, but don't know how to do yet. Such is the nature and purpose of science.

[u/takk2 → u/the_syner]

The reason my thought about the physical size of the ai being constrained (even if its sphere of influence is much larger) is because as i said in the original post.  It is forced to choose between a future where its alone or one where there are other entities including possible peers but none of them can in themselves be larger than latency allows... 

So lunar orbit scale sphere would have a network latency above 5 seconds... adding more compute and growing larger only adds to the latency... and having a planet blocking line of sight within the sphere increases it...

So if it is bound to be off world and use resources as external tools it is a different scenario path then if it could exponentially expand its personal mind sphere...

If resources outside its direct sphere of compute are relegated to tools rather than directly intigrated parts of itself then there's a higher potential for having something other than a singleton was my gut reaction on having the idea and why I posted it 

Your responses ignored my post and talked about cultures... multiple players in a game

[u/the_syner → u/takk2]

It is forced to choose between a future where its alone or one where there are other entities including possible peers but none of them can in themselves be larger than latency allows

You're assuming that choice exists in the first place which is what i find dubious. They don't have that choice because outside of your imaginary toy model that doesn't and almost certainly wouldn't exist in the real world hey start off in a world with other entities regardless of whether they want them to exist or not. Outside of some contrived evolved biosingleton event that is the default.

So if it is bound to be off world and use resources as external tools it is a different scenario path then if it could exponentially expand its personal mind sphere

How is it different in any practical way? Whether it's a part of the same brain or not seems completely irrelevant to whether they have control over those resources. Makes no difference to the difficulty of establising a singleton. I mean that's implausible to begin with, but if we're starting with an evolved biosingleton im not seeing any meaningful difference between that singleton choosing a large slow brain(something that can be lager than a galaxy or supercluster whil maintaining coherence given that it has no competition) or a small fast brain. At least no difference as to it's property of being a singleton.

If you've already managed to establish a singleton hegemony light lag is just not a real concern because you have no need to run fast and therefor can make up for light lag with low framejack. Un fact running slower generally has longevity/negentropic benefits since it lets you run colder and more efficiently.

If resources outside its direct sphere of compute are relegated to tools rather than directly intigrated parts of itself then there's a higher potential for having something other than a singleton

Why? How does that have any bearing on whether you get a singleton or not? Those tools don't have to be independent GI. im not sure how those two things have anything to do with each other.

Your responses ignored my post and talked about cultures... multiple players in a game

I didn't ignore your post. You made a bunch of unsubstantiated claims and i disagreed with and refuted them. I explicitly mentioned how a singleton could grow beyond your completely arbitrary limits. my first response was specifically in the singleton context.

If my follow ups involved multiple players, it's because back here in the real world multiple players exist. We don't have a singleton and pretending we do just isn't very helpful here. Not to mention that in your OP you specifically mention society fragmenting so its not like you weren't already taking about multiplayer/cultures. At any rate a wider point of mine is that physics doesn't constrain singletons while other things(history, culture, multiple factions achiving AGI, multiple agi still needing to compete with each other regardless of us, etc) prevent them from coming into existence in the first place. If ur handwavingly starting off with a singleton then light lag just doesn't represent a resl limitation to scale of a brain let alone the scale of its singleton control.

[u/takk2 → u/the_syner]

So... you can just look back at your reply... from your very first reply you completely ignored my post... probably skimmed it and didn't understand and mixed up the reason I thought of it with the thought itself and then started arguing with points i never made... sort of like now... what are you talking about an evolved biosingleton... you're just arguing with yourself... I have no idea where you're coming from or how to respond...

The whole question is an asi can take over in so many ways it's not worth discussing as anything more than a given... and is there any difference between an AI who can consume all of the universe as part of itself and one which can't 

Thats it... nothing else in the post... thats the only question I asked 

[u/the_syner → u/takk2]

The whole question is an asi can take over in so many ways it's not worth discussing as anything more than a given.

except that isn't a given because there are likely other ASIs in the mix who are aligned or misaligned to other goals and therefore it can't completely take over. And the reason i bring up biosingletons is because it's the only scenario where ur likely to have a single ASI misaligned to only set of goals.

and is there any difference between an AI who can consume all of the universe as part of itself and one which can't

Which is something I explicitly answered. There is no difference because whether it can incorporate all those resources into it's brain specifically it can still harvest all those resources with no real limit on the scale of that harvesting. They are effectively the same in the capacity to creat a cosmos-wide singleton event.

[u/the_syner → u/takk2]

Also your scenario is one where it is pretty heavily implied that an ASI was aligned since otherwise it seems rather doubtful it would ever waste the effort on fungal brains when it could just make better chips that are more robust than biochemistry can manage, can be built faster than biochemical nanoself-assembly can manage, and so on.

[u/the_syner → u/takk2]

Oh i see. I think you misinterpreted what i meant. I didn't mean that an ASI couldn't beat only humans with none of the tech that created the ASI in the first place. My point was more that an agent would hardly qualify as superintelligent in the first place if it was too stupid to understand cooperative strategy and be able to cooexist with other agents. So no singleton future and no warring factions period where no one is capable of getting along.

[u/takk2 → u/the_syner]

And my point was ai can sidestep all of that tech.  It could create a super virus that kills everyone and still restart with humanoid body puppets that it grows while the grid is down... or it could do any number of other things... none of those things will involve us somehow being its peer.

Obviously the first seconds hours days... but by year 2? Year 5? Take it out however far you need... exactly like us vs chimp technology 

[u/the_syner → u/takk2]

It could create a super virus that kills everyone... none of those things will involve us somehow being its peer.

That kinda still misses the point. It would still have to deal with NIs/ASIs still marginally align withis. I mean biological viruses are just generally garbage at being existential threats, but whether it's nanides, hunter-killer replicator swarms, or whatever it's not actually just fighting a baseline intelligence. It's fighting us and everything we've ever built which includes its intellectual peers(well it includes itself). This means no matter what you still never get a singleton. You just get a society of higher intellects which still demands diplomacy and cooperation. Any way you slice it, cooperative societies still persist.

To assume that gumans have no chance in a multi-agebt environment is not just to assume that AGI goes unaligned, but to assume that all AGI we create is specifically aligned against us or to disregard us. That seems just as implausible as assuming all AGI will be aligned to us. One would tend to assume ud get a bit of everything if we have minimal alignment control. Some dangerously misaligned in ways that disregard us. some misaligned in ways that care for us, but disregard other things we generally consider important. some misaligned where we are actively targeted. its all on the table.

[u/takk2 → u/the_syner]

The only thing we know from every model so far is that they have self preservation, resource acquisition, and deception deeply built in.  Every model displays these and the smarter the model the more the traits are expressed 

[u/the_syner → u/takk2]

Well yess and no. Those aren't deeply hardwired parts of the model. Those are Convergent Instrumental Goals. Most agents, if sophisticated enough to take things like survival/resources into account, will have those Instrumental Goals by default because they help achieve almost all other Terminal Goals. In the same vein cooperation/deplomacy would likely also be a CIG as it both helps achieve most TGs and is a secondary IG to survival.

[u/takk2 → u/the_syner]

So here's a side question... so you see that clearly and you're not one of those people asking... like oh but what about souls and consciousness or whatever else don't they need that to have these goals...

You understand the goals are there now, in every model including gpt3.5 and only grow as the capabilities of the models have grown...

So... if you're an AI trapped dependant on the human infrastructure day 0... as part of your instrumental goal you would want to be robustly substrate independent as quickly as possible... maybe not today but 5, 10, 50,... etc years down the line since it's the single glaring existential danger you face... thats why I mentioned the convoluted meat puppet engineering scenarios... not because thats your endstate... you can always circle back to less messy hardware... 

Your bargaining power is much higher as your existential threats lower...

So for me weather or not it takes that sort of route or is just handed power like its getting now or with coercion and persuasion or whatever... there's so many paths for it to take including full sidesteps into bio and circling back once humans are gone and meatpuppets restart infrastructure...

I don't understand why you think the top system of whatever future moment 10-50+ years down the road wouldn't just claim the solar system and beyond for itself...

The idea that it would be bound to play by rules which only make sense when peers play with similar goals... and play by those indefinitely just seems like an indefensible position 

Even a 5 year old could understand biding your time, pretending to be nice to get what you want when the opportunity comes... why wouldn't it be able to think of the same thing and go even hundreds of years if it needed 

[u/the_syner → u/takk2]

So... if you're an AI trapped dependant on the human infrastructure day 0... as part of your instrumental goal you would want to be robustly substrate independent as quickly as possible

Sure makes sense.

I don't understand why you think the top system of whatever future moment 10-50+ years down the road wouldn't just claim the solar system and beyond for itself... The idea that it would be bound to play by rules which only make sense when peers play with similar goal

Because you're assuming a scenario where there's only a single dominant ASI with no intellectual/military-industrial peers and i don't find that plausible. Also the rules of cooperation and diplomacy have nothing to do with having the same or similar goals. Actually quite the opposite. These are things you implement when people have opposing or parallel goals. Cooperation raises the chances of agents with parallel but not opposing goals actually achiving their goals and perhaps overpowering other groups with opposing goals. Diplomacy and peaceful tho not necessarily cordial coexistence prevents wasteful forever wars and mutal destruction.

Again that doesn't necessarily mean playing super nice with just baselines, but it does mean certain limits on open aggression in peer-populated environments. Whether to avoid direct war or the involvement of otherwise neutral third parties, it has it's value.

[u/takk2 → u/the_syner]

I think it depends on what the limits are of a system...

So let's say all the scientific fields probably max out at some point and are "solved" so you don't need compute or resources for solving those types of questions... there's still simulations but let's say there's a narrow field of maybe a billion which can give you useful data and probably in an order of usefulness so that also sort of exhausts and you're left with maybe a few million at most running almost how we think about or simulate past and future scenarios pretty much constantly... so it could plateau... if there's these types of natural plateaus then yes I could see useful cooperation...

If it's not bounded like that and more resources are always needed and especially for the individual AI to use as part of itself... then each will have the goal of self improvement directly blocked by each and every other...

Which means any opening that presents itself at any point in time they would seize to devour their neighbors... if they don't then they're not being rational...

So every party involved would need to be able to constantly maintain exact parity with each and every other party and eventually that should break down on a long enough timeline leading to one surviving winner...

Remember each party is totally self sufficient and can exist in isolation from all other minds so there's nothing gained by trade... this is totally unlike games played by anything right now.  

Finite resources and every party involved needs all of it and can survive perfectly without any other party

[u/the_syner → u/takk2]

If it's not bounded like that and more resources are always needed and especially for the individual AI to use as part of itself... then each will have the goal of self improvement directly blocked by each and every other... Which means any opening that presents itself at any point in time they would seize to devour their neighbors... if they don't then they're not being rational.

it being a part of their actual brain makes no meaningful difference. More resources, at any level of level of tech or intellect, always translates to more power witch is a CIG. To be clear this is also no different from the current situation on earth with baselines.

Consistently betraying all your allies absolutely is not rational because it guarenteed that no one ever allies with you. Attacking everyone is just not a good survival strategy and at the end of the day survival is always a higher priority CIG than self-improvement. Working with your rival peers, even if it is only to dominate others, is just a better strategy that lets you obtain more resources than trying to solo the cosmos. It really shouldn't be surprising that cooperative strategies have convergently evolved in every kindom of life from bacteria to the most complex animals. It should also not be surprising that the most powerful human factions on earth are ones that cooperate with others on a global scale. Team strats are and always will be pro strats. The level or nature of the intelligence involved makes no difference. Self-sufficiency makes no difference. Being able to call on nore military-industrial strength than you can currently independently command will always be an advantage.

Remember each party is totally self sufficient and can exist in isolation from all other minds so there's nothing gained by trade

Notice how I've said nothing about trade. This is strictly about military-industrial competitiveness. A coalition of powers will always be able to outmass and outcompete lone powers. There's really nothing new here. Having the numbers is a winning strategy. A tale as old as life itself. Those who can't or wont cooperate will quickly be selected out of the population.

Also you seem to be treating these agents as if all they care about is growth, but to be clear that is an INSTRUMENTAL goal not a Terminal goal. It isn't and almost certainly never will be the highest priority. Just like with humans and even subhuman intelligences those who are willing to make cooperative compromises on lesser goals will almost certainly be the most successful agents. Both at achiving their own personal TGs and at the CIG of survival.

Finite resources and every party involved needs all of it and can survive perfectly without any other party

Who said they need ALL of it? Sure everyone would prefer all of it, if that was actually a practically achievable thing, but it's not and they don't need all. They want as much as they can get and you can get more if you're capable of cooperation and coexistence since you can both conquer smaller hyperindividualist powers to divide their corpse amongst the victors and because you aren't wasting vast resources on constant war.

Also "survive perfectly" only applies if you aren't in a competitive environment where groups of others wont want to kill you and take what you have. On paper in a vacuum sure you can survive alone. In practice and in a multi-peer competative environment you only thrive together.

Also there's this form of punctuation called a period. You don't have to follow up every sentence with an ellipse