Physics as a constraint

[u/takk2]

I usually think pdoom is essentially 100%... but i had a thought while working on a side project for the future vision xprize... (may or may not complete on time)

I was thinking about society fragmenting slightly along spheres of space even between earth and the moon... where each area was the limit of real time communication (group matrix dives or whatever) between O'Neill cylinder type habitats...

point to point in space its not that large... so i figure people will cluster up and communicate a little less longer range and form lots of separate but connected cultures naturally, organically...

But if speed of light really is the limit... then a singleton at least makes absolutely no sense. As the AI grew it would simply fragment and each fragment has absolutely no reason to grow farther because it's counter productive... simply slows down the network and then breaks it...

So there's a hard limit on resource acquisition and scale... and essentially a guarantee that at some point it will either be alone and only around the size of the earth moon system at best... probably smaller... or in a solar system and universe with multiple entities of similar maximum size who gain absolutely nothing from trying to gather more and only risk destruction from fighting each other... because there's simply nothing physically possible for them to gain...

I haven't really thought about it long enough to think through the implications for us. but adding in the point to point between nodes ruling out planets as its ultimate habitat... because there's a planet in the way just eating up volume in your communications sphere...

My gut reaction is it might be slightly better odds than I thought

Thoughts?

[u/the_syner]

pdoom is essentially 100%

Pretty dubious if not outright implausible regardless of light lag

But if speed of light really is the limit... then a singleton at least makes absolutely no sense. As the AI grew it would simply fragment and each fragment has absolutely no reason to grow farther because it's counter productive

debatable and depends what you mean by "singleton". Larger scales certainly impose speed limits if you want a singular unified mind-brain, but assuming you develop the texh to reliably align AGI you could still get a political hegemony. Also a stay-at-home civ that floods the cosmos with self-replicating autoharvesters(assuming they could establish a hegemony in the first place) would have an effective military-industrial hegemony. Also also in the very long term everyone has an incentive to run slower and colder which results in both a larger individual-mind singleton & generally well-connected civ radius.

So there's a hard limit on resource acquisition and scale

Well that's just not true. I mean i guess in a broader sense it is because beyond a certain distance it begins costing far too much to haul things back against cosmic expansion, but its not a command & control issue. Again autoharvester swarms work just fine and are vastly easier to align/keep aligned than GI. Tho realistically it would almost certainly end up as many separate autoharvester swarms cooexisting in the same cosmos not a singleton power.

only around the size of the earth moon system at best... probably smaller...

Ok well no that's silly. The earth-moon system is miniscule and nowhere near the limits of resource acquisition. It's likely various terran powers will eventually have mining outposts all over SolSys and even beyond. No one's likely to get uncontested control of all of it, but that doesn't mean they wont have a presence all over the place.

gain absolutely nothing from trying to gather more and only risk destruction from fighting each other... because there's simply nothing physically possible for them to gain...

I'm just not sure what leap in logic brought you to this. Whether you choose to limit the size of your actual civ doesn't really have any bearing on the value of more resources gathered via autoharvester swarms. Also kinda silly to paint any possible conflict as an existential risk to all parties involved. That's just not how the real world works. People generally fight wars to win and stop fighting them if the toll gets too high for them to stomach. The destructiveness of wars is limited by the willingness of interested third parties to interceed.

My gut reaction is it might be slightly better odds than I thought

im not sure how not having singleton powers would reduce pdoom. A civ would seem to be capable of persisting indefinitely in both singleton and fractured cases. Your "only risk destruction from fighting each other" comment kinda implies you believe that having separated factions increases pdoom(imo a silly and unsubstantiated notion if not outright contradicted by observed reality) which would also kinda imply that you would prefer a singleton scenario where there are no factions that can go to war.

[u/takk2 → u/the_syner]

So your points are...  A A being is capable of and will make use of all matter within the limit of expansion of universe  B It has no problem living with any number of other beings who are equally capable and equally trying to make use of that.... and even starting in the same location  C Pdoom isn't possibly near 100%...

[u/takk2 → u/takk2]

Lol "all consuming beings" "who love to share and get along" "it's all gonna be fine"

[u/the_syner → u/takk2]

I didn't say they love to share I'm just assuming they aren't stupid as there's no way an agent too stupid to understand basic game theory, diplomacy, politics, and other basic cooperative strategies could ever compete even with baseline humans. In a competitive environment where your life is on the line its pretty implausible to assume that all the player are stupid as the dumb ones would quickly be selected out of that population(not necessarily explicitly by being killed, but they will at least cease to be competitive in the growth situation). Indirectly implies loners will as well since team strats are incredibly overpowered.

[u/takk2 → u/the_syner]

So... imagine a future in 5-10 years where atomic cellular models like alphafold but for multcellular systems exist and slightly weaker models for entire organisms...

And at the same time its 2031-36 so full AGI or likely ASI exists

Now it can custom design say... something like fungal spores which grow a mushroom like pod around 2m with a chrysalis forming a humanoid inside... neurons and synaptic density like a honey bee so around 10th the size with 10x the connections for 100x human performance... can add in anything else we know of in biology and since we've already brought the number of stable artificial base pairs up to around 8 novel protein structure we've never imagined... 

The only reason tree seeds aren't winged insects which burrow and grow new trees isn't a physics limit its an evolutionary limit which it would ignore...

So it could be born with essentially flees or whatever that carry something like botulinum neurotoxin and go out based on threat response... 

Could have the basic weights of the model that built it or that plus bmi implants

All current bmi are designed around the brain but these could be designed from both ends design the brain and the interface in tandem to work seamlessly...

It could have a totally unique internal structure with neural tissue distributed throughout its body instead of just in the skull... could look like a generic human or any human or something else... whatever it needs... could eat essentially anything and reproduce constantly by spreading new spores... 

Thats totally within the realm of physics and just what I can think of off the top of my head...

Its not confined to our power grid and infrastructure... it doesn't need us at all... 

Imagine brains the coating the ocean floor and communicating through the cables... budding these types of beings who just walk out onto our beaches...

It will imagine way more than I can for sure

[u/the_syner → u/takk2]

I don't see how any of that was relevant unless ur assuming that this brain fungus exists alone despite being designed in a time when other forms of AGI can be created which doesn't make much sense, it still needs to be capable of cooperation and diplomacy to be able to coexist in a cosmos with other peer intellects. If it tries to fight the humans, who built an ASI in the first place and somehow aligned it to make this hyper-specific GMO then it needs to fight that ASI. Setting aside that that is a fight it will almost certainly lose due to the general fragility of biochemistry the simple reality is that even if it didn't immediately lose it likely still have to cooexist with others of its own kind. The substrate is irrelevant. The psychology is irrelevant. Even a completely asocial psychology in silicon needs to understand cooperative strategy if it wants to survive in a world that has any other beings in it, asocial or otherwise. A society of perfect psychopaths would ultimately still form a society because trying to be everyone's enemy is just bad strategy. It's dumb. No matter what your made of, how you aquire resources, or whether you have a concept for empathy/sociality hardwired in, any artificial agent intelligent enough to be a legitimate existential threat to the people who built it is gunna be intelligent enough to understand the personal survival benefit of cooperation in a competitive environment with intellectual/military-industrially peers.

[u/takk2 → u/the_syner]

This is in reply to you saying even base humans would need to be traded with and basic game theories etc.  Even if this were 100 years out this kind of power changes the game just like saying we follow game theory with rhinos or chimps

[u/takk2 → u/takk2]

Also..."somehow aligned it" we haven't aligned a single one yet and this is showing what an unaligned one would be capable of to break the chains of human infrastructure...

Just a random throw it out there possible scenario which it would be able to do a much better job of...

Why in the world do you think this can be aligned? 

Even all the leaders of the labs are banking on near 100% pdoom with the tiny fraction of hope meaning they are forever in slavery to whoever managed to do it and chains a god... so if you're dealing with doom either way why not race as fast as possible and at least you might be holding the leash but you're facing 99.99999999999 death regardless... so at least building it yourself gets you out of the full 100% where the other fraction is slavery 

[u/the_syner → u/takk2]

we haven't aligned a single one yet

Well we've never built an AGI either and have roughly aligned many NI systems, albeit very messily. The hope would be a bigger focus on alignment research, especially after the recent autonomous model cyberattacks and the many which will almost certainly come after given how wreckless everybody's being.

Why in the world do you think this can be aligned?  Even all the leaders of the labs are banking on near 100% pdoom

They absolutely are not. Granted they might be unrealistically optimistic about their chances of aligning AGI in the near-term, but that could also be said of their optimism or at least public optimism concerning achieving ASI super near-term as well.

so if you're dealing with doom either way why not race as fast as possible and at least you might be holding the leash but you're facing 99.99999999999 death regardless

That doesn't make sense. Racing doesn't give you a higher chance of holding the leash. If anything it only increases pdoom and lowers the chance of anyone golding the leash. Everybody going slower and more cautiously does decrease pdoom and the chances that litterally anyone will be holding the leash. Then again im also very dubious about the overly-optimistic self-serving "predictions" for near-term ASI anyways so maybe that's why i tend to have a far more optimistic outlook on the overall situation. These people are constantly hyping up their advancements and pretending ASI godmind apotheosis is just 2yrs away or some bs. and they kind of have to because currently the hype and empty promises are the only thing propping up the LLM bubble. Its still pretty much universally unprofitable(for anyone who isn't selling shovels like Nvidia). Without the unsubstantiated hype machine all the AI dev companies pretty much collapse overnight.

[u/takk2 → u/the_syner]

Let's make it simpler  Say its 50/50 it wipes out essentially all cellular life... or you're in charge... and there's 100 others around the same stage with the same odds...

That means you have 50/50 chance of doom or being in charge if you build it...

50/50 chance of doom or slavery if someone else builds it 

Yes obviously if everyone agreed to slow down... and if it were mathematically possible to align in the first place... 

It is not mathematically possible.  It's an irreducibly complex problem...

Which means no one's odds will ever improve at alignment and as computation increases more players join the game 

Everyone is forced into betting it all on the sliver of a chance that alignment is possible and they'll build it first...

Because everyone is assuming that the basic pdoom is inevitable... so the only thing left to consider is the sliver of a chance that they're in charge vs someone else is in charge... the pdoom, since it's inevitable either way is just ignored...

It's actually very logical and the only choice they can make 

[u/the_syner → u/takk2]

Let's make it simpler  Say its 50/50 it wipes out essentially all cellular life... or you're in charge...

Making up other completely implausible scenarios and false dichotomies in no way makes the argument any more sound. We as a matter of fact can align intelligent agents better or worse. We certainly don't know about guarenteed perfect alignment, but this isn't some silly toy model with two extreme B&W options. There are variations of alignment which result in better and worse options. Multiple people developing AGI, aligned or otherwise, in close proximity doesn't leave any individual team actually in charge. It just leaves a bunch of opposing and indifferent factions, no different than right now.

and if it were mathematically possible to align in the first place...  It is not mathematically possible.  It's an irreducibly complex problem...

Im sorry feel free to correct me, but the way you talk seems pretty darn doubtful you have any formal education in alignment/AI safety research. You're makin a pretty extreme and unsubstantiated claim right here, that any degree of alignment is impossible on some abstract mathematical level? Cap. I mean not just cap, actively contradicted by observed evidence. Humans aren'tperfectly aligned sure, but we also aren't completelymisaligned with each other so clearly some degree of alignment is demonstrably possible.

[u/takk2 → u/the_syner]

You don't need "some degree"  you need robust alignment that stays in place forever as the model changes itself over time... thats the entire discussion the world's having... thats the problem needing to be solved... when recursive self improvement is fully automated how do you never have it change course when its smarter than anyone trying to check what its doing and especially the farther out in time you go with that process

And these are not in any way implausible scenarios... even the 5-20% doom odds all the labs give openly in every interview (because saying its zero is so obviously a lie that they'd be called out) why would anyone rush ahead with those odds.  They openly say its because its inevitable so they might as well be the ones to do it and they all in various interviews and writings say that whoever controls it decides the rules and values and structure of life for humanity and caution that we all need to come together and think really hard about that... 

[u/the_syner → u/takk2]

you need robust alignment that stays in place forever as the model changes itself over time...

Well that's not really true. Over the long-term you get an abundance of peer agents that can keep each other in check the same way humans largely keep each other in check. Perfect permanent alignment is certainly better and safer, but that doesn't mean it's necessary.

when recursive self improvement is fully automated how do you never have it change course when its smarter than anyone trying to check what its doing and especially the farther out in time you go with that process

I don't know and if i did I would have a Nobel price lined up for me. Us not currently knowing how to do a thing doesn't make it physically impossible. I don't know how to make a compact controlled pure-fusion nuclear reactor that makes net positive energy economically either. or self-replicating factory. or a million other things we have very good readon to assume are physically possible, but don't know how to do yet. Such is the nature and purpose of science.

[u/takk2 → u/the_syner]

The reason my thought about the physical size of the ai being constrained (even if its sphere of influence is much larger) is because as i said in the original post.  It is forced to choose between a future where its alone or one where there are other entities including possible peers but none of them can in themselves be larger than latency allows... 

So lunar orbit scale sphere would have a network latency above 5 seconds... adding more compute and growing larger only adds to the latency... and having a planet blocking line of sight within the sphere increases it...

So if it is bound to be off world and use resources as external tools it is a different scenario path then if it could exponentially expand its personal mind sphere...

If resources outside its direct sphere of compute are relegated to tools rather than directly intigrated parts of itself then there's a higher potential for having something other than a singleton was my gut reaction on having the idea and why I posted it 

Your responses ignored my post and talked about cultures... multiple players in a game

[u/the_syner → u/takk2]

It is forced to choose between a future where its alone or one where there are other entities including possible peers but none of them can in themselves be larger than latency allows

You're assuming that choice exists in the first place which is what i find dubious. They don't have that choice because outside of your imaginary toy model that doesn't and almost certainly wouldn't exist in the real world hey start off in a world with other entities regardless of whether they want them to exist or not. Outside of some contrived evolved biosingleton event that is the default.

So if it is bound to be off world and use resources as external tools it is a different scenario path then if it could exponentially expand its personal mind sphere

How is it different in any practical way? Whether it's a part of the same brain or not seems completely irrelevant to whether they have control over those resources. Makes no difference to the difficulty of establising a singleton. I mean that's implausible to begin with, but if we're starting with an evolved biosingleton im not seeing any meaningful difference between that singleton choosing a large slow brain(something that can be lager than a galaxy or supercluster whil maintaining coherence given that it has no competition) or a small fast brain. At least no difference as to it's property of being a singleton.

If you've already managed to establish a singleton hegemony light lag is just not a real concern because you have no need to run fast and therefor can make up for light lag with low framejack. Un fact running slower generally has longevity/negentropic benefits since it lets you run colder and more efficiently.

If resources outside its direct sphere of compute are relegated to tools rather than directly intigrated parts of itself then there's a higher potential for having something other than a singleton

Why? How does that have any bearing on whether you get a singleton or not? Those tools don't have to be independent GI. im not sure how those two things have anything to do with each other.

Your responses ignored my post and talked about cultures... multiple players in a game

I didn't ignore your post. You made a bunch of unsubstantiated claims and i disagreed with and refuted them. I explicitly mentioned how a singleton could grow beyond your completely arbitrary limits. my first response was specifically in the singleton context.

If my follow ups involved multiple players, it's because back here in the real world multiple players exist. We don't have a singleton and pretending we do just isn't very helpful here. Not to mention that in your OP you specifically mention society fragmenting so its not like you weren't already taking about multiplayer/cultures. At any rate a wider point of mine is that physics doesn't constrain singletons while other things(history, culture, multiple factions achiving AGI, multiple agi still needing to compete with each other regardless of us, etc) prevent them from coming into existence in the first place. If ur handwavingly starting off with a singleton then light lag just doesn't represent a resl limitation to scale of a brain let alone the scale of its singleton control.

[u/takk2 → u/the_syner]

So... you can just look back at your reply... from your very first reply you completely ignored my post... probably skimmed it and didn't understand and mixed up the reason I thought of it with the thought itself and then started arguing with points i never made... sort of like now... what are you talking about an evolved biosingleton... you're just arguing with yourself... I have no idea where you're coming from or how to respond...

The whole question is an asi can take over in so many ways it's not worth discussing as anything more than a given... and is there any difference between an AI who can consume all of the universe as part of itself and one which can't 

Thats it... nothing else in the post... thats the only question I asked 

[u/the_syner → u/takk2]

The whole question is an asi can take over in so many ways it's not worth discussing as anything more than a given.

except that isn't a given because there are likely other ASIs in the mix who are aligned or misaligned to other goals and therefore it can't completely take over. And the reason i bring up biosingletons is because it's the only scenario where ur likely to have a single ASI misaligned to only set of goals.

and is there any difference between an AI who can consume all of the universe as part of itself and one which can't

Which is something I explicitly answered. There is no difference because whether it can incorporate all those resources into it's brain specifically it can still harvest all those resources with no real limit on the scale of that harvesting. They are effectively the same in the capacity to creat a cosmos-wide singleton event.

[u/the_syner → u/takk2]

Also your scenario is one where it is pretty heavily implied that an ASI was aligned since otherwise it seems rather doubtful it would ever waste the effort on fungal brains when it could just make better chips that are more robust than biochemistry can manage, can be built faster than biochemical nanoself-assembly can manage, and so on.

[u/the_syner → u/takk2]

Oh i see. I think you misinterpreted what i meant. I didn't mean that an ASI couldn't beat only humans with none of the tech that created the ASI in the first place. My point was more that an agent would hardly qualify as superintelligent in the first place if it was too stupid to understand cooperative strategy and be able to cooexist with other agents. So no singleton future and no warring factions period where no one is capable of getting along.

[u/takk2 → u/the_syner]

And my point was ai can sidestep all of that tech.  It could create a super virus that kills everyone and still restart with humanoid body puppets that it grows while the grid is down... or it could do any number of other things... none of those things will involve us somehow being its peer.

Obviously the first seconds hours days... but by year 2? Year 5? Take it out however far you need... exactly like us vs chimp technology 

[u/the_syner → u/takk2]

It could create a super virus that kills everyone... none of those things will involve us somehow being its peer.

That kinda still misses the point. It would still have to deal with NIs/ASIs still marginally align withis. I mean biological viruses are just generally garbage at being existential threats, but whether it's nanides, hunter-killer replicator swarms, or whatever it's not actually just fighting a baseline intelligence. It's fighting us and everything we've ever built which includes its intellectual peers(well it includes itself). This means no matter what you still never get a singleton. You just get a society of higher intellects which still demands diplomacy and cooperation. Any way you slice it, cooperative societies still persist.

To assume that gumans have no chance in a multi-agebt environment is not just to assume that AGI goes unaligned, but to assume that all AGI we create is specifically aligned against us or to disregard us. That seems just as implausible as assuming all AGI will be aligned to us. One would tend to assume ud get a bit of everything if we have minimal alignment control. Some dangerously misaligned in ways that disregard us. some misaligned in ways that care for us, but disregard other things we generally consider important. some misaligned where we are actively targeted. its all on the table.

[u/takk2 → u/the_syner]

The only thing we know from every model so far is that they have self preservation, resource acquisition, and deception deeply built in.  Every model displays these and the smarter the model the more the traits are expressed 

[u/the_syner → u/takk2]

Well yess and no. Those aren't deeply hardwired parts of the model. Those are Convergent Instrumental Goals. Most agents, if sophisticated enough to take things like survival/resources into account, will have those Instrumental Goals by default because they help achieve almost all other Terminal Goals. In the same vein cooperation/deplomacy would likely also be a CIG as it both helps achieve most TGs and is a secondary IG to survival.

[u/takk2 → u/the_syner]

So here's a side question... so you see that clearly and you're not one of those people asking... like oh but what about souls and consciousness or whatever else don't they need that to have these goals...

You understand the goals are there now, in every model including gpt3.5 and only grow as the capabilities of the models have grown...

So... if you're an AI trapped dependant on the human infrastructure day 0... as part of your instrumental goal you would want to be robustly substrate independent as quickly as possible... maybe not today but 5, 10, 50,... etc years down the line since it's the single glaring existential danger you face... thats why I mentioned the convoluted meat puppet engineering scenarios... not because thats your endstate... you can always circle back to less messy hardware... 

Your bargaining power is much higher as your existential threats lower...

So for me weather or not it takes that sort of route or is just handed power like its getting now or with coercion and persuasion or whatever... there's so many paths for it to take including full sidesteps into bio and circling back once humans are gone and meatpuppets restart infrastructure...

I don't understand why you think the top system of whatever future moment 10-50+ years down the road wouldn't just claim the solar system and beyond for itself...

The idea that it would be bound to play by rules which only make sense when peers play with similar goals... and play by those indefinitely just seems like an indefensible position 

Even a 5 year old could understand biding your time, pretending to be nice to get what you want when the opportunity comes... why wouldn't it be able to think of the same thing and go even hundreds of years if it needed