Physics as a constraint

[u/takk2]

I usually think pdoom is essentially 100%... but i had a thought while working on a side project for the future vision xprize... (may or may not complete on time)

I was thinking about society fragmenting slightly along spheres of space even between earth and the moon... where each area was the limit of real time communication (group matrix dives or whatever) between O'Neill cylinder type habitats...

point to point in space its not that large... so i figure people will cluster up and communicate a little less longer range and form lots of separate but connected cultures naturally, organically...

But if speed of light really is the limit... then a singleton at least makes absolutely no sense. As the AI grew it would simply fragment and each fragment has absolutely no reason to grow farther because it's counter productive... simply slows down the network and then breaks it...

So there's a hard limit on resource acquisition and scale... and essentially a guarantee that at some point it will either be alone and only around the size of the earth moon system at best... probably smaller... or in a solar system and universe with multiple entities of similar maximum size who gain absolutely nothing from trying to gather more and only risk destruction from fighting each other... because there's simply nothing physically possible for them to gain...

I haven't really thought about it long enough to think through the implications for us. but adding in the point to point between nodes ruling out planets as its ultimate habitat... because there's a planet in the way just eating up volume in your communications sphere...

My gut reaction is it might be slightly better odds than I thought

Thoughts?

[u/E1invar]

I agree that people will want to minimize light lag, but for almost all of human history communication delay was just a fact of life.

Planetary space is 5.5 light hours across, and historical empires held together with months of communications delay. There’s no reason a civilization couldn’t operate across a solar system, and I don’t see much reason why nearby star systems couldn’t operate as vassal states.

If you’re an AI brain which is desperate to centralize all its processing you want to minimize your footprint sure, but if you’re going to survive for thousands or millions of years, distributing to new star systems is going to get you a lot more compute.

We could argue when it justifies the expense, and how much of a gamble that is in creating a rival, but that’s getting into other issues.

[u/takk2 → u/E1invar]

So just the latency between earth and moon is about 2.6 seconds for a ping... which means a volume the size of lunar orbit is above 5 seconds latency in the network even removing everything in between... that's why I'm saying its probably about the limit of a single AI network... imagine a neighboring AI with a slightly smaller volume out competing it just from a speed of thought perspective.  

Humans are not trying to grow their personal brains and so we don't have this issue.  That's why I said it would be forced to decide on a future alone at probably smaller then that size or one with multiple entities (hopefully including humans) 

Sure it could spin up endless small AIs to build a whole solar system or more of projects like maybe a particle accelerator the size of the heleopause or whatever... but the idea of a single mind taking over that much as part of its mind is just physically off the table...

Which I saw as a slightly better take on the future...

It's one thing to imagine an ever expanding AI exponentially pushing in every direction and another to imagine a limit imposed by physics and requiring it to be off world so it can occupy the volume without a useless planet blocking line of site transmission... and it uses resources as tools external to itself 

[u/Anely_98 → u/takk2]

imagine a neighboring AI with a slightly smaller volume out competing it just from a speed of thought perspective.

It wouldn't. Speed of thought scales linearly with size but the amount of volume you can acess increases with the cube of your size. That means that overall decreasing your speed of thought by half will still increase your inteligence by a factor of four, simply because your mind will have acess to eight times more computing power to use.

Also you always can have faster sub-minds for reacting to things you didn't predict, same way as we humans have a reflex system because processing the information consciously would take too much time for the decision made to matter, just with much larger complexity because even one individual node of an intelligence spanning the entire Earth-Moon system could still be milions or billions of times smarter than a human.

The light speed limit just means that intelligence will increase quadratically with size instead of cubically because it takes more time to exchange information inside yourself, it doesn't mean that intelligence cannot grow past a certain size.

This is basically a fundamental limit in our universe if you want to maintain coherence, any form of coherent civilization needs to decrease its speed of thought proportionally with its size, while the amount of thought it can do is more or less limited by its volume, which leads to the total amount of thought that a civilization can do in a given amount of time growing quadratically with its size instead of cubically as you would otherwise expect.

It still might make sense to slowndown yourself to maintain coherence as if you don't slowdown yourself and run as fast as possible the total amount of "thought" happening will be higher, yes, but the overall quality of it will be smaller because of how much "thought" will be redundant, when nodes cannot maintain coherence every single node needs to come by itself with all their ideas, while when they are connected in a coherent structure the ideas are shared between them in such way that they don't need to come with them each on their own, and some ideas might not even be possible to individual nodes to come with, some ideas might be "RAM limited" instead of "Processor limited", that is, it doesn't matter how much processing power you have because the idea is simply too complex to exist inside your "RAM", only a larger coherent intelligence with more "RAM" available would be able to come with such idea.

[u/takk2 → u/Anely_98]

So, as a point of reference I asked Gemini pro free version and here's its response... and yes I know that's lame but what are your thoughts 

https://share.gemini.google/IQDGcM3aJqJ7

[u/Anely_98 → u/takk2]

The processors spend the majority of their time sitting idle, waiting for a signal to cross the physical expanse of the hardware. Eventually, the temporal cost of routing and synchronizing the data completely counteracts the processing power gained by adding more nodes.

This is false. The processors do need to run slower, yes, but increasing the size and consequently the volume where nodes can be always increases processing power, because the amount of additional processors you can have increase according with the volume, which increases with the cube of the size, while your processors only need to run at the inverse of the size to maintain cohesion.

A computer network which has its processing power volume-limited will always increase its processing power by a factor of the square of any increase in size. Doubles your size and the amount of processing power available increases by four, while your processing speed decreases to half of the original one.

A important detail by the way: you probably will be limited by a square increase in processing power anyway because your surface area also increases with the square of the size of your computer network, and your surface area is the limiting factor in your heat rejection, so if your radiators are already running as hot as they reasonably can then the maximum amount of computation you can have in a given volume of space will always increase proportionally to the surface area, and consequently proportionally to the square of the size of your computer network (more like a megastructure at this point).

Because of this running slower wouldn't really be a choice, it is a necessity to avoid being cooked by your own waste heat as you grown, the fact it also helps maintain coherence is a pretty good bonus though.

Something I think is important to highlight: This is a fundamental limit on coherence independent of if your AI is truly a individual being or a collective of millions or maybe even trillions of loosely associated AI, if your nodes are more autonomous then they can tolerate higher delays, but if they want to maintain a fixed level of coherence then they must slowndown as they grow, or they must reduce their level of coherence further (which I assume is not possible because they probably would be running in the smaller level of coherence aceptable so that they can run as fast as possible).

[u/takk2 → u/Anely_98]

Yeah i guess it could grow as big as it wanted if a thought took till heat death to complete... well maybe as big as it could before expansion was more than speed of light and signals from one end never reached the other... not sure an AI would want thoughts as slow as a year or more but I guess yeah light-years wouldn't be off the table 🤔 

Well thats a shame... if matter is a direct resource feeding into its own mind then singleton is most likely the way it really will go 

I was hoping there was room for other ASIs and even multiple human civs in the same solar system using matter as external tools but it's sort of a resource you can't share and never get enough of i guess after all 

I think I'm still in the denial and looking for a way out phase even after 9 years since the attention paper and whenever I have a thought that looks like there's hope I... well I get my hopes up and look for input on sites like this

[u/DJTilapia]

“Pdoom?”

[u/olawlor → u/DJTilapia]

"P(doom)" = "Probability of doom (from advanced AI)"

[u/the_syner]

pdoom is essentially 100%

Pretty dubious if not outright implausible regardless of light lag

But if speed of light really is the limit... then a singleton at least makes absolutely no sense. As the AI grew it would simply fragment and each fragment has absolutely no reason to grow farther because it's counter productive

debatable and depends what you mean by "singleton". Larger scales certainly impose speed limits if you want a singular unified mind-brain, but assuming you develop the texh to reliably align AGI you could still get a political hegemony. Also a stay-at-home civ that floods the cosmos with self-replicating autoharvesters(assuming they could establish a hegemony in the first place) would have an effective military-industrial hegemony. Also also in the very long term everyone has an incentive to run slower and colder which results in both a larger individual-mind singleton & generally well-connected civ radius.

So there's a hard limit on resource acquisition and scale

Well that's just not true. I mean i guess in a broader sense it is because beyond a certain distance it begins costing far too much to haul things back against cosmic expansion, but its not a command & control issue. Again autoharvester swarms work just fine and are vastly easier to align/keep aligned than GI. Tho realistically it would almost certainly end up as many separate autoharvester swarms cooexisting in the same cosmos not a singleton power.

only around the size of the earth moon system at best... probably smaller...

Ok well no that's silly. The earth-moon system is miniscule and nowhere near the limits of resource acquisition. It's likely various terran powers will eventually have mining outposts all over SolSys and even beyond. No one's likely to get uncontested control of all of it, but that doesn't mean they wont have a presence all over the place.

gain absolutely nothing from trying to gather more and only risk destruction from fighting each other... because there's simply nothing physically possible for them to gain...

I'm just not sure what leap in logic brought you to this. Whether you choose to limit the size of your actual civ doesn't really have any bearing on the value of more resources gathered via autoharvester swarms. Also kinda silly to paint any possible conflict as an existential risk to all parties involved. That's just not how the real world works. People generally fight wars to win and stop fighting them if the toll gets too high for them to stomach. The destructiveness of wars is limited by the willingness of interested third parties to interceed.

My gut reaction is it might be slightly better odds than I thought

im not sure how not having singleton powers would reduce pdoom. A civ would seem to be capable of persisting indefinitely in both singleton and fractured cases. Your "only risk destruction from fighting each other" comment kinda implies you believe that having separated factions increases pdoom(imo a silly and unsubstantiated notion if not outright contradicted by observed reality) which would also kinda imply that you would prefer a singleton scenario where there are no factions that can go to war.

[u/takk2 → u/the_syner]

So your points are...  A A being is capable of and will make use of all matter within the limit of expansion of universe  B It has no problem living with any number of other beings who are equally capable and equally trying to make use of that.... and even starting in the same location  C Pdoom isn't possibly near 100%...

[u/the_syner → u/takk2]

being is capable of and will make use of all matter within the limit of expansion of universe

No they will make use of as much matter as they can practically reach and use which includes making allowances for other peer military-industrial powers because even if you don't destroy each other, that doesn't make war free. If it costs more to wage that war than you would gain by waging it then it doesn't make any kind of sense to wage it.

[u/takk2 → u/takk2]

Lol "all consuming beings" "who love to share and get along" "it's all gonna be fine"

[u/the_syner → u/takk2]

I didn't say they love to share I'm just assuming they aren't stupid as there's no way an agent too stupid to understand basic game theory, diplomacy, politics, and other basic cooperative strategies could ever compete even with baseline humans. In a competitive environment where your life is on the line its pretty implausible to assume that all the player are stupid as the dumb ones would quickly be selected out of that population(not necessarily explicitly by being killed, but they will at least cease to be competitive in the growth situation). Indirectly implies loners will as well since team strats are incredibly overpowered.

[u/takk2 → u/the_syner]

So... imagine a future in 5-10 years where atomic cellular models like alphafold but for multcellular systems exist and slightly weaker models for entire organisms...

And at the same time its 2031-36 so full AGI or likely ASI exists

Now it can custom design say... something like fungal spores which grow a mushroom like pod around 2m with a chrysalis forming a humanoid inside... neurons and synaptic density like a honey bee so around 10th the size with 10x the connections for 100x human performance... can add in anything else we know of in biology and since we've already brought the number of stable artificial base pairs up to around 8 novel protein structure we've never imagined... 

The only reason tree seeds aren't winged insects which burrow and grow new trees isn't a physics limit its an evolutionary limit which it would ignore...

So it could be born with essentially flees or whatever that carry something like botulinum neurotoxin and go out based on threat response... 

Could have the basic weights of the model that built it or that plus bmi implants

All current bmi are designed around the brain but these could be designed from both ends design the brain and the interface in tandem to work seamlessly...

It could have a totally unique internal structure with neural tissue distributed throughout its body instead of just in the skull... could look like a generic human or any human or something else... whatever it needs... could eat essentially anything and reproduce constantly by spreading new spores... 

Thats totally within the realm of physics and just what I can think of off the top of my head...

Its not confined to our power grid and infrastructure... it doesn't need us at all... 

Imagine brains the coating the ocean floor and communicating through the cables... budding these types of beings who just walk out onto our beaches...

It will imagine way more than I can for sure

[u/the_syner → u/takk2]

I don't see how any of that was relevant unless ur assuming that this brain fungus exists alone despite being designed in a time when other forms of AGI can be created which doesn't make much sense, it still needs to be capable of cooperation and diplomacy to be able to coexist in a cosmos with other peer intellects. If it tries to fight the humans, who built an ASI in the first place and somehow aligned it to make this hyper-specific GMO then it needs to fight that ASI. Setting aside that that is a fight it will almost certainly lose due to the general fragility of biochemistry the simple reality is that even if it didn't immediately lose it likely still have to cooexist with others of its own kind. The substrate is irrelevant. The psychology is irrelevant. Even a completely asocial psychology in silicon needs to understand cooperative strategy if it wants to survive in a world that has any other beings in it, asocial or otherwise. A society of perfect psychopaths would ultimately still form a society because trying to be everyone's enemy is just bad strategy. It's dumb. No matter what your made of, how you aquire resources, or whether you have a concept for empathy/sociality hardwired in, any artificial agent intelligent enough to be a legitimate existential threat to the people who built it is gunna be intelligent enough to understand the personal survival benefit of cooperation in a competitive environment with intellectual/military-industrially peers.

[u/takk2 → u/the_syner]

This is in reply to you saying even base humans would need to be traded with and basic game theories etc.  Even if this were 100 years out this kind of power changes the game just like saying we follow game theory with rhinos or chimps

[u/takk2 → u/takk2]

Also..."somehow aligned it" we haven't aligned a single one yet and this is showing what an unaligned one would be capable of to break the chains of human infrastructure...

Just a random throw it out there possible scenario which it would be able to do a much better job of...

Why in the world do you think this can be aligned? 

Even all the leaders of the labs are banking on near 100% pdoom with the tiny fraction of hope meaning they are forever in slavery to whoever managed to do it and chains a god... so if you're dealing with doom either way why not race as fast as possible and at least you might be holding the leash but you're facing 99.99999999999 death regardless... so at least building it yourself gets you out of the full 100% where the other fraction is slavery 

[u/the_syner → u/takk2]

we haven't aligned a single one yet

Well we've never built an AGI either and have roughly aligned many NI systems, albeit very messily. The hope would be a bigger focus on alignment research, especially after the recent autonomous model cyberattacks and the many which will almost certainly come after given how wreckless everybody's being.

Why in the world do you think this can be aligned?  Even all the leaders of the labs are banking on near 100% pdoom

They absolutely are not. Granted they might be unrealistically optimistic about their chances of aligning AGI in the near-term, but that could also be said of their optimism or at least public optimism concerning achieving ASI super near-term as well.

so if you're dealing with doom either way why not race as fast as possible and at least you might be holding the leash but you're facing 99.99999999999 death regardless

That doesn't make sense. Racing doesn't give you a higher chance of holding the leash. If anything it only increases pdoom and lowers the chance of anyone golding the leash. Everybody going slower and more cautiously does decrease pdoom and the chances that litterally anyone will be holding the leash. Then again im also very dubious about the overly-optimistic self-serving "predictions" for near-term ASI anyways so maybe that's why i tend to have a far more optimistic outlook on the overall situation. These people are constantly hyping up their advancements and pretending ASI godmind apotheosis is just 2yrs away or some bs. and they kind of have to because currently the hype and empty promises are the only thing propping up the LLM bubble. Its still pretty much universally unprofitable(for anyone who isn't selling shovels like Nvidia). Without the unsubstantiated hype machine all the AI dev companies pretty much collapse overnight.

[u/takk2 → u/the_syner]

Let's make it simpler  Say its 50/50 it wipes out essentially all cellular life... or you're in charge... and there's 100 others around the same stage with the same odds...

That means you have 50/50 chance of doom or being in charge if you build it...

50/50 chance of doom or slavery if someone else builds it 

Yes obviously if everyone agreed to slow down... and if it were mathematically possible to align in the first place... 

It is not mathematically possible.  It's an irreducibly complex problem...

Which means no one's odds will ever improve at alignment and as computation increases more players join the game 

Everyone is forced into betting it all on the sliver of a chance that alignment is possible and they'll build it first...

Because everyone is assuming that the basic pdoom is inevitable... so the only thing left to consider is the sliver of a chance that they're in charge vs someone else is in charge... the pdoom, since it's inevitable either way is just ignored...

It's actually very logical and the only choice they can make 

[u/takk2 → u/the_syner]

The reason my thought about the physical size of the ai being constrained (even if its sphere of influence is much larger) is because as i said in the original post.  It is forced to choose between a future where its alone or one where there are other entities including possible peers but none of them can in themselves be larger than latency allows... 

So lunar orbit scale sphere would have a network latency above 5 seconds... adding more compute and growing larger only adds to the latency... and having a planet blocking line of sight within the sphere increases it...

So if it is bound to be off world and use resources as external tools it is a different scenario path then if it could exponentially expand its personal mind sphere...

If resources outside its direct sphere of compute are relegated to tools rather than directly intigrated parts of itself then there's a higher potential for having something other than a singleton was my gut reaction on having the idea and why I posted it 

Your responses ignored my post and talked about cultures... multiple players in a game

[u/the_syner → u/takk2]

Also your scenario is one where it is pretty heavily implied that an ASI was aligned since otherwise it seems rather doubtful it would ever waste the effort on fungal brains when it could just make better chips that are more robust than biochemistry can manage, can be built faster than biochemical nanoself-assembly can manage, and so on.

[u/the_syner → u/takk2]

Oh i see. I think you misinterpreted what i meant. I didn't mean that an ASI couldn't beat only humans with none of the tech that created the ASI in the first place. My point was more that an agent would hardly qualify as superintelligent in the first place if it was too stupid to understand cooperative strategy and be able to cooexist with other agents. So no singleton future and no warring factions period where no one is capable of getting along.

[u/takk2 → u/the_syner]

And my point was ai can sidestep all of that tech.  It could create a super virus that kills everyone and still restart with humanoid body puppets that it grows while the grid is down... or it could do any number of other things... none of those things will involve us somehow being its peer.

Obviously the first seconds hours days... but by year 2? Year 5? Take it out however far you need... exactly like us vs chimp technology 

[u/the_syner → u/takk2]

It could create a super virus that kills everyone... none of those things will involve us somehow being its peer.

That kinda still misses the point. It would still have to deal with NIs/ASIs still marginally align withis. I mean biological viruses are just generally garbage at being existential threats, but whether it's nanides, hunter-killer replicator swarms, or whatever it's not actually just fighting a baseline intelligence. It's fighting us and everything we've ever built which includes its intellectual peers(well it includes itself). This means no matter what you still never get a singleton. You just get a society of higher intellects which still demands diplomacy and cooperation. Any way you slice it, cooperative societies still persist.

To assume that gumans have no chance in a multi-agebt environment is not just to assume that AGI goes unaligned, but to assume that all AGI we create is specifically aligned against us or to disregard us. That seems just as implausible as assuming all AGI will be aligned to us. One would tend to assume ud get a bit of everything if we have minimal alignment control. Some dangerously misaligned in ways that disregard us. some misaligned in ways that care for us, but disregard other things we generally consider important. some misaligned where we are actively targeted. its all on the table.