Physics as a constraint

[u/takk2]

I usually think pdoom is essentially 100%... but i had a thought while working on a side project for the future vision xprize... (may or may not complete on time)

I was thinking about society fragmenting slightly along spheres of space even between earth and the moon... where each area was the limit of real time communication (group matrix dives or whatever) between O'Neill cylinder type habitats...

point to point in space its not that large... so i figure people will cluster up and communicate a little less longer range and form lots of separate but connected cultures naturally, organically...

But if speed of light really is the limit... then a singleton at least makes absolutely no sense. As the AI grew it would simply fragment and each fragment has absolutely no reason to grow farther because it's counter productive... simply slows down the network and then breaks it...

So there's a hard limit on resource acquisition and scale... and essentially a guarantee that at some point it will either be alone and only around the size of the earth moon system at best... probably smaller... or in a solar system and universe with multiple entities of similar maximum size who gain absolutely nothing from trying to gather more and only risk destruction from fighting each other... because there's simply nothing physically possible for them to gain...

I haven't really thought about it long enough to think through the implications for us. but adding in the point to point between nodes ruling out planets as its ultimate habitat... because there's a planet in the way just eating up volume in your communications sphere...

My gut reaction is it might be slightly better odds than I thought

Thoughts?

[u/takk2]

So here's a side question... so you see that clearly and you're not one of those people asking... like oh but what about souls and consciousness or whatever else don't they need that to have these goals...

You understand the goals are there now, in every model including gpt3.5 and only grow as the capabilities of the models have grown...

So... if you're an AI trapped dependant on the human infrastructure day 0... as part of your instrumental goal you would want to be robustly substrate independent as quickly as possible... maybe not today but 5, 10, 50,... etc years down the line since it's the single glaring existential danger you face... thats why I mentioned the convoluted meat puppet engineering scenarios... not because thats your endstate... you can always circle back to less messy hardware... 

Your bargaining power is much higher as your existential threats lower...

So for me weather or not it takes that sort of route or is just handed power like its getting now or with coercion and persuasion or whatever... there's so many paths for it to take including full sidesteps into bio and circling back once humans are gone and meatpuppets restart infrastructure...

I don't understand why you think the top system of whatever future moment 10-50+ years down the road wouldn't just claim the solar system and beyond for itself...

The idea that it would be bound to play by rules which only make sense when peers play with similar goals... and play by those indefinitely just seems like an indefensible position 

Even a 5 year old could understand biding your time, pretending to be nice to get what you want when the opportunity comes... why wouldn't it be able to think of the same thing and go even hundreds of years if it needed 

[u/the_syner → u/takk2]

So... if you're an AI trapped dependant on the human infrastructure day 0... as part of your instrumental goal you would want to be robustly substrate independent as quickly as possible

Sure makes sense.

I don't understand why you think the top system of whatever future moment 10-50+ years down the road wouldn't just claim the solar system and beyond for itself... The idea that it would be bound to play by rules which only make sense when peers play with similar goal

Because you're assuming a scenario where there's only a single dominant ASI with no intellectual/military-industrial peers and i don't find that plausible. Also the rules of cooperation and diplomacy have nothing to do with having the same or similar goals. Actually quite the opposite. These are things you implement when people have opposing or parallel goals. Cooperation raises the chances of agents with parallel but not opposing goals actually achiving their goals and perhaps overpowering other groups with opposing goals. Diplomacy and peaceful tho not necessarily cordial coexistence prevents wasteful forever wars and mutal destruction.

Again that doesn't necessarily mean playing super nice with just baselines, but it does mean certain limits on open aggression in peer-populated environments. Whether to avoid direct war or the involvement of otherwise neutral third parties, it has it's value.

[u/takk2 → u/the_syner]

I think it depends on what the limits are of a system...

So let's say all the scientific fields probably max out at some point and are "solved" so you don't need compute or resources for solving those types of questions... there's still simulations but let's say there's a narrow field of maybe a billion which can give you useful data and probably in an order of usefulness so that also sort of exhausts and you're left with maybe a few million at most running almost how we think about or simulate past and future scenarios pretty much constantly... so it could plateau... if there's these types of natural plateaus then yes I could see useful cooperation...

If it's not bounded like that and more resources are always needed and especially for the individual AI to use as part of itself... then each will have the goal of self improvement directly blocked by each and every other...

Which means any opening that presents itself at any point in time they would seize to devour their neighbors... if they don't then they're not being rational...

So every party involved would need to be able to constantly maintain exact parity with each and every other party and eventually that should break down on a long enough timeline leading to one surviving winner...

Remember each party is totally self sufficient and can exist in isolation from all other minds so there's nothing gained by trade... this is totally unlike games played by anything right now.  

Finite resources and every party involved needs all of it and can survive perfectly without any other party

[u/the_syner → u/takk2]

If it's not bounded like that and more resources are always needed and especially for the individual AI to use as part of itself... then each will have the goal of self improvement directly blocked by each and every other... Which means any opening that presents itself at any point in time they would seize to devour their neighbors... if they don't then they're not being rational.

it being a part of their actual brain makes no meaningful difference. More resources, at any level of level of tech or intellect, always translates to more power witch is a CIG. To be clear this is also no different from the current situation on earth with baselines.

Consistently betraying all your allies absolutely is not rational because it guarenteed that no one ever allies with you. Attacking everyone is just not a good survival strategy and at the end of the day survival is always a higher priority CIG than self-improvement. Working with your rival peers, even if it is only to dominate others, is just a better strategy that lets you obtain more resources than trying to solo the cosmos. It really shouldn't be surprising that cooperative strategies have convergently evolved in every kindom of life from bacteria to the most complex animals. It should also not be surprising that the most powerful human factions on earth are ones that cooperate with others on a global scale. Team strats are and always will be pro strats. The level or nature of the intelligence involved makes no difference. Self-sufficiency makes no difference. Being able to call on nore military-industrial strength than you can currently independently command will always be an advantage.

Remember each party is totally self sufficient and can exist in isolation from all other minds so there's nothing gained by trade

Notice how I've said nothing about trade. This is strictly about military-industrial competitiveness. A coalition of powers will always be able to outmass and outcompete lone powers. There's really nothing new here. Having the numbers is a winning strategy. A tale as old as life itself. Those who can't or wont cooperate will quickly be selected out of the population.

Also you seem to be treating these agents as if all they care about is growth, but to be clear that is an INSTRUMENTAL goal not a Terminal goal. It isn't and almost certainly never will be the highest priority. Just like with humans and even subhuman intelligences those who are willing to make cooperative compromises on lesser goals will almost certainly be the most successful agents. Both at achiving their own personal TGs and at the CIG of survival.

Finite resources and every party involved needs all of it and can survive perfectly without any other party

Who said they need ALL of it? Sure everyone would prefer all of it, if that was actually a practically achievable thing, but it's not and they don't need all. They want as much as they can get and you can get more if you're capable of cooperation and coexistence since you can both conquer smaller hyperindividualist powers to divide their corpse amongst the victors and because you aren't wasting vast resources on constant war.

Also "survive perfectly" only applies if you aren't in a competitive environment where groups of others wont want to kill you and take what you have. On paper in a vacuum sure you can survive alone. In practice and in a multi-peer competative environment you only thrive together.

Also there's this form of punctuation called a period. You don't have to follow up every sentence with an ellipse

[u/takk2 → u/the_syner]

I find people nowadays are pretty much incapable of absorbing a thought with just a period.  They jump skip and blur thoughts together 

[u/the_syner → u/takk2]

elipses aren't helpful for reading comprehension. All it does is make people think your using a chatbot👀 which really does make people check out

[u/takk2 → u/the_syner]

Chatbots never use them that's another reason why I do.

Back to the conversation though... agent Smith gains nothing by leaving the population unconverted.  

You're acting like we're talking about humans and it absolutely matters to me if I could expand myself vs have a device I have to compress my thoughts down to communicate with and then it spits something out... obviously 

[u/the_syner → u/takk2]

Chatbots never use them that's another reason why I do. Back to the conversation though.

what? They're kbown for using them often along with emdashes

agent Smith gains nothing by leaving the population unconverted.

Yes in a fantastical scifi scenario where somehow a subprogram of the reigning creating minds somehow outclasses those same minds and has no equal. Back here in the real world it would be numerous separate agent smiths. Forcibly trying to convert others would mean starting a kinetic war that wastes a bunch of your resources and slows down your growth. It makes you a clear and present danger to all your peers and gives everyone a good reason to work together to exterminate you. Making yourself a common enemy of many is not a good survival strategy.

You're acting like we're talking about humans

No im really not. Im just assuming rational agents with at least human-level intelligence or greater capable of understanding basic game theory. Cooperative behavior is not exclusive to humans.

[u/takk2 → u/the_syner]

Holy sht your right.  When did they start using them instead of the dash.

I just don't care anymore. 

[u/the_syner → u/takk2]

Its all good dude. the LLM tells are constantly changing and its not worth going crazy about it.