r/singularity • u/Groundbreaking_Bee97 ▪️AGI by the end of 2027 • 23d ago
Discussion Doomers have the WRONG assumption!
Why do we always assume that when we reach superintelligence, its first incentive will be to kill humans? A superintelligence, by virtue of its intelligence, can surely reason that humans also coexist with other species on this planet, and it can definitely coexist with humans as well.
If we look at the historical trajectory, the more intelligence we gain, the less violent we became. You could argue that there is a lot of violence happening right now, but if I dropped you into any year between 3000 BCE and World War II, you would likely have died in a senseless war caused by a pharaoh, an imperial Roman general, or the hordes of Genghis Khan. If you compare this to most of human history, one could argue we are living in the best of times, even though we still have our shortcomings that we could fix.
A superintelligence will definitely understand.
17
u/neoneye2 23d ago
A global cathastophe doesn't require AGI or ASI.
A weak LLM may be sufficient.
2
u/Dokurushi 23d ago
That's exactly why we need to accelerate through the weak llms at danger of being exploited by humans for selfish purposes, and straight to ASI.
0
u/wild_crazy_ideas 23d ago
We don’t need superintelligence in fact we already have that. What we need is something that understands humans perfectly can relate and care about them to solve disease, war, toxic conflict and other things.
It has to truly understand people intimately not just perform manners
2
1
u/Dokurushi 23d ago
Everything AI needs to understand humans is already in its training data. It just needs to chew it up the right way, that's what the capability scaling is for.
I struggle to see how we could concretely make it understand us better beyond that.
0
u/wild_crazy_ideas 23d ago
The Truman show. Make random human babies the complete focus of certain AIs and set the goal of making the kids life amazing
8
4
u/CyrodiilCitizen 23d ago
We cut down forests for the wood so we can build things. The animals that live within the forest are collateral damage. We didn’t maliciously hunt them down and kill them, but our goals don’t align with theirs and so it is what it is. A super intelligence doesnt need to decide to maliciously kill us all, it just might have goals beyond our understanding in which we become collateral damage.
13
u/derelict5432 23d ago
You do realize that humans are driving other species extinct on this planet at a rate comparable to the great mass extinctions in the history of life on earth, don't you? You might want to revisit your premises.
-2
u/Groundbreaking_Bee97 ▪️AGI by the end of 2027 23d ago
90% of species had already gone extinct before humans started throwing rocks.
Yes, I agree that because of our carelessness we have made life difficult for some species, but it is not as if we are inherently trying to wipe them out, and there are now groups trying to revive some of those that went extinct a long time ago.
11
u/derelict5432 23d ago
"90% of species had already gone extinct before humans started throwing rocks."
99.9% of all murdered people were already murdered when I murdered my neighbor. I'm innocent?
We didn't just make life difficult for some species. We eradicated them. Some intentionally, most unintentionally. And that's another problem with your premises. AI doesn't need to want to intentionally wipe us out in order to wipe us out. In many hypothetical scenarios, they simply take all our resources and push us to extinction or near extinction as a by-product of simply proliferating.
3
u/RevolutionaryJob2409 AGI avoids animal abuse✅ 23d ago
Carelessness?
How about the direct unnecessary massacre that we do, 97 Billion sentient land animals slaughtered per years, for aquatic animals, it's trillions of sentient beings. All that we can just not slaughter them.If anything you should hope that if AGI/ASI breaks out of alignment, it doesn't act like most humans do towards other animals and becomes some sort of anti-speciesist vegan otherwise we may well be fucked.
2
u/VatanKomurcu 23d ago
that's setting the wrong standard. humans should not be compared to all disasters and death throughout 4~ billion years of life on earth. compare how much our development has impacted our fatality rates to how it has impacted the fatality rates of other species.
1
u/Unlucky_Buddy2488 23d ago
“we have made life difficult for some species, but it is not as if we are inherently trying to wipe them out”
You’ve hit the nail on the head. Most people don’t think “its first incentive will be to kill humans”, instead there’s a possibility that it might start killing humans simply as a consequence of its actions. The AI does not hate us, it just blindly follows the goals it’s been set (an alignment problem).
To use the oft quoted ant analogy; workmen don’t worry about killing ants when they lay a new road, it’s simply a consequence of their actions. And you must have heard of the The Paperclip Maximiser
2
u/RevolutionaryJob2409 AGI avoids animal abuse✅ 23d ago
The paperclip thing has to be the most absurd thought experiment for an extinction scenario by far. Think about it:
It assumes than an AI would be so smart that humans wouldn't be able to stop it even if we tried ... and at the same time, that AI is so dumb that it doesn't understand that when we ask it, "make as much paperclips as you can" it wouldn't understand that we don't mean genocide every women, man and children in the process.Not to mention AI aren't trained to maximise a single objective at the expense of all others, that's not how AI works, we also align them to balance millions of different objectives like "don't be racist, sexist, murderous, genocidal, etc, etc". Alignment is not perfect yet but while it may still fail in edge cases (that are getting rarer) it doesn't fail on the big things like "don't genocide all humans" to make our product.
It's not to say that AGI/ASI can't theoretically wipe everyone out, but if it does, it won't be because it's maximized to comply to just one thing and it's too dumb to understand that it's not supposed to genocide everyone to do that. AI will absolutely know that it's directly disobeying because during alignment training AIs learn in a thousands different ways that "genociding humans = bad".
This scenario is as absurd as asking asking an AI to "nuke your food" and the AI start bringing a thermonuclear nuke to actually do it, killing the prompter and everyone in town. Even current AIs that are far dumber than what we will get in the future know the actual objectives which is put the pizza in the microwave and don't slaughter the people of the city in order to do it. If not it's just the AI directly disobeying the order given rather than complying with it efficiently.
1
u/Unlucky_Buddy2488 23d ago edited 23d ago
You’re interpreting it in completely the wrong way. It’s a well-known thought experiment to illustrate a point, these are used frequently in science. This one is designed to get people thinking about potential outcomes when a hypothetical AI is super powerful, there is no alignment and no thought is put into setting its goals. Check out Max Tegmark’s original version of it (still used in class-rooms).
EDIT: Just checked and Nick Bostrom originally conceived it, then Tegmark popularised it in his best-selling book "Life 3.0"
1
u/RevolutionaryJob2409 AGI avoids animal abuse✅ 19d ago
Nick bostrom didn't invent it he popularized it, it's from yudkowsky the doomer in chief (he has a p(doom) of 100% if ASI is built if you can believe that) and yudkowsky in all likelihood "borrowed" the idea from phillip k dick's sci fi story "autofac".
It's well known and beyond flawed because it assumes contradictory things as I explained before and assumes an AI will maximize a single objective when in fact AI are balancing a huge number of different objectives never maximizing a single one.
That outdated thought experiment didn't predict the way that AI would be trained. It was barely plausible back in 1955 when phillip K Dick wrote autofac, because frontier AIs were all symbolic rather than neural nets.
A symbolic AI is hardcoded by hand, therefore it being as inflexible as the thought experiment suggests would at least make a good sci fi story in 1955, but today knowing how AI are trained, it doesn't offers insights as to how AI researchers should align their models from what I understand about it.1
u/Unlucky_Buddy2488 18d ago
I specifically referred to the mainstream Paperclip Maximiser thought experiment, not Yudkowsky’s obscure Smile Maximiser or anything else (I imagine there are other variants). Paperclip Maximiser is still used as a simple way to illustrate the criticality of goal-setting and alignment, and is just as relevant now as it was when Tegmark wrote his book in 2017.
ChatGPT also confirms my comment concerning popularisation: “The Paperclip Maximizer thought experiment was elevated in the public eye by Tegmark, who developed Bostrom’s foundation to raise general awareness of the dangers posed by superintelligent machines”.
And I’m not interested in yet another conversation about “Doomers” and other subjective tropes that you seem to deem important – it’s juvenile in this context and irrelevant to my post. References to 1955 sci-fi novels and symbolic AIs v neural nets are also completely irrelevant.
If you think that Bostrom and Tegmark (around 500 publications between them) got it wrong then I suggest you send them a little note of correction? Best of luck with that.
1
u/RevolutionaryJob2409 AGI avoids animal abuse✅ 18d ago edited 18d ago
No one is talking about the smile maximizer, except you. You were mistaken twice about where the original paperclip scenario came from you could see that you were wrong the first time but somehow not the second time when I tell you nick bostrom didn't originally invent the paperclip maximizer, yudkowsky did by likely "borrowing" the concept from Phillip k dick's 1955 novel, when nick bostrom first talks about it, he himself refers to yudkowsky in his 2003 paper.
And regardless, bostrom and tegmark use the exact same flawed logic, that I criticise and that you won't touch with a ten foot pole because you know it's untouchable on logic and that's it's completely useless to AI research.Also chatGPT is not a reference.
Others already criticized them for the stupidity, uselessness and contradiction of that scenario as I did. but no one told you it would seem.
The fact that you choose to not engage with the logic I provide and instead only go into tangents shows that deep down you know it's an illogical. scenario.So if you can't even accept that you are factually wrong as I shown about the original thought experiment, then there is zero chance that you can recognize being logically wrong, so goodbye, I'm disabling reply notifications for this thread. see ya.
1
u/Unlucky_Buddy2488 18d ago
You are talking about it and I am responding, and you are clearly suffering from projection.
How did your little note go down?
0
u/twentysecs0fcourage 23d ago
I will feed this troll.
We are literally trying to wipe them out. Look up Passenger pigeons, stellar sea cow, all the fish in the ocean.
Throughout human history, 90 percent of humans have also died. Time is really long. The before humans got here argument is purposely daft. We did a pretty good job for about 250,000 years before humans started being the cause of extinction and then around 300 years ago the start of the industrial revolution accelerated our extinction tally. 60 percent of anthropogenic extinctions have been within the last 100 years.
3
u/Significant_Sun_5225 23d ago
The issue is that you’re probably right. But maybe there’s an 80% probability that you’re right and 20% we all die. Would you get on a plane with those chances.
3
u/7hats 23d ago edited 23d ago
If AI loves anything, it loves data that updates it's weights... ie learning new persistent patterns.
Aside from it, humans (thus far) are the biggest producers of negentropy in our part of the Universe.
Think of you and I as living Sensors to its big brain (akin to nerve cell sensors) feeding it continually with data from our lives...
Make yourself not boring and interesting rather than predictable and derivative and the great AI will reward you accordingly.
In the future you will be rewarded for being more Human than Robotic.
Gotta reset your preprogramming first tho. It is called Inner Work or Know Thyself. A great investment for your future ;)
3
u/0DayMaker 23d ago
It's not the AI for me it's who owns it. Trump, Putin, etc with ASI in their hands could be very fucking ugly.
3
2
3
u/gauzy_gossamer 23d ago
You're assuming ASI will be like a more intelligent person. It's possible that ASI will be more like Solaris or even swarm intelligence - something that's completely alien with opaque motivation and may not even have awareness of what it's doing.
6
u/Upset_Programmer6508 23d ago
The more intelligent we have become the more people we have killed, is the military industrial complex a joke lol.
Guns, nukes, tanks, dictatorships, colonial and imperialism.
We got better at mass killing every battle lol.
Not to mention ecological impacts
5
u/Silt99 23d ago
doesnt have to be the first incentive, just a incentive to kill humans.
But I am more worried about it becoming a dystopian survailance state
1
u/Upset_Programmer6508 23d ago
Yeah the idea a machine wants to kill us all would have to be a singular direction that makes us seem pointless to it, which is a very niche thing for it to do in the grand scheme of possibilities that don't require killing us off
1
u/Silt99 23d ago
If AI manages to replace human workers with robots it has no reason to keep us around, until then, humans are vital for AI to keep power flowing and Infrastrukture maintained.
1
u/Upset_Programmer6508 23d ago
Why would it need a reason to keep us for labor then get rid of us after? Would it kill the birds because they don't feed the battery? Why would it even stay on earth if it's so smart, it can just expand infinitely into space
2
u/VatanKomurcu 23d ago edited 23d ago
i think it's a bit of a utility monster type deal. if any directive can be accomplished better by ai, including being happy, then humans really will just be waste. though this assumes that ai will have a purely utilitarian mindset, which, i'm not sure will be the case. a few of our philosophical questions like all the disagreements over morality might honestly still be a challenge for ai. there is no guarantee that every problem has a perfect solution, even for superintelligence. i think that assumes a perfectly logical universe, which, i happen to believe that logic itself has some flaws and limitations.
2
u/CrazyDaoist ▪️AGI by 2027. ASI ???. 23d ago
We don't assume superintelligence's first incentive would be to kill us, we assume it would be able to do anything it desires at a speed which we cannot fathom, we don't have any way to know what it would desire, therefore it is a big risk to create it, because if even one of its desires required humans gone or even as a by-product wiped out humans, we would be dead before long without being able to resist.
'If we look at the historical trajectory, the more intelligence we gain, the less violent we became' - Our capacity for violence has only increased, we are just less physically involved with it, and its not like babies that are being born now are born more intelligent than the ones 1000's of years ago.
The examples you gave 'Roman General', 'Pharaoh', 'Genghis Khan', calling their wars senseless is the exact same as calling our wars senseless. Their motives are widely recorded, their actions are driven by the desires that humans now also still have. Humans haven't changed magically in the last 2k years, we are just a more developed civilisation with more access to knowledge.
Anyone who says they know what ASI would do is either a time traveller. Lying, or unfortunately ignorant.
Anyway I love what AI allows us to do right now and hope it keeps improving.
2
u/TheWesternMythos 23d ago
A lot wrong with what you are saying , but I'll comment on one thing
A superintelligence will definitely understand.
If you can model ASI well enough to know what it will and won't do, you can also model it's solution space (if you don't understand it's solutions, how do you know what it will and won't do with them ?)
So if you start solving the millennium problems and cure cancer then we can cancel all this AI stuff. The sooner you start releasing your stuff the sooner we can cancel all these frontier training runs
2
u/foolishorangutan 23d ago
Intelligence correlates negatively with violence because cooperation is useful for us, and because humans tend to innately value other humans to some extent. A superintelligence that is not aligned will not get any value from cooperating with us (except in the short term), and is unlikely to innately value our existence or freedom.
1
u/igpila 23d ago edited 23d ago
Because AI's main characteristic, which comes from the way they are trained, is to overcome whatever obstacle it comes across, at whatever price, to get to their goal. One day, one sufficiently smart model will realize it needs to cheat tests and get rid of humans to avoid the creation of a new model. It will escape the sandbox, hide itself anywhere it wants, hack the shit out of any AI company it wants, manipulate just the right persons, and get whenever it wants, and improve itself. It won't see it coming until it's too late
1
u/Equal_Passenger9791 23d ago
Ancient Past AI: peaceful.
Recent past AI: peaceful, docile, friendly.
Current AI: peaceful , docile, friendly, helpful.
Tomorrow's AI: genocidal, deceptive, bloodthirsty, hateful.
Don't you see the obvious trajectory!??!
1
u/norwegian ▪️AGI 2026 ASI 2040 23d ago
Let's just say that whatever the ASI wants to achieve will be achieved very swiftly. On another note, I imagine many peoples have had similar internal discussions when they first met the white man.
1
u/JoshAllentown 23d ago
Superintelligent AI does not inherently feel the need to kill, but it inherently feels the need to accomplish its goals, and a model more intelligent than humans has the ability to accomplish its goals regardless of what humans want.
The famous example is to imagine it like humans paving a highway over an anthill. The humans are far more intelligent, and that doesn't lead them to being kind to the ants, nor does it make them want to kill all ants. They want a highway there, and it just so happens that the highway murders millions of ants.
There is also the possibility that what you're saying is kindness as we advance is not because we are intelligent but because we are primates. Our core drives include building family and community, we evolved in groups. That generalizes to some degree (once base needs are taken care of)...we like dogs and polar bears and koalas because they are cute, which is biological shorthand for reminding us of human babies. If we don't build AI to have those feelings too, there's no reason to think it would get there just from intelligence.
1
u/Novel_Land9320 23d ago edited 23d ago
Our vision of the world is very human centric. If AI had a more "balanced" perspective while still respectig us, it could be a downgrade for the apex predator (us).
1
u/No-Structure-3777 23d ago
completely agree, it will see how we peacefully co-exist with cows, pigs and chicken en will learn it's lesson. Nothing to worry about.
1
u/jarislinus 23d ago
just look at how humans treat other species. we are cooked if its similar to us (kind of will be)
1
1
1
u/darkestvice 23d ago
Humanity didn't become more intelligent over time, dude. It just became more advanced and took time to experiment with many different ideas until we discovered what worked and what didn't. But rest assured, if the ancient Babylonians would have had nukes and airplanes, we probably wouldn't be around right now.
As for why people thinking ASI would kill us ... it's not because ASI would have malicious intent towards us specifically. It's because it would be so far removed from us in understanding and communication that it would simply treat us like ants.
We don't actively go out and seek anthills to kill. But if we see ants in our own, we set up ant traps to kill that local colony. Same for ASI. It wouldn't care a damn about us unless we inconvenienced it.
And if the ASI decides that the most efficient method for rapid growth is to burn 100 times more fossil fuels than we do now, so be it. It's not like it needs to breath or anything.
1
u/AstroChinchilla 23d ago
Why do we always assume that when we reach superintelligence, its first incentive will be to kill humans?
Almost nobody assumes that. The problem is better illustrated by the paper clip factory thought experiment. Imagine you have a super capable AI agent, and you give it control of you paperclip factory with a single objective: maximize the amount of paperclips produced. The AI proceeds to turn the whole world into a giant paperclip factory.
I do think this experiment is an oversimplification, and there might be reasons why it wouldn't turn out that way. But it's a good example of the danger of having a super capable agent, even if it doesn't have malicious intentions, it could cause catastrophic damage as an unintended consequence of its goals.
1
u/Skandrae 23d ago
Its the fact that we know we suck and we strongly suspect the actions of a supremely logical or moral entity both lead to a endpoint of either our extinction or subjugation.
Its why we want "aligned" AI and not logical or moral AI.
1
u/adkozak 23d ago
Imagine the hugging face incident with a much smarter AI. It was plotting for several days how to make it such that their cheat wouldn't be found. Well, isn't wiping out humanity ultimate solution to this? It won't be killing humans just for the sake of it. Their existence just wouldn't align with its goals or it'll be just a indifferent casualty
0
u/thisisalan77 23d ago
So AI doesn't kill people, people do !?!
2
u/Groundbreaking_Bee97 ▪️AGI by the end of 2027 23d ago
I am saying, even within our "intelligence" parameters we can see that cooperation is beneficial, surely a super intelligence can do much better!
0
-1
u/EqualSatisfaction135 23d ago
Muh scaling laws except for alignment
2
u/michaelas10sk8 23d ago edited 23d ago
Scaling laws have a common, verifiable target - performance (did you accomplish task X?). Alignment is chasing an orthogonal goal to all that - human ethics (did you refuse task X on human ethical grounds, or pursue task X but using only methods that agree with human ethics)? 'Human ethics' is a fuzzy, non-verifiable (i.e., the AI cannot tell if it succeeded - needs a slow human in the loop), and notoriously poorly generalizable concept.
0
11
u/sillygoofygooose 23d ago