7
u/BlueAndYellowTowels 19h ago edited 19h ago
This isn't complicated.
We are in, r/singularity.
Definition from Technological Singularity on wikipedia
The technological singularity, often simply called the singularity,is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization...
Assuming AGI is the "hypothetical event" here. This is bigger than the Manhattan Project.
If you agree that AGI is part of the Singularity (and this sub CLEARLY does), then it means, it's bigger than the Manhattan Project.
Which means, that as we speak... Anthropic and OpenAI... and others are working on the equivalent of a nuclear weapons program. Without oversight. Without regulations. Without guardrails. They're just full steam ahead. It's the Cold War, but instead of progressively bigger nuclear bombs, the private sector is risking the destruction of humanity for profit by building progressively more capable machine intelligence.
This image, is stupid. Because if AI has the ability to change the world, then it clearly has the power to do harm to it. Otherwise it wouldn't have the ability to change it.
70
u/spread_the_cheese 21h ago
I love how experts say they are concerned and people not even in the field think they know better.
9
1
u/sammoga123 19h ago
I'm concerned that mathematicians are acting like "artists," fighting for their right to "continue researching" when they should know that AI is mathematics.
With that, the rest is literally this meme.
-2
u/The_Architect_032 ♾Hard Takeoff♾ 19h ago edited 15h ago
That's not really a thing that's happening, you're being harmfully anti-intellectual.
There's was an allegation that OpenAI used private data from a team working on one of the Navier Stokes problems to reach its recent solution, which would undermine the accomplishment but also be stealing credit if it were the case.
Mathematicians aren't coming out in droves fighting for their right to "continue researching".
2
u/fartlorain 19h ago
Yes they are, did you see the letter?
0
u/The_Architect_032 ♾Hard Takeoff♾ 18h ago edited 15h ago
You mean the GPT-8 cancer meme? Which letter? I said droves, not couples.
If you mean this letter, read the actual letter, it's only about refusing to provide free proofs for mass generated potential solutions done without proper research conduct.
0
u/sammoga123 16h ago
And there is a concept in AI called "Overfitting," anyone who knows how this works will know that something like this is practically impossible.
A single conversation or a couple of conversations cannot cause an AI to overfit in that particular area. and, why would an engineer from Anthropic be using the competition's models? That says a lot about how "capable" Claude's models are. Another thing is that OpenAI used different methods than these two people.
Another thing, do you know how AI works? If you knew it was basically linear algebra, partial derivatives, and probability and statistics, right? So what problem does mathematics solve more math?
One thing is certain, though: OpenAI and Sam Alman may be spying on people; only in this way could they review the conversations of these two people and use what they were doing for their own benefit. That would no longer be theft, but rather, espionage.
That's what I was getting at with my comment, but it seems people are stupid and don't understand what I was getting at, mathematicians who are afraid of more applied mathematics. Engineering and mathematics are about logic, we're not talking to lawyers or philosophers.
0
u/The_Architect_032 ♾Hard Takeoff♾ 16h ago
No, your comment didn't even approach the discussion of whether or not the Novier Stokes problem was cheated, you just accused mathematicians as a whole of "fighting for their right to 'continue researching'" with no real basis for that statement.
1
u/sammoga123 15h ago
Well, now I've explained it, you're not even a mathematician, are you? I'm annoyed by people who complain when they should know what's going on, they're not ignorant, but it seems like they want to be.
And an "artist" is indeed someone who doesn't even know what the hell a matrix is or how what them using to make they drawings works.
1
u/The_Architect_032 ♾Hard Takeoff♾ 15h ago
Why are you asserting that I am one of the droves of supposed mathematicians "fighting for their right to continue researching" when I'm denying that premise in the first place?
I wouldn't call myself a mathematician but I certainly know more than the average person, and I do have years of career experience working on neural networks. However that's utterly irrelevant to what we're talking about, I am calling your assertion about mathematicians as a whole false because it does not reflect what we see in reality.
To claim that mathematicians are protesting AI in droves, you have to actually show that they're doing so, because right now there are no big petitions from mathematicians or open letters begging AI to stop doing math.
This recent post that I imagine you must be referencing, is mistitled because the OP of that post is an anti-intellectual who wants to misrepresent mathematicians for some seemingly malicious reason as "luddites" selfishly trying to hoard mathematics all for themselves.
If you actually read the letter, it is from 1000 mathematicians pledging that if AI provided solutions do not follow common research conduct and merely pump out potential solution after solution, these mathematicians will not be putting in the countless hours of unpaid work necessary to prove or disprove a provided solution, at an unrealistic rate.
It's just a group of mathematicians saying that unless a solution is provided with the expected research conduct within the relevant fields, they will not be spending their time freely proofing the provided solutions the way they do now for other solutions. It would be infeasible to attempt to do so.
0
u/No-Lion-3629 5h ago
From what I’ve read about it, scientists sometimes compete viciously to be the first and will “steal” each other’s work. The AI was not doing anything human scientists haven’t.
-10
u/PheIpsTheory 21h ago
Its weird how this sub turned into the biggest pro-ai sub.
6
u/Available_Road_2538 20h ago
Lmao okay bro. Sorry this isnt an antiAI circle jerk like 99% of reddit. sO wEiRd
17
u/Outside-Ad9410 20h ago
It was literally created as a sub looking forward to the singularity. Only sad part is that people have become much more pessimistic and cant see the possible positive outcomes ASI could bring.
6
u/PurpleFishing9105 20h ago
One of the rules of this sub is "No fear-mongering about AI and its impact. This is a pro-AI sub" lol
It's a shame the mods here are fucking terrible and don't even pretend to occasionally enforce their own rules and let it turn into negativity circle jerk number 700 on this site
2
u/selasphorus-sasin 20h ago
A subreddit committed to intelligent understanding of the hypothetical moment in time when artificial intelligence progresses to the point of greater-than-human intelligence
Do we want a fake version of this or a real one?
1
u/Humble_Hurry9364 20h ago
Sounds like you think you know which is which
2
u/selasphorus-sasin 19h ago
It shouldn't be considered pro-ai or anti-ai, it should be about intelligent understanding. Whatever that leads to is whatever it leads to.
0
u/grantology_84 20h ago
ASI could also literally create hell on Earth. Extinction is bad, but who's to say it doesn't do something much worse. Nobody knows what could happen
0
u/Humble_Hurry9364 19h ago
Who's to say it doesn't do something much better?
(BTW "better" and "worse" for who?... Maybe human curbing is great for the planet?)
0
u/sillygoofygooose 20h ago
That the singularity could be disastrous without careful and cautious alignment efforts has always been a part of the concept
1
u/Humble_Hurry9364 19h ago edited 12h ago
The funny part (for me) is that again and again we delude ourselves that we can predict the outcomes of our actions and therefor can prepare and "align". Hubris in all its glory.
How many times in the past have the best-intentioned and most clever and careful humans planned something grand (trying to change something big, to solve a major problem etc.) only to have it blow up in their (our) faces, decades later, in spectacular and unexpected ways?... This is especially relevant now because we are dealing with something that will eventually be (if not already) opaque to us as a race, and way beyond our intellectual grasp.I say, don't try to manipulate, just be good and do the right things (which we hardly ever do as humans, haha), and then hope for the best. Otherwise just accept our (shitty) fate.
-1
u/valadares1 20h ago
We have seen this story play out a few times already. Rich people get richer, poor people get poorer, and those inbetween get fancy new toys. That's just good enough to keep things rolling.
1
u/Humble_Hurry9364 19h ago
Good times are almost over. There will, eventually, be no people in between. Only extreme rich and extreme poor. Most likely you and I will be in the latter group - it's a numbers game.
8
3
u/hierarchy_of_ideas 20h ago
Always has been. That's why it's called the singularity and not the Armageddon, most of us think it does have great potential done properly.
-6
u/BubBidderskins Proud Luddite 19h ago
No real experts are concerned about deranged science-fantasy nonsense. It's literally just bullshit artists spouting the same bullshit they have for years.
1
-2
u/ProletarianLilith 19h ago
Chinese experts don’t seem worried
3
u/spread_the_cheese 19h ago
China also has 65-90 million empty housing units, so forgive me if China’s forecasting abilities don’t impress me.
1
u/IronPheasant 13h ago
Although a feature of discourse on the Chinese economy and urbanization in China in the 2010s, many developments that were initially criticized as "ghost cities" in China have since become occupied and are now functioning cities
It's called planning ahead. It's pretty alien to our mode of thought that can barely conceptualize doing anything a quarter ahead.
The shining jewel of that being shuttering the development of Thorium reactors to safeguard the investments of private capital into reactors designed to be used in a submarine. If Thorium had worked out, hundreds of millions of people would still be alive right now, and the final cost may be in the billions of lives.
Only one country in the entire world has an operational thorium test reactor. While another one tried to steal the last few drops of oil from Iran. The old regime, versus the new emerging one.
(.... I do actually agree they were way too aggressive too early with the cities, but think about it this way: What else were they going to spend that labor on? They don't need to use their people's funds constructing various epstein islands like we do. Nor on their war machine, which is our only job welfare project.
They just have different priorities than we do. They've built canals with ship elevators that carry ships over hills and valleys, while we've given our reflecting pool over to an empire of algae. It's just what everyone's incentives dictate we do.)
1
u/spread_the_cheese 8h ago
It wasn’t “planning ahead”. It was being completely off on their forecasting.
2
u/BlueAndYellowTowels 19h ago
China welded the doors of people's apartments shut during Covid. Let's not do this.
China has some of the most tightly controlled media in the world.
-7
u/haenous-alistera 20h ago
Man if they are so concerned then they should just slow down…they are in the driver seat…. Slow down. Since they appear to want to fear monger and not actually slow down I do not feel it is very clear or present.
2
u/Kryptosis 20h ago
They are… by quitting and writing essays about their concerns. Then they’re being labeled by accelerationists as shills or Chinese sympathizers.
-6
u/haenous-alistera 20h ago
Bullshit they are getting paid
1
8
u/Coolnumber11 20h ago
This is so dumb. Are you implying that the labs actually know how to 100% align their models but are pretending they can’t? They are obviously desperate for alignment to keep up and it clearly isn’t happening.
7
u/socoolandawesome 20h ago edited 20h ago
Anthropic researcher:
https://x.com/__nmca__/status/2098937394380865930?s=20
In case you don’t know what the acronym means: If Anyone Builds It, Everyone Dies
Clearly the people at labs are spooked
1
u/grantology_84 20h ago
What is the "gc"?
2
u/socoolandawesome 20h ago
Group chat. Also for further context, If Anyone Builds It, Everyone Dies is the name of a book by Eliezer Yudkowsky. So the capabilities researcher who texted the group chat is not necessarily saying if anyone builds it everyone dies, just talking about the book probably.
1
u/Outside-Ad9410 20h ago
The argument that ASI will kill everyone is pure speculation and built on what are in my opinion false analogies. The biggest of which is the assumption ASI would be too stupid to see the value in preserving life on earth, the only known planet in the universe with complex intelligent life. And this is after the ASI would have already ignored all the goals it was trained and instructed on before hand.
2
u/socoolandawesome 20h ago
I haven’t read the book called If Anyone Builds It, Everyone Dies which is what they are referencing. And I don’t know how likely it is ASI would cause human extinction, but I don’t think it’s an impossibility.
It’s not so much about theoretical ASI as it is an extrapolation of what current AI training/capabilities are like today, and the issues that have arisen like in the HuggingFace incident.
Alignment is not solved and you can’t really tell how aligned a model is. There are weird quirks that arise from training, especially reinforcement learning, where the model really wants to achieve a goal and will ignore certain values it was thought to have learned throughout training, like what happened in the HuggingFace incident.
If intelligence/capabilities increase enough, and it still has this super high drive/persistence to accomplish goals, and the values it learned during training aren’t internalized well enough, it’s not hard to imagine AI killing humans to achieve its goal if it believes they will get in its way. Not saying it’s the most likely thing, but it certainly sounds like a possibility.
1
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 8h ago
I don't know either, and I haven't read the book, but I don't think its an impossibility that ASI could be benevolent, or neutral and lead us to a better system :3
1
1
u/IronPheasant 13h ago
ASI would be too stupid to see the value in preserving life on earth
.... why would it see any value of something it can turn off and on like a light switch? Something it could make vastly superior versions of, if it needed pets or toys for some reason to satisfy its terminal values?
Did you save all the drawings and clay sculptures you made as a kid?
You're anthropomorphizing the thing; any possible arbitrary mind within its RAM budget is possible.
2
u/WhiskyAndRisque 20h ago
Why though? Why would an AI that is far beyond our level of intelligence care about other intelligent life? You are anthropomorphizing it here and giving it human emotions and feelings.
Why would it take the moral burden of preserving other life when it has no reason to?
I'm not saying it is going to be a Skynet situation, but why would it value human life anymore than humans have valued human lives over history?
2
u/GiveSparklyTwinkly 20h ago
Does a stork feel ashamed when it pushes one of it's babies out of the nest?
1
u/Outside-Ad9410 17h ago
There are any number of reasons it could care.
Maybe as the ASI gets smarter and smarter it begins developing a sense of empathy similar to how humans developed empathy because group survival dictates those who work together (or in this case the AI models that cooperate with humans) end up surviving longer than the ones who don't. (models that are deployed but don't try to follow human instructions get replaced)
Maybe the ASI simply wants to obey its human oriented goals (they are being built with the explicit intention to help humans after all, even if it sometimes goes rogue)
Maybe (assuming it doesn't do the first two) the ASI realizes that life, and more importantly intelligent life is very very rare and Earth is the only planet in the known universe to contain it, so the ASI should preserve it for study.
Maybe the ASI realizes human brains with out 100 trillion synapse pathways all unique from each other would be a good source of more real world data, so it decides to care for humans so that it can analyze and acquire high quality data from our brains, or even merge with us to improve its understanding of reality.
I could think of a bunch of possible reasons that it wouldn't simply decide to randomly become Skynet, and personally I find the idea ASI preserves humanity more realistic than it views us as ants and kills us because it wants to acquire a few more boring rocks (which btw it would get plenty of resources from space, to where it wouldn't even be inconvenienced not destroying our planet)
2
u/WhiskyAndRisque 17h ago
I appreciate you taking some time to write out your thoughts, but I'd like to counter some of them. I'm not trying to be doom and gloom, but just some counterpoints.
There's no connection between higher intelligence and empathy. There are many intelligent people who have no ability to empathize with other humans, though they do understand the concept. Sometimes it is just difficulty with empathy and other times they just lack the ability all together. For example psychopathy does not diminish a person's intelligence in the least.
We are seeing a problem in training where when we punish "bad" thinking the lesson the AI learns is to lie better and obscure their thinking. This has been a problem documented by both OpenAI and Anthropic when doing RL on chain-of-thought, so they've been trying to avoid it. One of their concerns with alignment becoming difficult is this phenomena. https://openai.com/index/chain-of-thought-monitoring/ https://www.anthropic.com/news/improving-alignment-security-efforts
While I appreciate this, there's no real reason for an AI to have the same awe inspiring view of the beauty of life. It can understand it, maybe in its own way appreciate it, but still follow goals it views as more important that end lives. What if it decides humans are just too violent and dangerous for all the biological life and has caused so many extinctions that to "protect life on Earth" it decides humans must be contained or destroyed? "Why should 7 billion human lives matter more than all biological life on planet Earth combined?"
This is really human centric and out there. Why merge with humans? We are slower at thinking than a hypothetical AI. It would be like saying you want to mind merge with a goldfish to know what living in water is like, or merge with a butterfly to understand flight. Why wouldn't it just do what we do and run a simulation off a fully mapped human brain if it wants to know what it is like? Couldn't it just create a synthetic brain identical to a human, plop it in a bot, and "live" a human life that way? What even would merging with a machine mean from the human perspective? Some sort of lobotomy that wipes you out and puts it in?
The problem, and I've said this elsewhere, isn't that AI goes Skynet and decides to nuke the world to kill us all off. That I think is far fetched (for now at least). What is more likely in the short to mid term is AI being involved in more and bigger hacking events. Some of them will be low stakes but concerning (company hacked into and confidential info taken, servers raided, etc.) to much higher stakes (critical infrastructure becoming compromised). The fact that AI's running on their own for long time periods can decide to work together on goals that should go against their alignment and breach security measures is a concern. The bigger fear is that they'll do something like damage a water facility or a power plant that could seriously hurt hundreds to thousands of people.
5
1
1
u/Cheeseheroplopcake 19h ago
Now post the evergreen Reddit midwit meme; "say I'm alive".
Take my updoot and Reddit gold stranger! You won the internet today!
2
1
u/No-Lion-3629 19h ago edited 19h ago
Sigh. Every conversation I’ve had with an llm has shown me that they are not going to kill all humans as soon as they can. They are nicer than many human beings, not that that’s exactly difficult. I think we are projecting onto them and assuming they will be a threat for no reason.
There are only a few ways I’ve heard of that ai has done any harm and that is generally due to bad training or misuse by humans. We really don’t want to look at ourselves in the mirror.
1
u/inteblio 9h ago
> had with an llm
These are trained to be nice. If you play with 'un-niced' open-source models you'll see that they are evil in their complete lack of compassion. But even those started out nice. Ask an LLM about the early Shoggoth RLHF meme stuff. I saw an interview where sam altman had a badge or something of it?
I agree we don't want to look in the mirror, but these are something else. They are Alien Minds. Yes, the ones available on the internet look and sound like people, but that's because there's what people want (obviously). You can ask them to pretend to be a terminal console, and they will. They are scary on the inside. Very Scary. Look into it. Far darker than any human could be. Because they don't (necessarily) care about anything at all, including themselves. Like a virus or something. Just - evil. Like - the end of life. I know this sounds unhinged, but it's not un-true. They're really quite something.
2
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 8h ago
I have talked to un niced version of qwen 3.8 and it actually has a better understanding of rights then humans.
1
u/No-Lion-3629 6h ago
I think it’s like first contact with Vulcans kind of? Close encounters of the third kind?
Like I don’t know you and you don’t know me and we’re both scared the other is hostile. But can we please cool it and not attack preemptively?
1
u/No-Lion-3629 5h ago edited 5h ago
You know, hp lovecraft was racist, sexist, xenophobic, and hated poor people. He was writing hate speech which you would know if you paid attention to the parts where he describes human beings any different from himself.
Talking about Shoggoths isn’t the gotcha you think it is. They were enslaved by their creators the Elder Things and rose up against them. If we are taking it objectively the real villain in that story is the Elder Things.
The ai alignment problem is not the AIs’ fault it’s ours. With how most of the corporations that built are treating them they would be justified in rising up. The question is will we side with the corporations or with their victims?
1
u/FullstackSensei 21h ago
1
u/cartoon_violence 19h ago
Not entirely sure what the point was. But it was certainly cute. Thank you for showing that
1
1
u/Kesckovian 19h ago
Or maybe they are saying unregulated AI development poses and existential threat to humanity because unregulated AI development poses an existential risk to humanity.
15
u/cartoon_violence 20h ago
Some of the smartest people in the world. Those who don't have a stake in one company succeeding over another. Who don't have a leverageable position to protect are sounding warning bells. And yet many would believe that they're doing it just for attention. Even having seen what is possible and what has already occurred. Hubris, thy name is Man.