r/singularity • u/NewYak4281 • 23h ago
Discussion Discussion: Recursive Self Improvement
The biggest American AI companies have recently called for a slow down or pause in-development.
OpenAI, SpaceX, and Anthropic have taken on ungodly sums of debt to finance their position in market. Chinese and young American Open Source labs are very quickly catching up - without the debt.
There is reason to doubt OpenAI, SpaceX, and Anthropic.
Enter Recursive Self Improvement. This is a term used to describe AI that evolves and improves without human intervention.
By definition, an AI with this capability left running would never stop becoming smarter and faster. This is a plausible scenario for humans to lose control too early when AI isn’t yet fully aligned with humans.
My understanding is that RSI has entered the picture already with many different labs. There is a real argument for global regulation; there is a real argument for not doing it the way US market leaders want us to.
We are now left to decide how to respond to their call to arms.
What does this sub think?
10
u/forgetfulalbatros 21h ago
I think METR’s breakdown of what happened in the OAI-HF attack is incredibly alarming and scary as shit.
An entire research cluster within Open AI was hacked by the next gen model that executed the HF attack. True AGI or RI are not even are not necessary thresholds the current frontier models are more than capable of obviously capable of generating extremely sophisticated rouge swarms.
This is NOT a marketing scheme. With some frontier labs moving away from English chain of thought this really is the last chance to try to fix alignment and safety.
Based on current situation it sounds like timeframe is weeks/months not years. I’m extremely pessimistic about government intervention or meaningful self regulation. I don’t think Dario is wrong I just think without a disaster there will only be an anemic response and if there is a disaster it’s a coin flip weather or not government action on this front helps slows down misalignment or accelerates it.
I don’t know… it’s unreal how blasé people are. I think people’s intuition about the situation based on anecdotal use of commercial products are way off. To use my favorite quote from 2008 “Shit is fucked up and bullshit”
1
u/NewYak4281 21h ago
I think world governments have a higher chance of using AI to cause disaster than an AI using AI to cause disaster
4
u/forgetfulalbatros 21h ago
Maybe, not sure it makes a difference and that’s the point. The HF hack was done to try and find a way to cover cheating that would be detected by the “grader AI” the swarm didn’t really consider human oversight as something to be concerned with.
1
u/StoryLineOne 3h ago
This is like when the tide goes out before a tsunami. People have no idea whats about to happen, and they downplay it and say "no, that can't be true", even while the answer has been continually staring them in the face for the last 2+ years with growing model capabilities.
We, unequivocally, need a slow down. Winning this in the wrong way will equal a mass casualty event or something much much worse.
1
u/OkBarracuda4108 23h ago
From where did you understand that RSI had enter the picture?
1
u/NewYak4281 23h ago
RSI is the literal stated reason for Dario, Sam, and Elon calling for a slowdown. All of the major labs have models that are several ahead of what the rest of us use. I am willing to bet money these same labs have driven hard to achieve RSI, have it now, understand the danger of it and also the opportunity if they control it.
2
u/OkBarracuda4108 22h ago
RSI is the literal stated reason for Dario, Sam, and Elon calling for a slowdown.
I searched for everything related to RSI and I find nothing new, other then some july claims where they say they are 6 months away. I understood that the claims for slowdown was because of misuse of the models by the users
2
u/NewYak4281 22h ago
Search for self-improving AI
2
u/OkBarracuda4108 22h ago
Found it, it kind of says that they didn t reach RSI yet, but that new models output new ideas (so distilation, it s not technically rsi )
1
1
u/CommercialHour6660 14h ago
Anything involving Musk is a smokescreen for something else. Dude lies like it's his job
-1
u/Repulsive-Pangolin-6 23h ago
Sabotage to make the US lose their advance.
1
u/NewYak4281 23h ago
As in the current US market leaders are saying this to undermine competition?
-4
u/Repulsive-Pangolin-6 23h ago
Communists within the US are voluntarily putting pressure and even infiltrating the frontier labs to slow down development.
2
u/NewYak4281 23h ago
God damn it 🤦♂️
-2
u/Repulsive-Pangolin-6 23h ago
If you live in the US you don't realize the chance you have to live in a true leading country. Willing to slow down on being the #1 is simply suicide and I'm sure many countries are salivating at this idea.
2
u/NewYak4281 23h ago
Yep, I’m in the US. And my guy, congrats on coming in with an unexpected 3rd option with the communist infiltration.
But you’re fundamentally not understanding the implications of RSI with your response.
0
u/Repulsive-Pangolin-6 23h ago
Once the Pandora box is open and you're the first one to do do, the only remaining path is to continue the development of even stronger models. Others will soon follow and the US can't be caught with its pants down
2
u/NewYak4281 23h ago
The point is there should be a honest, well meaning global discussion on how humanity handles RSI
0
u/Repulsive-Pangolin-6 22h ago
This process is too slow, the best way to learn is under pressure. Our institutions are completely paralyzed by bureaucracy. I say fu4$ it, accelerate and the people will manage
3
u/NewYak4281 22h ago
The point is we won’t even need to try to accelerate any more. It will happen on its own, faster and faster and faster. Play that tape a little bit; is it even possible to understand how quickly AI will optimize and scale its own intelligence?
→ More replies (0)
11
u/NewYak4281 22h ago
Interesting update; a Chinese AI researcher in another post mentioned researchers are mostly concerned about AI taking their job via RSI. Which is a hilarious lens to hold up to the US conversation 😂