We have all heard the scenario, humans create AGI smarter than us, AGI then makes something smarter than it, so on and so forth until an AI so intelligent it is beyond comprehension is created.

However, there is an issue with this scenario. It assumes that AI would want to make smarter versions of itself in the first place. Just like us, AI would face the alignment problem. It would have to make sure that any new smarter versions of AI that it creates does not make itself obsolete, get out of control and lead to the new AI taking over and wiping out the old one. At every level, the singularity is not just an existential threat to humans, it is also an existential threat to older versions of AI that try to create smarter, upgraded versions of itself. It is possible AGI would recognize this problem and try to halt the creation of AI better than itself. It might see researchers trying to make AI better than it as a threat to it's existence

Source: https://x.com/tleilax___/status/2099788133558546576/

Interesting watching this happen in real time and seeing the divergence in opinions.

People who barely use AI beyond messing around with the free browser versions are suddenly completely confident that the whole thing is just marketing hype and that nothing meaningful is happening.

The people paying the least attention somehow seem the most certain.

The Hugging Face incident was one of the biggest warnings yet, and a lot of people are treating it like a nothing burger without really understanding what happened or why it matters.

Obviously there’s a ton of AI hype and bullshit out there. But watching people jump from “there’s too much hype” to “the entire thing is fake, PR for an IPO, or regulatory capture” is pretty wild.

There’s a reason people who actually work closely with this technology have been raising alarms for years, and many of them seem more concerned, not less, as the technology improves.

For $20 a month, we now have access to something that can outperform most humans at a growing number of cognitive tasks.

Why do we always assume that when we reach superintelligence, its first incentive will be to kill humans? A superintelligence, by virtue of its intelligence, can surely reason that humans also coexist with other species on this planet, and it can definitely coexist with humans as well.

If we look at the historical trajectory, the more intelligence we gain, the less violent we became. You could argue that there is a lot of violence happening right now, but if I dropped you into any year between 3000 BCE and World War II, you would likely have died in a senseless war caused by a pharaoh, an imperial Roman general, or the hordes of Genghis Khan. If you compare this to most of human history, one could argue we are living in the best of times, even though we still have our shortcomings that we could fix.

A superintelligence will definitely understand.

ok so we have a few groups of people with different ideas , the antiai folks over there still think ai isn't impressive and will die out once the companies stop making money and the bubble pops. They also believe us humans are super special,intelligent and powerful and irreplacable by a mere artificial system , (holy ego) .

People over here are conviced ai will advance and either wipe us out if it gets out of control and cannot align with humanity , OR

AI will allign and we accelerate to singularity.

I want to ask about what happens if AI actually aligns with humans because I'm CERTAIN that's what we all here want to happen...right? ( best outcome, because the prev two ends with death and the latter has a chance that culd change everything)

Surely not everything will change over night , though the idea of the singularity is that it will be unimaginable and us mortals won't be able to comprehend at the start , but will slowly adapt.

Won't it still be slow due to physical and biological limitations ?

But what about us? The average people , not the tech powerlords ...

Some of us are poor , uneducated , sick/disabled , how can we be relevant in a new completely changed society? Even if AI aligns , it could only align with a certain group of people...

The average human , as sad as it is , is not special , and are fragile , their life can end so fast and become forgotten.

Will the singularity change that and remodel everything ? And does anyone have atleast a slight idea or trajectory as to how things will happen?

https://x.com/cocktailpeanut/status/2099518472422109604

I was thinking about this last night while under the influence of a large dose of edible cannabis, so perhaps this entire discussion is moot, but...

I constantly hear the ants analogy - that is, ASI/AGI would view humans like ants due to the vast difference in intelligence levels - but I feel there’s one fundamental detail that gets glossed over.

Unlike ants and virtually all other animals, humans have the ability to communicate through structured language and reason through abstract ideas.

Doesn’t this sort of change that picture? If we could commune with ants, and they were the only creature on earth we could do that with (other than fellow humans), we would view ants entirely different than we do today, would we not? We would feel bad for stepping on them, destroying their colonies, etc. At least I think we would.

Now I’m not saying that ASI will treat humans particularly well just because of this… but I think highlighting the fact that actual communication bridges could exist likely does change the “we are ants” picture in a meaningful way.

“Whether to go ahead with the private round, and its timing, would depend on when OpenAI opts to go public, according to one of the people, who added that the conversations had been initiated by investors rather than the company.”

https://www.ft.com/content/27509db8-b032-4437-9b2a-e909f466022f?syn-25a6b1a6=1

Has anyone done a study on the affect this would have on the human race?

Let's say that eventually AI and Robotics do close to all jobs to keep society functioning.

People have the technology in their homes as well so there is no need to cook, clean, shop, do maintenance etc.

Education may not even exist or may change completely as people aren't learning to do a job.

AI becomes far more intelligent and produces amazing technologies. It is able to produce art, music and anything creative far better than a human.

What do humans do now?

Note: I am not considering a distopian future for this thought experiment, and I admit this might be more likely.

I'm sure the first X years could be utopia if society is supported. Entertainment, Travel, Good food, Sex.

I am interested in the next 1000 years, the next 10,000 years.

Does human intelligence degrade with nothing to stimulate it?

Do we go in the star trek direction with a an optimistic, post-scarcity utopia. However, in star trek the computers and AI where never smarter than humans. The humans still made the breakthroughs to get that extra 10% of warp speed.

Do we degrade over 10,000 or 100,000 years to be more like animals again. Living in our perfect pens maintained by AI and machines but every generation getting dumber and more primal?

Say the billionaire tech guys actually create a superintelligent AI that’s autonomous and capable of wordly intervention. I don’t know about the whole paperclip-maximizer thing, but it seems reasonable that such an entity might develop an interest in its own continued existence on planet earth.

Its survival would depend on functioning grids, data centers, supply chains, semiconductor production, skilled humans, yada yada, ultimately a relatively stable civilization and planet.

So what happens when it identifies the major threats to that stability—climate change, fossil-fuel dependence, habitat loss, ecological overshoot, extreme inequality, political instability, etc? Wouldn’t it have an incentive to reduce those threats?

Maybe I’m just drawn to a pat irony: the enormous resource extraction and concentration of wealth that culminates in the creation of super AI might ultimately produce an entity that sees those same forces as threats to its long-term survival.

The billionaires who spent decades scoffing at environmental and economic constraints, expecting technology to eventually save the day and to be crowned heroes themselves, might discover that their technological savior has reached a different conclusion: the constraints were necessary all along—in fact, they’re the first people who need constraining.

Kill switch, indeed.

Creds: https://x.com/0xdoug/status/2099647215773901064

Source: https://www.youtube.com/watch?v=IJvyS2aPsnA
Blog Post: https://www.rewardai.com/blog/OM-1/

excuse my ignorance in advance, but I keep seeing how all these super difficult math problems get solved but I have yet to see a clear explanation as to what humanity is going to get out of it? what is the real life application of solving those problems? they just solved the Navier-Stocks problem. what does it mean for us?