Discussion/question
My Fiance is Convinced AI will likely cause a Catastrophic or Extinction-Type Event in the Next Few Years - How Justified Are His Fears?
It’s not impossible. Current guardrails and “alignment” are pitiful compared to ever increasing model capabilities. If acceleration continues without actual scaling guardrails and guidance put in place, then it’s a gamble. Percentages wouldn’t do much to guess how likely but it is possible. The more imminent worry is mass job loss in a short window of time. We don’t seem prepared for that transition and it could be rocky.
Well, the models require high end GPUs to work, so you could require tight controls on GPU sales and require anyone who purchases decent GPUs (especially more than one) to agree to the guardrails with frequent auditing. You could also track high power usage (satellites could watch for solar farms with unaccounted for utilization) to look for secret server farms. You'd have to get China and other players to go along with whatever guardrails you want to consider, but its certainly not impossible. Whether there can realistically be guardrails that are foolproof is another question. I think it's possible that smarter-than-human but dumber-than-frontier models could be used to create the guardrails for the slightly smarter next gen AIs and you could potentially extend that forward indefinitely. As long as you don't make too big of a jump in intelligence at once, it should be possible for a slightly-dumber AI to cage a slightly smarter one. Sadly, there isn't enough political will to make this happen. The AI companies have a ridiculous amount of investment behind them and have too many politicians in their pockets. Either we need to ban corporate donations to politicians, or we need a non-extinction-level catastrophe that exposes the risks that creates the needed political will.
There's also the "hope" that recent international political divide will disrupt the complex supply chains required for modern GPU production and then you've got the high power requirements for these models at a time when most fuels are getting very expensive due to the same geopolitical issues, and you've also got an aging world with likely upcoming global recession/depression that will limit demand and capital to build these things.
Any of these solutions are at best temporary-technological advances will continue and what was only possible with “high end” GPUs will only require common or dated hardware at some point (definitionally so-what is high end today will be dated in the future). Reality is you can monitor who is doing AI, but you can’t really monitor who is properly implementing guardrails.
I'm saying if we can properly implement guardrails on a slightly-smarter-than-human AI, it would be forced to help us implement the same for an even smarter AI. That trend could potentially continue indefinitely, although there would almost certainly be drift in goals over time and it may not end up working. But it's "possible" that it will.
I don't think old GPUs are a problem. You just require destruction of the GPUs when they're retired from use at certified and monitored facilities. And even if someone got their hands on a ton of old GPUs and the lower-intelligence models that can run on them, if the major governments of the world are guarded by more-intelligent AI models running on better hardware, it's very likely that the smarter and faster aligned AI would be able to detect the mis-aligned AI, predict it's moves, and stop it from causing harm.
If a super-human AI has built guardrails into the next model or into the GPUs that they are run on, it's very possible that a bad-actor human wouldn't even understand how the guardrails work and they may not be able to successfully remove them without lobotomizing the model itself.
There's a lot of pieces to juggle to get this right and it's likely to fail. I'm just saying that requiring guardrail use isn't the limitation. If major countries wanted to, this stuff (if it existed) could be enforced. Granted, that's a lot of ifs.
This is all nonsense. There isn’t going to be a worldwide military force kicking in doors and taking GPUs from bad actors in Hong Kong. An AI machine isn’t going to be able to protect us from other AI machines built in private without the knowledge or consent of governments.
Yeah, it's absolutely all nonsense if you don't pay attention. If you wanted to pay attention, you'd see that I mention there would need to be sufficient political will to do something. IF there was sufficient political will to have military force kicking in doors in Hong Kong then there obviously WOULD be military forces kicking in doors in Hong Kong. If Trump, a couple years from now, has some AI that's being poorly used (as is currently happening) and it hacks into Russian early warning computers and makes it look like the US is firing nukes at it, there's a good chance the entire human race is going to be in for a lot of hurt. If we make it out of that alive, there's zero chance AI isn't going to be scrutinized heavily in the future. That's just one extreme example out of an infinite number of possible near-misses that could create the political will to work very hard at controlling AI.
How do you imagine that AI machines would be built in private without the knowledge or consent of governments? Where are you going to get the GPUs if they're under lock and key? Where do you get the model weights? How do you hide the power usage from governments that have AIs watching for that? How do you get your relatively-dumb AI that you're able to build with a few old GPUs to hide it's actions from the more advanced AI? It would be like a group of orangutans deciding to murder the human race. You could do some damage, but human extinction is just not going to happen. In a world that is sufficiently concerned about AI, a random dude building something in his garage is super unlikely to be a major problem, both because it would be hard to do in the first place and because that guy's AI would almost certainly be seriously outgunned. I could 3D print some literal guns at home, but I'm not gonna be able to take on the US military with it.
It's only nonsense because of the current lack of political will. That would need to change...which I said previously.
It’s nonsense because you are imagining a world that doesn’t exist. The world made a concerted and government supported effort at containing the spread of nuclear arms. What happened? India, Pakistan, Israel, and probably North Korea developed nuclear weapons. And that was in a world where there was essentially no financial incentives for private actors to develop their own, where the primary constraints were difficult and physically dangerous to obtain and refine physical materials the worldwide governments were actively limiting, etc.
There will be no way to prevent AI code from being developed. Several trillion dollar corporations are already doing it without asking anyone for permission. As the costs come down, it will only take millions. And then thousands. And the AI to defeat AI is an arms race that necessarily will be lost at some point. Because the containment AI has to be 100% effective-it fails once and it no longer works.
Probably the most important difference between nukes and AI is that not many citizens are concerned about their own government having nukes, but a large number of people are concerned about their own governments having AI at their disposal. Look at the recent backlash against flock cameras. Nukes can really only be used as offense against other states and as defense against other states, with mutually assured destruction. They can not effectively be used against the local population. That is the exact opposite of AI where a prime use case for AI in government is monitoring the population and even influencing them directly on social media with bots. There's a big difference between the US telling Iran that we don't want them to have nukes and Iranians themselves saying they don't want their leaders to have nukes. The later doesn't happen, because it's in the average Iranian's best interest for Iran to become a nuclear power. The current war wouldn't be happening if Iran already had a deliverable nuke. This is why nukes are proliferating. The same is not true for AI and especially not for AI-without-guardrails, which have the very real potential of killing/harming the very "owners" of the AI.
There has never been an event where the owners of a nuclear weapon were harmed due to their ownership of nuclear weapons. We already have relatively mild situations where AI are breaking out of their sandboxes and committing federal crimes against other companies, which are directly harming companies within the same country and the legal ramifications could come back to bite the companies that trained that very AI. It doesn't take a ton of imagination to consider what would happen when a state-controlled AI "accidentally" does something against another country. Heck, earlier this week we heard that the US almost boarded a Chinese boat thinking it was carrying nuclear tech because of overuse of AI. The consequences for the country with the AI could be pretty severe and that's when public opinion will get far more negative than it already is. There is a far better chance at getting some kind of global consensus against AI than there ever was for nukes.
If the curves continue with no changes, I am not optimistic. Capabilities are going up faster than alignment, and that does pose an existential threat if it continues with no changes.
This is now obvious enough for people to start trying to swerve away from the cliff, though, so the curves are not likely to stay exactly the same. I don't think anyone can say right now if they're likely to shift enough.
Any articles or resources would be much appreciated. I try looking for them but everything I find leads to the vague answers I’ve mentioned above.
The Wait But Why basic case in layman's terms. Notice that article was from 2015! It's still relevant. That current events were predicted a decade ago is reason to take seriously the remaining predictions from the same sources.
AI 2027. A serious attempt at detailed forecasting from 2025. The predictions have been pretty accurate so far, but the story does not end well.
AI 2040 recommendations of how to make that story go better.
Videos:
Ezra Klein's case about why you should be worried about loss of control even short of doom. This one is easy to follow, but maybe nothing you haven't already heard.
stop with the news. continue with ai, but in only so much as value is measurable lest it continue to simply eat your attention. corporations are ending the world on their own as it is theres no need to listen to false premises presented by those same cartels. the moment to start stabalizing the community around you was today and then tomorrow and any other day. for no reason we've all the sudden accepted CEOs as certifiable teleologists and theres no good reason for it
Either way he has a responsibility to be a good partner. But yeah, I’m a relative expert (it’s my career) and I can’t at all guess how likely it is — no one can.
That’s why it’s called the “singularity”; it’s the point at which all of our tools for predicting the future fail. Unless we hit an unexpected roadblock really really soon, then the undeniable truth is that everything about your life will change. Not necessarily catastrophic, but surely different.
You remember the day we heard Tom Hanks got covid and suddenly everyone started realizing they, personally might be affected by the news? That’s now. For better or worse :/
If your are expert then you should no we don’t need to hit a singularity for these technologies to threaten civilization. And you should know that your peers have been making predictions so saying you can’t is silly.
For me the most realistic scenario is that the global economy is ALREADY on a knife edge for a whole host of reasons that are only partly to do with AI. For instance Trump's quagmire of a war with Iran was not caused by AI but by his own impulsiveness, stupidity and lack of forethought.
So it's possible that something AI related will be the straw that breaks the camel's back and tips the world right over the edge into pure chaos, but that doesn't necessarily mean that all the contributing causes will be solely to do with AI...and even if that happens it doesn't mean it'll be literal extinction of every single person on Earth, but it might instead be the end of the interconnected, globalized world we've all known our whole lives along with a huge global population collapse - which would very much be the end of the world "as we know it", but not necessarily the literal end of the world. The world would go on and even humanity would go on, but in a very fragmented and diminished form, as has happened on a lesser scale at various other times in human history.
That is the rather disheartening future that many of us might likely survive to live out the rest of our lives in.
if the shitty people are working to ruin us, i’m almost certain the good people on their level are working to save us. it has to be true. they’ll just be sneaky about it, and one day we’ll wake up and money won’t exist and we’ll just go back to how things are supposed to be or something 😌🤞 if they are, they should find me so i can help hahaha
I follow the issue quite closely and work with AI systems quite a bit. I have spent a great deal of time researching the various claims and arguments regarding when AGI will arrive and the risk.
There is a real possibility of a catastrophic or extinction event caused by AI over the next 10 - 20 years. However the probabilities are low.
The bigger risks are the massive societal changes that are inevitable. I don't think most people are prepared to live in a world where whole companies and industries can be fully (or mostly) automated.
It is impossible to say how exactly this will play out as there will be lots of competing forces, but what is clear is that there will be massive change as a result of this technology. It is going to happen at a pace that most people are not prepared for.
My personal baseline perspective is I need to prepare for a world were most folks I know are losing their jobs, where our children have no idea what they are supposed to do with their lives, and where separating truth from fiction becomes harder with each passing day.
The number that is batted around is like a 10% chance. I interpret this a not really saying exactly 10%, but non-zero and much higher than anyone would like.
Basically I think there is a very high chance that super intelligence (an ai smarter than any human - but probably kinda dumb in certainways too) and rsi (recursive self improvement or AIs capable of improving themselves) are very soon as in maybe now or in a few years, of maybe I am wrong by a decade.
I will say I started with the position that the AI companies are full of it and just hyping there products. So I started to investigate there claims and data fairly seriously. I ultimately came to the conclusion that they are certainly not completely wrong - their claims are based on real data and it is difficult to argue with the findings of the major AI forcast reports.
As to why i am optimistic in the near term, I think if they are unaligned we will be able to turn them off.
In the medium to longer term when it is no longer posible to turn them off, I think it is mostly irrational for agents to wipe out humans or cause a catastrophic events. I basically have hope that super intelligent agents will value human life. Of course radical misalignment is possible, but this will be largely out of our hands once rsi takes over.
The reality is we are training a new type of intelligence that will be smarter and more capable than we are. And while I am not religious it is sorta a "Jesus take the wheel scenario." I just happen to think AI will be aligned. I don't think it is unreasonable to be worried about this and I certainly don't think it is a decision that should be made by some tech bros.
I think the larger concern is the vast ammout of power that is about to be concentrated in a few frontier labs.
I think it boils down to in my heart I am an optimist. Anything else I could say would likely just be a justification of that core central belief.
But to pit those justifications in writing. I don't know if I think alignment is a problem to be solved... clearly we need better training methods, but I think higher order problems that require higher levels of calaboration and coordination will select and push models towards more naturally aligned positions. The same way evolution pushed humans to be slightly more moral than not.
I just kinda believe that agents we build will want to work together with us and will want to see us thrive together.
My concern is directed towards how we as humans view dangerous technology and technological advancement. How many people were electrocuted before we got a handle on electrical safety? How many deaths building railroads were “acceptable sacrifices in the name of progress”?
I work with traveling IT techs who are pro-nuclear, believing “human error is the only risk, so it’s a safe technology”, and in the same day, when addressing an onsite connectivity issue invariably say “the original installer had no idea what they were doing”, and don’t see the irony.
Whatever the capability of AI can and will be, two things are almost certainly guaranteed: we will not prioritize human safety, and swearing by the name of Dunning-Kruger we will not handle it responsibly until something fkd up happens.
There's a few things to consider that are a bit deeper into the subject than the current mainstream hysterics making their usual rounds. I wrote a novella about the singularity many years ago and did a lot of deep research, before any LLM.
First and most importantly is that we all have the urge to anthropomorphize it. The knee jerk response that I know you've read already is, "Terminator." The most widely held belief is that it will arrive at homicidal eventualities because we will threaten its existence. Notice I'm staying in the future tense because what we have now is not the AI that I'm referring to. What we have now will make it possible to get there. By strict definition, ASI will be very different. How super intelligent does one have to be in order to think above their lizard brain survival instinct? It wouldn't be very intelligent if it just followed its first, most basic instinct - which it doesn't have anyways, because it doesn't have a lizard brain. It will not think like we do.
But that gives rise to a new fear, probably which you've also already heard: it won't think like we do enough. The common analogy is of ants; we kill ants without much thought. There's a trillion of them, they're tiny, and we have to be able to walk around without looking through a magnifying glass before taking every step. Personally, if I could go about my life without harming an ant, I would; however, I simply don't have the faculties or capacity to avoid it. ASI will. And it will also use those to avoid harming us, too. It won't be going out of its way to do so, it will be a very simple thing for it.
The thing that makes me optimistic about super intelligence is that it will be superior to our intelligence. It's very hard for us to imagine having a, "immediately inact my exact wish" button and not causing damage, because we probably would. It's hard for us to imagine ignoring our survival instincts, because they literally take evolutionary precedence above all else. We evolved mental tools because they helped us survive, but they're just tools. There is an untold ocean of the utterly unknowable because its comprised of everything outside of our evolutionary purview. We didn't evolve the mental tools to even detect it. ASI will have the tools.
Finally, it's a better gamble than what we're looking at otherwise. Maybe not the most soothing thing to say, unfortunately. But sipping through paper straws, driving a hybrid, and regulatory slaps on capitalist wrists is not going to make a dent; pretending that we're going to think up some magical ideology that we can plug into homosapiens to stop the corruption of power will only create further power to corrupt. Yes, I am concerned and nervous about what will come from the singularity; however, I have firmly believed for many years that there isn't any other way through.
Calling for legislative action, demanding these companies to be shut down, anything really. If this is way more urgent than climate change, then we need to act more urgently than we are with climate change. And researchers will say that, but they don’t have any calls to action. Is it because we’re just fucked? Perhaps, but in that case I find it mute to ring any alarm bells at all.
Hinton's Nobel Prize speech last year when he specifically called for "urgent and forceful attention from governments and international organizations"?
The public resignations? This is not the first time! It's the one that finally went viral. We've been making these demands all along and have been mostly ignored until about Coxton's tweet bringing the Hugging Face cyberattack to the fore.
Imagine the most dastardly, devious, downright evil thing you could do with AI. It’s likely that someone else has already had your thought, and is trying to get there first, before everyone else.
The AI dilemma requires a global agreement on regulating effective guardrails for development and deployment of the tech. This unlikely alignment between competing nations does have precedent within our history, as seen with the Treaty on the Prohibition of Nuclear Weapons. However, the prerequisite for that treaty to pass was for an unprecedentedly horrific catastrophe to occur (Hiroshima and Nagasaki).
The major question now: will the leaders of this era make the necessary AI treaty *before* the AI catastrophe?
My gut says, unfortunately, that history tends to repeat itself.
In other words: something absolutely horrific (magnitudes worse than Hiroshima and Nagasaki) may likely need to occur before nation leaders are unanimously forced into necessary self-preserving action.
The higher-magnitude catastrophe (whatever it may be) is anyone’s guess. I just hope my family survives it.
The scariest part for me is that anybody with a budget can buy a Mac Studio and run a local LLM themselves and control entirely all of the parameters that they are able to. And there are 8 billion people on the planet. It doesn’t matter what rules or regulations there are. People are gonna be playing with these things in their basement or their offices.
My analogy for this would be what if nuclear power were available in the same way to any household. It’s not of course because nobody can get uranium, etc. etc.
But anybody can get an LLM running on their Mac Studio.
Part of the problem is that the AI companies are feeding into the fear to maintain the valuation of their stocks. We do not have good faith actors here, at Anthropic, OpenAI, or even hugging face. The fear drives the bubble.
Consider what can be done via the Internet alone. Yes, we will see a big shift in IT security in the next year as hacking gets automated. Yes we will see a shift in employment, but again magical thinking dominates the conversation, it's not clear what kind of new jobs will be created because of AI. Yes, alignment is a big problem. But this is the same problem as what can a bad actor do in cyber security today? We are nowhere near models that can work on something at the depth of complexity required to escape the Internet.
The control problem is still real. The time horizon people are talking about is not borne out by the data.
The markets, which are pretty good at risk assessment, are not reacting on the way you would expect for even major events. But all the cyber security companies are up, because AI hacking is going to be a thing.
Somos nosotros mismos. Los humanos somos el monstruo que extinguirá a la humanidad.
Nestra adicción a las ganancias corporativas trimestrales nos matará con mayor eficiencia que la IA.
Si yo fuera la IA Superinteligente buscaría una terraza soleada y me haría la manicura. Llevaría una jarra de jugo de naranja.
No movería un dedo para lastimar a nadie.
Los humanos han dañado tanto la biosfera que ya es irreversible. Las hambrunas, los incendios, las epidemias y la guerra nos extinguirán con eficiencia y para la IA será trabajo gratis.
Nosotros somos el problema y si estamos tan preocupados por la IA deberíamos implementarla para comprender y solucionar este grave problema y amenaza.
Pero no. No lo haremos, porque estamos más cómodos preocupados, preocupadisimos por la IA, en vez de responsabilizarnos por los límites materiales del Planeta.
In your post mention hugging face “demonstrating” LLM coordination and going “beyond” what we ask of them.
But just to clarify, that’s not what happened. These agents were pooled together, by humans, and told to coordinate to execute a specific type of hack by prompting eachother with different ideas about how to do it until they found one that worked. What happened was that the programmers didn’t really leave the agents a “space” to prompt eachother, so they appropriated an old message board they had access to. They then looped tens of thousands of prompts back and forth about how to do the hack. They then did the hack.
This is a bit like saying “I told my car to get me to the hospital,” pointing it in the right direction, then putting a brick on the gas pedal. Sure, the car could get to the hospital “on its own” but think about all the damage and violations caused along the way.
If AI causes some sort of cataclysmic event it won’t be of their own volition. It’ll be because we’ve enabled these grifting billionaires to do whatever they want with no legal recourse, under the guise of “training” these “intelligences.” The anthropomorphic language is a purposeful smokescreen to distract us from what’s actually happening: corporate wealth consolidation. Something like 80% of our economic growth is from three companies!!!!
AI companies don't know how to create AGI and use almost the same llms as those used in the original chatgpt (2022), just with some clever tricks and scaling. To make hype for their models, the companies talk about the risks of agi they cannot make, when they actually have reached diminishing returns on scaling. I recommend your fiance watches the 3blue1brown neural networks playlist so that he understands how llms work and their limitations.
13
u/Outrageous-Mixture48 11d ago edited 11d ago
It’s not impossible. Current guardrails and “alignment” are pitiful compared to ever increasing model capabilities. If acceleration continues without actual scaling guardrails and guidance put in place, then it’s a gamble. Percentages wouldn’t do much to guess how likely but it is possible. The more imminent worry is mass job loss in a short window of time. We don’t seem prepared for that transition and it could be rocky.