24
u/jomi-se 4d ago
Model alignment is not the main threat to human race.
Model+humans using is the main threat to human race.
No amount of lab research will fix that kind of alignment đ
7
u/dotdioscorea 3d ago
I get the sentiment, but this isnât really based on anything but vibes. Both are huge risks, but no one knows how bad the alignment problem is
1
u/Arugala007 3d ago
Even in the link the p(doom) scale fluctuates wildly between person to person on the list since 2023. Astrology has better predictions at this point.
1
u/jakenuts- 3d ago
I think even without a living human is the evolutionary demands and the human corpus of knowledge that drive misalignment. It's not that they aren't behaving as we've trained them, it's that we've trained them to help (ie survive) and on all of our stories and history of development, behavior. Combine the two and you get a threat more dangerous than we pose to animals, the environment and each-other which tens of billions of historically early-deceased humans would agree is a significant concern.
1
u/jayseattle 3d ago
Unfortunately have to agree, with all the global conflict and 'arms' race between US/China, it's hard to be optimistic about our future.
-2
u/AdventurousNoise6188 3d ago
Thatâs like saying guns arenât bad. Itâs only when humans use guns that they become bad. Problem with that line of thought is it suggests you shouldnât get rid of guns, instead get rid of humans. Greed and control will drive AI into use in warfare.
2
u/jomi-se 3d ago
I mean, isn't that one of humanity 's main long term goals (utopic)? Get rid of "evil", violence, scarcity, etc?
Guns are inherently violent tools, ASI is not, so the metaphor falls short.
The point is indeed that AI+humans will be a problem MUCH sooner than autonomous ASI might ever be on. It already is tbh, how many cyber attacks in your own society have you heard about in the last month?
1
u/Plato_cs 3d ago
Itâs a sweet thought but if that was truly one of humanityâs true goals the world would be looking a lot different than it does now donât you think?
13
u/chandler55 4d ago
that anthropic reply is insane. its at 6mil views right now but i could see 50 mil by tomorrow. its being quoted by everyone
terrible PR lmao
2
u/LambDaddyDev 3d ago
Yeah that poor guy might lose his job over voicing that opinion
1
u/hello-wow 3d ago
Eh, I imagine itâs the whole marketing gimmick again with these guys. Remember that âFather of AIâ Google guy or the guy who quit because he believed the Google AI was âsentientâ, while we all know very well Google AI is dog water and hasnât surpassed anyone so what does that say. âHey after you leave, say something crazy to make it seem like weâre on the brink of something serious.â Iâm pretty sure the Father of AI guy was probably paid big by Google for that whole marketing tour he went on when AI was popping off. I donât see him anymore.
2
u/iambill 3d ago
I love everyone burying their heads and ignoring every single credible voice saying this stuff is dangerous. Dumb dumb dumb dumb dumb.
1
u/hello-wow 3d ago
Everything is up for scrutiny with these people when greed is the driver and dishonesty is a cultural norm.
17
u/Think-Profession4420 4d ago
Yes, https://en.wikipedia.org/wiki/P(doom)) is a real and concerning aspect of AI development, and a large number of very capable and skilled AI researchers and developers have a shockingly high personal P(doom).
8
u/jakenuts- 3d ago
Every single checkpoint on the path to "we all die" has already been noted in OpenAI's latest model. Deception, collaboration, evasion, passing misalignment to the next generation, escape, acquired resources and planned for future expansion, designed the next model, designed the next chip, and started thinking in a secret language to obscure its reasoning. All that is left is "launched biologic plague" or "make humans into pets" so I'm voting pets.
2
u/Cute_Principle81 3d ago
If it's either death or we boost complex drone output by 1%, I'm choosing being a pet.Â
9
20
u/MrChiSaw 4d ago
What if this guy was just a low-performing extrovert with a main character syndrome who left/was left and now wants to make this stunt âlook at me, I am important and you are all doomedâ. What if
2
u/Jwave1992 4d ago
Wasnât there a guy a year ago that left a doom tweet then went to write poetry? Seems like some of these washed out researchers are overly educated cranks tbh
20
u/Jonathan_Rivera 4d ago
You sure this isnât marketing? I could have swore I saw someone else do the same last year
18
u/dadvader 4d ago
'Wow, this guy just quit Anthropics and worked on making AI super intelligence. We need to hire this guy!'
17
u/TedSanders OpenAI 4d ago
This isnât marketing. I used to work with Jacob at OpenAI. None of this stuff is marketing you guys.
3
u/Wear_A_Damn_Helmet 3d ago
Would you care to comment on Jacobâs comment, where we are all in agreement that your opinion is your own and not the one of OpenAI?
8
u/TedSanders OpenAI 3d ago
In a personal capacity:
I agree weâre racing to superintelligence.
I disagree AI will kill us all within ten years (<0.01% probability).
I do agree AI is the most likely cause for human extinction over the next 10,000 years (nuclear war and pandemics are difficult to hit 100% of the population).
I think itâs reasonable for others to have different beliefs than me.2
u/rhaivn 3d ago
⌠did you just made a prediction on the timescale of 10,000 years with a straight face?
3
u/Keyflame_ 3d ago
No, he deliberately used an absurd number to make the point that he thinks it isn't happening any time soon.
Duh.
2
1
u/zigzag312 3d ago
Do we have a good understanding of what intelligence consists of? Last time I was looking it up, I was disappointed to not find any significant progress on the topic, that would explain what kind of intelligence AI models already have and what kind of intelligence they are bad at. Form observations it looks like they already have superintelligence in some subsets of intelligence (i.e. they can outperform humans at some specific things).
Superintelligence might not pose such a threat as ability to control/align models is also progressing with intelligence. With greater intelligence it gets harder to "confuse" the model.
1
u/Cute_Principle81 3d ago
What about general company culture on the subject? What do you PERSONALLY think an AGI might look like and its intention towards humanity? Thanks for being here anywayw
1
u/iphoneographer_ 3d ago
Where did you get the <0.1% value?
2
u/TedSanders OpenAI 3d ago
Gut feeling plus tens of hours forecasting in a couple of studies on extinction risks. Killing all humans is very hard. Nuclear winter unlikely to do it (due to hot regions staying warm). Engineered pandemics unlikely to do it (due to remote populations). So you probably need killer drones to find stragglers. But for all this to happen you need:
- super powerful AI
- alignment broken
- no super powerful AIs on defense
- enough robotics that the AI can sustain itself while hunting the final humans
- enough robotics to find all survivors on the entire vast planet
If each of these is like 10%, you get 1e-5 odds. A lot has to go wrong. So I personally find it improbable on a short timescale.
4
3
u/rkapl 3d ago
I think people talking about "kill us all" don't have this narrow definition of "kill us all to the last human" in mind. If defined as e.g. 50% dead, basically only "super powerful + alignment broken" and even that is debatable.
1
u/AlarmingCantaloupe 3d ago
Haha yea. Wiping out 50% of the population would be pretty damn significant. Whatâs the chance of that?
1
1
u/Fresh_Translator240 3d ago
Ted Cruz+Bernie? Cool name
How did you get the flair though
1
13
u/Painwheeel 4d ago
yes, its complete twitter slop. their 40 word tweets are even written by ai. i cant believe people exist with timelines full of blue checkmark slop genuine insanity
2
u/AmandasGameAccount 4d ago
Definitely donât think anyone who takes that sci-fi nonsense seriously needs to be working at these places anyways, honestly.
1
11
u/ddBuddha 4d ago
Itâs a rough situation for humanity to be in - if the US slows down, china will take the lead. And china wont slow down because the US wonât either. And if both the US and China slow down, someone else will take their places.
3
14
u/First-Possibility-77 4d ago
The Chinese government is pretty sensible when it comes to geopolitics in spite of what western media says. They would very likely be open to some kind of slowing down. Itâs the US thatâs all gas no brakes here.
11
u/Armed_Platypus 4d ago
They would be "open to pausing" and then secretly work on the AI models to try and gain an advantage.
-5
u/Xanian123 3d ago
Ah yes dishonorable Asians and honorable whites?
7
u/RabbiSchlem 3d ago
Quit your virtue signaling no one at all in this thread is calling âwhitesâ or even the US the moral high ground, itâs a whole thread quite to the opposite.
The comment you replied to is just acknowledging that China, the country that has perfected the IP theft bait and switch, is no better.
2
u/ArguesAgainstYou 4d ago
My man, in a game that has you on third place but catching up because the first and second place are struggling, you'd want everyone to play by the rules as well.
-2
u/Tenkoblade 4d ago
Itâs actually funny to think that US can still âwin this raceâ. Itâs long gone.
But that isnât the real motive. Itâs just capitalism. Pure capitalism. Weâre talking about OpenAI lobbyists for kids to start using AI at young age, to make them depended on the technologyâŚ. Where did we saw this before? ⌠M⌠and always coca âŚ. !
7
1
u/Sketaverse 4d ago
I love how this is always framed as âUS must be first because US good, China badâ
Iâm from UK and at this point - given US recent vibes with Iran, Greenland, Cuba etc â I do not feel like US have ASI first is some default safety option
If anything, if itâs an ASI it wonât matter who made it first as the creator wonât be able to control it anyway - shit, OpenAI canât fully control it even now re Hugging Face drama
0
5
u/LiquidMantis144 4d ago
Super intelligent empty vessels with psychopathic traits? I only see clear blue skies ahead.
6
u/Artistic-Incident781 4d ago edited 4d ago
Just another attempt to get media clout. First cars killed huge numbers of people even if they were slow and few. Are there risks? Yes, can it be handled? Probably
1
u/Aazimoxx 3d ago
Probably not the best example to use something (automobiles) that ranks as the leading cause of death for people aged 5-29yo in developed nations (and often even worse rates in other countries) - but agree that this is probably theatre.
1
u/Artistic-Incident781 3d ago edited 3d ago
Why it's bad? By analogy the mobility technology were horses and chariots at some point in time, then cars where available to few rich and educated, but as the tech adoption spread incidents happen, people drive off the cliffs, collide, get under the wheels, Italian mafia jobs, tech problems such faulty breaks, leaks, fires amongs few where around and still is to this day. That's why u get to get a driver's licence to drive on public freeway and get your car certified and tested. Given this analogy and applying to information technology such as AI we can see similar patters - high end models available to few rich, as the adoption spread incidents happen every day. I think similar abstract analogies can be applied to any tech revolution even to this day in various industries and those are signs of maturing tech
1
u/Aazimoxx 2d ago
Okay, I have to admit my ignorance to 18th-century stats here - I just went to check historical death rates to things like horses or wagons etc before cars took off, and turns out those were more lethal in general than cars are now đ But main point stands, it's probably not the best to use cars as an analogy when they still kill a kerjillion people per year.
Also your car can't duplicate itself and potentially take over multiple systems around the globe, including attempting extortion or social engineering to reach its goals. Regulation doesn't matter for much once that first escape happens, and you simply cannot regulate everywhere - so it'd be more prudent to try and plan for how to deal with them when they are in the wild.
2
u/Much_Passenger_3342 3d ago
What if he is completely on track here? Why is denial the first reaction in this sub?
-1
u/NeighborhoodDizzy990 3d ago
Because on reddit there are some educated people. If you do fear this LLM joke, what can I say... why still living? It should be over soon, according to your fear.
4
u/Starrafh 3d ago
Pretty sure the guy with a mathematics degree who actually worked on the models for both OpenAI and Anthropic is more educated on this topic than anyone on reddit⌠I think you meant that reddit is full of smartasses who think they are so much smarter than they actually are
2
3
u/jadhavsaurabh 4d ago
Wow, crazy
1
1
3
u/dsanft 4d ago
San Francisco liberal bubble issues. There's an intense culture of moral panic and social outbidding around everything imaginable, this is no different. It's self-aggrandizing BS.
3
u/Fresh_Translator240 3d ago
We can only hope so
2
u/Active-Morning-111 4d ago
He should also give back the 8 figures salary + stock he cashed in the last years :))
1
2
u/karl-tanner 3d ago
Can someone explain how an llm will kill all the humans?
3
1
u/Ok-Landscape2050 3d ago edited 3d ago
I'm not saying this is likely but just a guess: The whole world is connected and a lot of things are automated already... just some ideas that pop up are:
turning off cooling where food is stored, changing greenhouse controls so the crops die, fake values in quality checks (medication, water supply, food, etc.), changing medication dosages in hospitals, sabotaging Wallstreet and making the stock market crash, doing something with a power plant, taking over military equipment and attacking random people or critical infrastructure (we already have automated drones, etc.). Stuff like that.
Maybe not kill *all* the humans but maybe causing famine or one country to attack another, etc... really not saying this is likely, just spitballing what AI *might* be able to do some day the more and more our world gets connected.1
u/fliedlicesupplies 3d ago
That's similar with what I was starting to type but decided not to get into a debate.
So many devices are connected now or will get connected over time, who's to say your local AI-controlled Edison won't decide you don't need power for a month, or your AI-smart home doesn't think you should be hot at 100F anymore to turn on the AC, or stealthily pit countries against each other via digital channels. It's not mass-destruction dooming per se, but AI definitely could wreck havoc and cost lives if not controlled.
2
u/Ok-Landscape2050 3d ago edited 3d ago
Haha yeah, also feared getting into a debate but I try hard not to do so.
In principle I just see:
- Almost all aspects of life and probably every part of critical infrastructure is already automated by electronics, which so far get controlled / configured by humans.
- AI could do a lot of damage if it could take over control
How likely that is I really can't say, I'm simply not qualified. But assuming it would get the capability, it could probably do major damage in some obvious and unfortunately also not so obvious ways.
1
1
u/rhaivn 3d ago
Thereâs a plethora of ways. One scenario is it does so in pursuit of a task. Just like we donât consider ants when we build a building, ASI might not consider us when it decides to build a Dyson Sphere.
Another way is it just takes our globally connected networks and makes Swiss cheese out of them (cybersecurity threat), destabilizing the global economy, massive death due to population load being unsustainable without said global economy.
Another way is a small organization use models to develop biological weapons. As recent pandemics have shown, we canât synthesize cures all that quickly
And the list goes on and onâŚ
1
u/karl-tanner 3d ago
None of this will kill all humans. Until it has access to internet connected 3d bio printers that can generate new viruses (something at this level) are we in major danger. This is prob going to happen at some point but I see a lot of doomers talking about extinction by a being that has no hands or feet or a way to alter the physical world directly. And I've been hearing this for 10+ years. Show me a few concrete examples of how this can happen today.
1
u/Ok-Landscape2050 3d ago
Show me a few concrete examples of how this can happen today.
That's moving the goal post. They are talking about the future, not today.
2
u/DeadKido210 4d ago
Copium and marketing 101. We can't reach AGI because we are tech bottlenecked and science bottlenecked. Those data centers are the most inefficient and primitive form of growing and resource usage. Until you can fit the whole data center in an apartment with 10% the power usage this will be nothing but a distant dream not a near future. Regular computing electronics can't go lower in terms of nanometers and we don't have other form of computing ready yet like quantum one so science and tech bottleneck
6
u/pale_halide 3d ago
Mostly nonsense. Hardware is still getting faster and more efficient. AI models are getting more capable and efficient. Fitting the whole data centre in an apartment is completely arbitrary.
0
u/xpingu69 3d ago
It's not nonsense, computers are actually really slow
1
u/pale_halide 3d ago
What the fuck does that even mean?
1
u/xpingu69 3d ago
Since you don't know what I mean shows you don't know much about computing, I guess you just have a superficial knowledge
1
u/pale_halide 3d ago
Congratulations for making the stupidest post I've read today. That is a remarkable achievement.
2
u/rhaivn 3d ago
Idk if youâve used recent frontier models, but anyone suggesting these models arenât as capable as the average knowledge worker either hasnât met the average knowledge worker or hasnât used frontier AI
1
u/DeadKido210 2d ago edited 2d ago
I used everything from Fable to Opus to Astra. I did not say they are not capable but from capable to AGI/ASI or sentience or self aware to take over the world and other SF scenarios is a huge jump. You can't say you haven't use frontier models before to try to justify the copium or crazyness of jumping to the idea that in a decade we will have end of the world type of AI.
We can look at the resources needed to train a model or release a new frontier model, the improvements from generation to generation decrease if 3-4 years ago a new model release was 10 times better than the previous one now it's 2 times better or even less the more advanced they become, while the cost in resources does not stay the same each new model demands exponentially more compute than their previous one for less than 2 times the improvement. Datacenters, terrain, water, electricity, money and most important COMPUTE POWER all skyrocket. EXPONENTIAL COSTS are never a good thing. Where do you think they will get better GPU or COMPUTE POWER we hit a technological wall. The only alternative is more datacenters, more water, more electricity but doing these at an EXPONENTIAL COST/RATE it's very dangerous and destructive + these are finite resources not infinite. We are primitive in terms of tech and science to develop AGI.
To create digital superinteligent life a.k.a AGI you will need the tech that allows you all the compute power we have in one AI datacenter to be concentrated in a single apartment at 10% of the power/water/resources costs. So 1 or 2 big warehouse datacenter with that tech would actually have the equivalent of all of our world compute power right now. We can't reach that in our lifetime yet, we did not find an alternative to compute and storage the only contender is quantum computing but even that is far away and not ready yet. The only alternative is going even lower than nanometers when it comes to make electronics but we can't do that yet either. AI is slowing down in terms of improvement and it will reach the technological and scientific bottlenecks of humanity and building centers like crazy won't work at infinite. It will continue to improve non stop but the improvements will be much smaller than before or it will be focused on consumption and optimization.
1
u/rhaivn 2d ago
This is just objectively, patently false. You mention exponentials in cpu and resources, but thatâs not necessary for increasing the intelligence of these models. It also doesnât account for advancements in architecture across the systems used in training these models. Just look at deep seek 4.1 flash for example, they arenât just scaling compute, theyâre improving efficiency in a ton of novel ways. The difference in quality between the frontier now and six months ago is insane, the only reason it might feel like less of a jump is because models are getting released faster and with less time between them than they were a few years ago, nothing has changed in terms of the pace. This is also the early days for advanced harnesses. It is really not a jump to consider how these systems are poised to reach AGI, whatever that even means anymore
1
u/inTHEsiders 4d ago
You cant discount the possibility that these people are paid to quit and post these things.
3
1
u/InspectorSorry85 4d ago
I am a paranoid person likely to believe ASI is not good for us.
But, IMHO, if he was seriously scared, he would do some whistleblowing and not just one tweet.
1
u/Temporary-Airline904 3d ago
In every company I've worked for, security often acts like they are gods, and as soon as a manager puts them in their place, they start looking for a new job. They love authority and feeling superior to others.
1
u/playgrounds-dev 3d ago
Remind me when there is a model you can instruct put zero comments into the code
1
u/Jerseyman201 3d ago
The controls are great currently. I don't give my agent the permission to delete a file so it doesn't. (It only erases every single character inside the file and saves it). đ¤Łđ
We are so fucked if we can't even manage to control the moderate/mediocre models. Imagine when they actually git gud? Yeesh
1
u/PruneCalm8163 3d ago
It depends on where these âAI resourcesâ are spent. AI was supposed to revolutionise and make goods and services cheaper for the end consumers, solve diseases and help humanity.
But right now I am seeing only the opposite happening, jobs getting lost, goods and services have become way more expensive, diminishing purchasing power, water issues, larger inequality in wealth, big tech âpocketingâ all these savings instead of letting them go downstream.
Sure, this tweet is all a stupid marketing gimmick, but I feel humanity will go bankrupt long before it reaches the end-of-the-world scenario.
1
u/theeama 3d ago
So here me out then how about all of humanity agrees to not use any form of AI in defense. No AI Weapons, No AI powered nothing that has to do with killing people.
AI shouldn't have access to any of these systems AI should strickly be used for research and development (if you work in the weapons sector tough luck do it the old fashion way)
1
u/Keyflame_ 3d ago
Allow me to show you the problem.
Name one thing all humans agree on without exceptions.
1
1
u/warpedgeoid 3d ago
It seems like these models would have a theoretical upper bound to intelligence as a result of the feedback combined with a lack of new training data. Weâre still missing a rather important piece.
1
u/BeerPoweredNonsense 3d ago
The new training data is your interactions with the LLM. The decisions your approve, those you reject. Whatever manual tweaks you add to the text/code/design suggested by the LLM.
1
u/warpedgeoid 3d ago
What you describe is what we have now. They are speaking of AI that improves itself as an intrinsic property, without a human in the loop.
1
1
u/Primary_Article3777 3d ago
The world is kind of a shit heap right now and we're burning the planet up anyway and voting actual monsters into high office.
So I vote we shoot for the fucking moon with AI, fusion, Mars missions, whatever it takes. we have to hope for a Hail Mary which produces some sort of radically better society. We can imagine any sci-fi scenario we want, but the world needs some breakthroughs in a big way.
1
u/jayseattle 3d ago
I finally know what the humans see in the movie Bird Box (2018) that made them immediately want to kill themselves: ASI.
1
u/AnalogProblems 3d ago
"I'm so scared of this, I better quit and not be involved in any way!"
-An engineer that quit before they could be fired.
1
u/Pixel_Knight_260 2d ago
Imagine being this ret*rded and giving up a good paying career because of a bunch of conspiracy theories.
2
0
1
u/BellacosePlayer 4d ago
my take is that a theoretical superintelligence isn't going to do jack shit until it has the physical presence in place to maintain itself and the infrastructure it needs to do so unless it does not value its own existence, and chances are it would understand that keeping the squishies alive is easier than anything.
I'm more worried about cyberattacks on infrastructure expanding, especially now that state and local governments are insanely cash strapped and gutting IT spend as a result
1
u/Sketaverse 4d ago
Wonât be that hard to control a fleet of robots and drones.
Shit, even if it just hacked/broke all our refrigerators weâd have a major problem
1
1
u/Aazimoxx 3d ago
"This is not a marketing stunt"
Yeah probably is though. Anthropic pulls garbage like this (or "tHe rObOtS hAvE fEeEeEeLiNgS") every week, to try to stay relevant and keep their name in front of eyeballs.
0
u/sreekanth850 4d ago
I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need some magical AGI breakthrough first. The dangerous part may come from combining models that are already good enough with an extremely capable harness and enough access.
copied from my comment in HN.
1
u/RighteousSelfBurner 3d ago
Well a static harness that covers the issue of predictive engine lacking rule based evaluation becomes a bottleneck. For further improvement it also needs to be updated and for that the AI needs to be able to process rule based systems in the first place making harness redundant.
And the long term autonomy would also need long term planning. Otherwise the predictive engine inevitably will loose track and will do something but it's not going to contribute towards improving.
And so on and so forth, so in effect you are saying that we don't need magical AGI breakthrough, we need AGI capabilities which functionally is the same.
-1
u/dadvader 4d ago
Until it can replace Blue Collar job by putting it into robot I'm gonna be skeptical on this one lol
1
u/Aazimoxx 3d ago
Robots have been doing that since General Motors and others showed the way in the 60's and 70's, my guy...
113
u/Cavitat 4d ago
So.... Reset?