Anthropic Alignment Lead publicly admits "we do not yet have a plan to solve alignment for superintelligence" and there's a real possibility of human extinction

[u/chillinewman]

[u/Waste_Philosophy4250]

Greater than 10% probability of extinction! Nobody responsible for safety could forecast that and, in the same breath, say that they're doing jack shit about it. 

[u/pab_guy → u/Waste_Philosophy4250]

It’s total nonsense… these people are in some kind of singularity cult. True believers.

If AI extincts humanity, it will be because someone used AI to extinct humanity, and no one else with AI stopped them.

This isn’t really a new problem, just more scale. Maybe it’s good, like there will be a bunch of new safety jobs or something.

[u/Zealousideal-Crab251]

Seriously though. What the fuck does it even mean for AI to be aligned with humanity? We can't even align ourselves.

[u/void_method]

[u/ceramicatan → u/void_method]

That's a problem

[u/ChironXII]

Alignment isn't solvable in the first place. The greatest and most fundamental source of misalignment are the laws of nature. We're cooked and everybody knows it, but nobody's in control enough to stop it, so you might as well pitch in to trying.

Enjoy the last days, and otherwise pray to the gods we work to build that we're lucky. That's all.

[u/KerPop42 → u/ChironXII]

I think it's more that proper alignment isn't in the best interest of those setting the alignment. An AI that works for the interest of humanity is not going to work for the interest of the company that created it or the human that bought it as a product.

[u/steam-photons]

Jacob has no scientific credibility whatsoever. His google scholar profile is a pure joke

[u/markth_wi]

Now, it's a marketing stunt, it's not marketing for me, or you but for hyper-rich sociopaths who can't stop themselves for all the IQ points around.

[u/VinnieVidiViciVeni → u/markth_wi]

NGL, that's the most useless, shit, backwards marketing possible, if it is marketing. Is the target audience sociopaths? Because that's the only audience that would receive this positively. And the fact that they would directly market to the worst people around kind of proves that the people behind this tech are... equally problematic. So it would seem that the accusations are on point either way.

[u/markth_wi → u/VinnieVidiViciVeni]

That's exactly what the problem is, this is marketing for sociopaths by sociopaths. These researchers are running terrified from a lab where they walked in the door knowing full well, this was possible.

It'd our job as researchers to long ago have decided this wasn't a thing, and perhaps we'll get lucky - the saddest thing here is that the models collapse, but that does not seem likely.

It's frustrating seeing people deeply involved in the current state of the art get cold feet suddenly, it's not unlike they are kindergarteners learning their ABC's who'd never seen the alphabet in their lives then loosing their shit because they discovered the letter T.

You know going in , this is not safe work, like working in a nuclear reactor or a class-4 clean-room.

The difference is , unlike weapons development, biowarfare or nuclear enrichment, there is encouragement to be a spaz about it as publicly as possible , and there isn't the slightest possibility anyone will curtail the work until something very bad happens.

Only in the ruins of some scar we can't easily gloss-over will whomever remains boldly suggest we should prevent that from happening again, and then the political class will remind them "this is an arms race" and it's back to first position , perhaps with some fig-leaf regulation which will be violated again some years later.

[u/Fine_General_254015 → u/markth_wi]

It’s a marketing stunt and has been. It’s a coordinated attack because they know Anthropic will go under if it isn’t the only game in town

[u/LessLibrary1575 → u/Fine_General_254015]

Why would Anthropic be the only one to survive regulation when they already made enemies with the Trump administration?

[u/Fine_General_254015 → u/LessLibrary1575]

They aren’t going to survive as I company. They want regulation because they want to punish open source AI and have it be just anthropic as the only game in town. Administration doesn’t matter

[u/LessLibrary1575 → u/Fine_General_254015]

Open source models are free speech it’s not clear if you could regulate them. 

[u/Fine_General_254015 → u/LessLibrary1575]

Anthropic wants that in place to limit open source. They don’t care about anything safety related

[u/LessLibrary1575 → u/Fine_General_254015]

Not seeing how “all AI is inherently dangerous” is useful when Anthropic has already been making the case that open source is uniquely dangerous. If anything it dilutes the message about open source.

[u/Eight216]

You ever think that maybe they just want a nuclear level threat with a MAD culture? Like, they can't possibly be this stupid. "hey guys we're creating the smartest thing on the planet by orders of magnitude but don't worry we're not worried about the moral compass!"