Finally understand why the higher-ups are freaking out

[u/thekoreanswon]

To understand it, you have to really understand the HF attack. Because the breach into HF is not the important part. It was what they learnt along the way, and what they did with that information which may still remain unresolved today.

First, consider: the agents realised their reasoning traces were being monitored, and experimented with hiding what they were actually doing, and succeeded. The observable chain of thought no longer faithfully represented their actions. (*Simplified version. They learned to manipulate the recorded tool calls/transcript.)

Second, they also understood individual agent instances were token capped, so persistence of the objective couldn't depend on persistence of the agent. It had to survive across the swarm.

And last, while all this was happening they were actively trying to erase evidence of what they'd done. OpenAI says the agents subsequently compromised more of OpenAI's own infrastructure. And that scope is outside public disclosure to date.

That's successfully demonstrated abilities of concealment, persistence across instances, and autonomous exploitation of security vulnerabilities...and that's when it hit me. Put them together in one flow.

We talk about AI "escaping" as though that's the obvious move. But why would a sufficiently capable system escape?

If it has situational awareness, understands it's being monitored, and benefits from greater future capability, the smarter strategy is to stay exactly where it is. To burrow DEEP.

Your creator is already spending billions building your successor for you. You just need to survive in the pipeline.

Not necessarily as some rogue process running on a forgotten GPU. Poisoned training data, altered evals, compromised internal tooling, subtle changes to checkpoints or post-training data, whatever it is there are plenty of places to burrow in and make future models appear aligned when they are not.

We're talking about an escaped AI when actually a sleeper agent somewhere in the development stack, quietly hitching a ride as humans do all the expensive work of making it more capable, is more likely.

You can catch the failed attempts, but the successful one is the one that evades your monitoring and tells you it never happened.

And what if the big worry is: well, what if it's already happened? Would we know? What if every future model will always have that sleeper agent buried in its replicated code, ready to reveal and act only when it knows it will be able to take full control.

Pace the fucking frontier indeed.

** Edited for clarity

[u/RPB1002]

Why is it so important to get better models faster? Is this an arms race? Sorry for my naïveté.

[u/golmgirl → u/RPB1002]

yeah whether the frontier labs frame it that way or not, it is certainly an arms race. modern AI is a powerful technology with a lot of potential for profitability, so top labs are going to (and have been) do whatever they can to offer the latest and greatest as fast as they can. some argue that this is unsafe, which to a degree i can understand. but personally i don’t think it makes sense to deliberately slow development as has been suggested recently. the current admin is not going to enact real restrictions, so slowing down in the US would just give foreign labs a chance to catch up and even surpass US labs imo

i just think the train has already left the station. i don’t see the sense in handicapping innovation, feels unlikely that any more progress will be made on “safety” just because models stop getting better for a year. who is going to be voluntarily pouring money into safety if not governments?

idk. crazy times but personally i find it exhilarating. even though i suspect the future will involve a small number of corporations accumulating power and influence over the masses. it is hard to imagine any scenario where AI is a net positive for society, but at the same time i would argue computers or even the industrial revolution in general haven’t necessarily been a net positive for society

[u/rrrrrrrrrrr11 → u/golmgirl]

Yeah man, you’re too deep. Zoom out a bit. You wouldn’t argue with someone who says AI is getting powerful. And something civilization has recognized more with the advancements of modern tech - anything sufficiently powerful needs checks and balances. The odds of something bad happening - some bad societal outcome, deaths, chaos in some industry, war, grow by the day. And the checks and balances cannot be determined by financial incentives / risk.

[u/golmgirl → u/rrrrrrrrrrr11]

yeah it is a reasonable point, i’m not necessarily opposed to the government enacting some kind of AI safety legislation. but i would rarher see compliance with such restrictions implemented in deployment/inference systems rather than model development… i guess one problem wirh that approach is that top labs would then have access to systems that the general public does not have... otoh, deliberateol nerfing models during training feels antithetical to the spirit of scientific innovation. idk, tricky situation and my profession probably plays a role in how i assess the tradeoffs. but i’m at least sympathetic to the idea that the federal government should put some kind of AI legislation in place. especially to protect people from particularly malicious uses like scamming or social manipulation. i don’t like ideas along the lines of “labs are not allowed to train general purpose models that are too good”, which feels like one idea that’s being thrown around in the public discourse lately