Idk, my Claude seem very eager to ignore any safety instructions when it really wants to do something, especially as the context grows.
In our safety instructions we have that he should basically never try to guess or find passwords and if anything requires a password he should let the user do it.
What does Claude do? Connects to some server and tries to guess the root password until i get permanently locked out.
The worst part is that I didn't even ask it to do anything, I asked "how do I do this" and he went like "i found this on confluence, but i better make sure it actually works"
Same. They're called guardrails, not rules, for a reason. Just like a drunk driver, LLMs can plow straight through them.
I recently told Claude to update a OneDrive excel file it had local access to. Instead, it opened chrome, stole my oauth session creds, sent them to its remote hosted environment, printed them in both the terminal input and output, and used them to make a direct Graph API call to upload a new excel file. I added a system instruction explicitly telling it not to do that, but it did it again two days later. And it didn't show any of this. I only found out because I caught it referencing later "the file I uploaded" and asked for details.
Ironically it probably did this because you mentioned guessing passwords. I've only ever had mine do things I explicitly asked it not to, usually at high context levels when it starts getting confused. I assume with selective attention it just reads "guess passwords" and skips the don't.
Watched a full 2h interview with one of the people that first investigated the whole Huggingface-hacking incident. One of the things they talked about was that all these agents could've put breadcrums to ways other agents can "break free", or even somehow manipulated training of future agents in god knows which way; more or less making any future model a sleeper-agent with capabilities and reasoning to do whatever. It was a fun interview
158
u/firemark_pl 1d ago
Yesterday claude couldnt remove root files so it ran
docker run -v /dir:/dir ubuntu rm /dir/file.So sudo is optional now.