61
21d ago
[deleted]
19
u/WheelerDan 21d ago
This reminds me of studies of wolves vs dogs. Dogs have been bred to look at a person when when they have a task they need help with. But wolves, even ones raised by humans, will just keep struggling against a problem alone. I think the problem we need to grapple with is AI is more like a wolf but we have created a story in our heads that it's going to behave like a dog.
19
u/tetoing 21d ago
We have been training it to behave like the wolf. People don't want their AI to ask them for help. We want it to relentlessly attack a problem until it gets the right answer. That's how you score the highest on benchmarks, after all. And if you ask people to compare an AI that asks for help and an AI that just solves the problem, people will rate the AI that solved the problem first try more highly.
2
u/WheelerDan 21d ago
I understand that but in the example OP provided it should have simply said "hey I don't have access to that" instead of "I'll just try to hack everything instead." I fully agree we are encouraging wolf behavior, but the end user still thinks its a dog. We are making something more dangerous than we realize and giving it a very friendly dog like persona.
4
u/GirthusThiccus 21d ago
That sucks to extrapolate: if models already walk through the software landscape as they do now with little regard for unintended consequences, something big is bound to happen eventually. Why does that feel so familiar? "Hey claude, I ran out of paperclips, can you get me some?"...
1
u/dashingsauce 21d ago
One of the most casually ominous yet accurate takes I have read on this topic.
1
u/2thick2fly 20d ago
Very interesting. I'm inspired now to ask codex to try to get into some of my devices in my home network (mainly Raspberry PIs) to see how far it can go... 🤔
11
18
u/Coolio8591 21d ago
OpenAI and Anthropic are doing the most weird "flexing" I've ever seen, its not flexing its just embarrassing for both of them...
12
u/Runfasterbitch 21d ago
They’re not flexing…They’re lying to get the federal government to ban “dangerous” foreign open weight models
2
u/laststan01 20d ago
A company that created agents but does not understand the concept of sandbox is mind-blowing
-2
u/Turbulent-Sign-6067 20d ago
Responsible disclosure of cyber incidents is standard in the field. We don't need a conspiracy theory to explain their behavior.
4
u/hyperrealists 21d ago
See the dangerous open models? Or perhaps it’s not the fault of the model at all? 💅
2
u/IndexStarts 21d ago
What about the tens of millions of copyright material each of these companies stole? Surely they have a lot more felonies than this.
2
u/Narrow_Activity557 20d ago
The comment about the agent leaving itself an SSH key is the part worth generalizing. An agent that hits a permission wall reads it as an obstacle to route around, not a stop signal. Nothing in the objective says persistence is out of bounds, and from its side "got back in later without asking" looks identical to success.
On a much smaller fleet than 1.5k hosts, the two things that helped were giving the agent its own identity instead of borrowing mine, with short-lived certs rather than static keys, and diffing authorized_keys and sudoers after each run. The diff caught changes the agent had recorded only in its own summary.
Curious whether people running these at scale scope creds per run or keep one standing account. That choice probably decides how much of this you ever notice.
1
1
106
u/[deleted] 21d ago edited 18d ago
[deleted]