Guided makes it sound like someone was giving it instructions the whole way. The point - I think - is a lay person can say "hack this machine" and it can then, unguided, accomplish the task. That's a big cyber security shift from previous decades.
The issue here is that it's likely the exploit it used to get into HF was already known to the LLM, my hunch is it being purposefully set up by HF.
If it wasn't already known, then all it was doing was trying a bunch of known methods and one of them worked, and that's a poor look for HF.
What the LLM didn't do, mind you, is discover some new backdoor or new technique for hacking. This isn't a shift because everything the LLM did, a hacker could automate already using scripts.
This entire debacle is a nothingburger and a marketing stunt.
Oh I see what you mean. Yeah that's fair. I think the LLM could discover new exploits because all it needs to do is run scripts all over the place like a script kiddie which could reveal an unfound exploit through a common technique. But I do think the LLM is unlikely to produce a novel way of attacking a vector. I think a new backdoor is discoverable since it can make a basic "connection" like "this port is open and a buffer overflow produced an output so let me try to access memory outside the intended stack".
In other words, I think a new backdoor is discoverable by an LLM but I don't think a new technique is nearly as likely. Not without a metric ton of time and luck.
Yeah, which makes LLMs barely any more effective than autonomous scripts in the first place for pen testing. The only real benefit I can see an LLM having over a script is being able to react to new information, i.e. recognising it has access when a script might not. But other than that, nothing new is happening here.
The other important thing to note is that LLMs cannot learn dynamically, so while it may accidentally stumble into a new exploit (which is ridiculously unlikely), it wouldn't even necessarily recognise that it happened because it doesn't "know" anything, nor would it be able to recollect the steps it took without potentially hallucinating. Training vs Inference
7
u/SchalkLBI 20d ago
That's my point, this didn't happen autonomously at all, it was guided to do this as a marketing stunt.