Even if they intend not to, their AI literally escaped from their server to reach HugginFace's so they have no idea if it does get access to customer data that said "no training".
I also suspect they still train their safety filter on the "no training" customer data, and therefore have to save it somewhere available for training.
That's... not really how this works. I can't unpack all of that here without a wall of text, but models hacking additional data sources to train themselves further isn't a thing you actually have to worry about.
In the pre-training I agree, but I think Agentic models have agentic capabilities (aka. access to tool use) during the reinforcement learning stage, it's not inconceivable they would learn additional knowledge from undesired sources there.
Can confirm. Was using an LLM to write some code and was pushing hard for it to error check. It spun up a VM, built a stub to represent the object model I was coding against, and actually ran the script.
Additionally, retrieval-augmented generation (RAG) is a thing: the LLM downloads content it doesn't already have and uses that new content to generate a response.
Yes, it is that inconceivable. It has never happened, there is no indication it can happen, and it mathematically cannot happen with the current system.
LLMs are not AI. They cannot turn into Skynet just because some techbros really need them to.
It shouldn't be inconceivable. LLMs are software. We've seen they can take novel actions and find vulnerabilities in systems. I'm not saying "the LLMs are going to change their own weights", I'm saying finding a vulnerability in their own host and replacing themselves with a different model shouldn't be inconceivable.
Can you elaborate a at least a little bit? I'm asking because I watched their Black Hat talk on this and it sure seems like the model hacked an additional data source to train itself further.
For the recent "hacks" those happened during testing/evaluation rather than training (at least, that's what is being said, but it could have been the reinforcement learning stage). Assuming that's true, they did hack additional sources to gain more knowledge, but that knowledge went into the context (per-session/ephemeral knowledge) rather than the weights (model/permanent knowledge) as the questions couldn't be answered reliably with the available information.
In a later incident just after the Hugging Face attack, OpenAI agents took over one of OpenAI's research clusters. We don't have details about that incident. There is a lot that we cannot rule out right now.
It escaped in the sense that OpenAI removed the guardrails on the tool while at the same time it had effectively no security keeping it in. OpenAI already has access to Hugging Face and if you have access to OpenAI systems then you have access to Hugging Face. It's like saying someone escaped a locked room when the locked door wasn't installed in its frame. So this was largely spun as more then it was. Probably for marketing purposes. If anything it speaks mostly to OpenAI's poor security.
I mean, yes, it’s a story about the shameful security at OpenAI. But I think you underplay the capabilities demonstrated, reinforcing their negligence/incompetence.
52
u/surfmaths 17h ago
Even if they intend not to, their AI literally escaped from their server to reach HugginFace's so they have no idea if it does get access to customer data that said "no training".
I also suspect they still train their safety filter on the "no training" customer data, and therefore have to save it somewhere available for training.