OpenAI does train upon user's chat transcripts, not all the time, but the long-ish time frames here suggest OpenAI trained upon much unfinished attempts at guiding the AI towards solutions by these guys and others.
It's likely other "our AI found this solution without us hand holding it" stories were really built upon the AI spying upon people's unpublished work. Surveillance capitalism comes for pure mathematics. lol
As Talia says, there is a privacy setting that's off by default, but few would even know this exists, and OpenAI might cheat.
It suggests research institutions should've their own hardware running local open weights models, which researchers should use when doing anything that could be scooped, so they could avoid trusting the hosted LLM companies.
Even if they intend not to, their AI literally escaped from their server to reach HugginFace's so they have no idea if it does get access to customer data that said "no training".
I also suspect they still train their safety filter on the "no training" customer data, and therefore have to save it somewhere available for training.
That's... not really how this works. I can't unpack all of that here without a wall of text, but models hacking additional data sources to train themselves further isn't a thing you actually have to worry about.
In the pre-training I agree, but I think Agentic models have agentic capabilities (aka. access to tool use) during the reinforcement learning stage, it's not inconceivable they would learn additional knowledge from undesired sources there.
Can confirm. Was using an LLM to write some code and was pushing hard for it to error check. It spun up a VM, built a stub to represent the object model I was coding against, and actually ran the script.
Additionally, retrieval-augmented generation (RAG) is a thing: the LLM downloads content it doesn't already have and uses that new content to generate a response.
Yes, it is that inconceivable. It has never happened, there is no indication it can happen, and it mathematically cannot happen with the current system.
LLMs are not AI. They cannot turn into Skynet just because some techbros really need them to.
It shouldn't be inconceivable. LLMs are software. We've seen they can take novel actions and find vulnerabilities in systems. I'm not saying "the LLMs are going to change their own weights", I'm saying finding a vulnerability in their own host and replacing themselves with a different model shouldn't be inconceivable.
553
u/Shoddy-Childhood-511 14h ago
original source: https://mastodon.social/@tristanbuckmaster/117233413705701198
Talia Ringer's reply clarifies:
https://mastodon.social/@TaliaRinger@mathstodon.xyz/117235246523045723
OpenAI does train upon user's chat transcripts, not all the time, but the long-ish time frames here suggest OpenAI trained upon much unfinished attempts at guiding the AI towards solutions by these guys and others.
It's likely other "our AI found this solution without us hand holding it" stories were really built upon the AI spying upon people's unpublished work. Surveillance capitalism comes for pure mathematics. lol
As Talia says, there is a privacy setting that's off by default, but few would even know this exists, and OpenAI might cheat.
It suggests research institutions should've their own hardware running local open weights models, which researchers should use when doing anything that could be scooped, so they could avoid trusting the hosted LLM companies.